20022010 Ericsson AB. All Rights Reserved. The contents of this file are subject to the Erlang Public License, Version 1.1, (the "License"); you may not use this file except in compliance with the License. You should have received a copy of the Erlang Public License along with this software. If not, it can be retrieved online at http://www.erlang.org/. Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. Performance comparison Håkan Mattsson Håkan Mattsson Håkan Mattsson 2007-06-15 %VSN% megaco_performance.xml
Comparison of encoder/decoders

The Megaco/H.248 standard defines both a plain text encoding and a binary encoding (ASN.1 BER) and we have implemented encoders and decoders for both. We do supply a bunch of different encoding/decoding modules and the user may in fact implement their own (like our erl_dist module). Using a non-standard encoding format has its obvious drawbacks, but may be useful in some configurations.

We have made four different measurements of our Erlang/OTP implementation of the Megaco/H.248 protocol stack, in order to compare our different encoders/decoders. The result of each one is summarized in the table below.

The result above are the fastest of these configurations for each codec. The figures presented are the average of all used messages.

For comparison, also included are performance figures where the flex driver was built as non-reentrant flex (figures within parenthesis).

Codec and config Size Encode Decode Total pretty 336 22 76 98 pretty [flex] 336 22 (22) 41 (40) 63 (62) compact 181 19 63 82 compact [flex] 181 19 (19) 38 (36) 57 (55) per bin 91 63 69 132 per bin [driver] 91 43 45 88 per bin [native] 91 47 51 99 per bin [driver,native] 91 26 29 55 ber bin 165 35 42 77 ber bin [driver] 165 35 37 72 ber bin [native] 165 19 26 45 ber bin [driver,native] 165 19 20 39 erl_dist 875 5 10 15 erl_dist [megaco_compressed] 405 6 7 13 erl_dist [compressed] 345 86 21 107 erl_dist [megaco_compressed,compressed] 200 71 12 83 Codec performance
System performance characteristics

This is primarily a way to show the effects of using the reentrant flex scanner instead of the non-reentrant.

As can be seen from the figures above there is no real difference between a non-reentrant and an reentrant flex scanner when it comes to the decode times of an individual message.

The real difference is instead in system characteristics, which is best shown with the mstone1 test.

When running SMP erlang on a multi-core machine the "throughput" is significantly higher. The mstone1 test is an extreme test, but it shows what is gained by using the reentrant flex-scanner.

MStone1 with mstone1.sh -d flex -s 8
Description of encoders/decoders

In Appendix A of the Megaco/H.248 specification (RFC 3525), there are about 30 messages that shows a representative call flow. We have also added a few extra version 1, version 2 and version 3 messages. We have used these messages as basis for our measurements. Our figures have not been weighted in regard to how frequent the different kinds of messages that are sent between the media gateway and its controller.

The test compares the following encoder/decoders:

pretty - pretty printed text. In the text encoding, the protocol stack implementors have the choice of using a mix of short and long keywords. It is also possible to add white spaces to improve readability. The pretty text encoding utilizes long keywords and an indentation style like the text examples in the Megaco/H.248 specification.

compact - the compact text encoding uses the shortest possible keywords and no optional white spaces.

ber - ASN.1 BER.

per - ASN.1 PER. Not standardized as a valid Megaco/H.248 encoding, but included for the matter of completeness as its encoding is extremely compact.

erl_dist - Erlang's native distribution format. Not standardized as a valid Megaco/H.248 encoding, but included as a reference due to its well known performance characteristics. Erlang is a dynamically typed language and any Erlang data structure may be serialized to the erl_dist format by using built-in functions.

The actual encoded messages have been collected in one directory per encoding type, containing one file per encoded message.

Here follows an example of a text message to give a feeling of the difference between the pretty and compact versions of text messages. First the pretty printed, well indented version with long keywords:

MEGACO/1 [124.124.124.222] 
  Transaction = 9998 { 
    Context = - { 
      ServiceChange = ROOT { 
        Services { 
          Method = Restart, 
          ServiceChangeAddress = 55555, 
          Profile = ResGW/1, 
          Reason = "901 MG Cold Boot"
        }
      }  
    }
  }    

Then the compact text version without indentation and with short keywords:

!/1 [124.124.124.222] T=9998{
  C=-{SC=ROOT{SV{MT=RS,AD=55555,PF=ResGW/1,RE="901 MG Cold Boot"}}}}    
Setup

The measurements has been performed on a Dell PowerEdge 1950iii with 2* Intel Xeon L5430 @ 2.66 GHz, with 8 GB memory and running SLES 10 SP2 x86_64, kernel 2.6.16.60-0.34-smp. Software versions was open source OTP R13B and megaco-3.11.

Summary

In our measurements we have seen that there are no significant differences in message sizes between ASN.1 BER and the compact text format. Some care should be taken when using the pretty text style (which is used in all the examples included in the protocol specification and preferred during debugging sessions) since the messages can then be quite large. If the message size really is a serious issue, our per encoder should be used, as the ASN.1 PER format is much more compact than all the other alternatives. Its major drawback is that it is has not been approved as a valid Megaco/H.248 message encoding.

When it comes to pure encode/decode performance, it turns out that:

our fastest binary encoder (ber) is about equal to our fastest text encoder (compact).

our fastest binary decoder (ber) is about 47% (44%) faster than our fastest text decoder (compact).

If the pure encode/decode performance really is a serious issue, our erl_dist encoder could be used, as the encoding/decoding of the erlang distribution format is much faster than all the other alternatives. Its major drawback is that it is has not been approved as a valid Megaco/H.248 message encoding.

There is no performance advantage of building (and using) a non-reentrant flex scanner over a reentrant flex scanner (if flex supports building such a scanner).

Please, observe that these performance figures are related to our implementation in Erlang/OTP. Measurements of other implementations using other tools and techniques may of course result in other figures.