ZF-7493: ZendAMF serialization slow?

Issue Type: Performance Improvement Created: 2009-08-05T13:51:19.000+0000 Last Updated: 2010-09-22T14:12:06.000+0000 Status: Resolved Fix version(s): - 1.10.4 (28/Apr/10)

Reporter: Prabu Singh (cayennecode) Assignee: Cal Evans (calevans) Tags: - Zend_Amf

Related issues: - ZF-10487

Attachments: - Amf.Combined-NoObjectsByRef.diff


I've compared ZendAMF to AMFPHP, by returning a dataset with 5000+ rows from a table, using the exact same code (both in PHP, and AS3). ZendAMF averages a 16 second return time, whereas AMFPHP returns in 3 seconds. I'm guessing the added time is coming from the process of serializing data to send back to Flash.


Posted by Matthew Grippo (mgrippo) on 2009-10-27T18:42:09.000+0000

I'd like to confirm this issue. Spent the day trying to chase down why the performance difference was so drastic. With 20000 rows/3 columns: AMFPHP - approx 5 seconds ZendAMF - approx 70 seconds

I was able to narrow it down to the Serializer (serialization of objects) class (AMF3 object / array writing). probably related to the Stream class. Was also able to confirm this was not a "output" issue at the apache/php level to the browser.

Posted by Andreas Adam (acadam71) on 2009-12-23T03:34:29.000+0000

I'd like to confirm this issue too: we don't send thousands of rows but we create complex objects with 10 or more arrays, e.g.: class person var vehicles: Array; var groups: Array; var a1: Array; var a2: Array; ... In each array there are also some complex objects, but not many. It takes more than 3 seconds to get an object person from PHP to Flex.

Posted by Mark Reidenbach (mreiden) on 2009-12-30T08:34:25.000+0000

I just got finished working on a project which showed a drastic performance hit when switching from AMFphp to Zend_Amf. Our serialization performance exploded from 2.3 seconds to 25.5 seconds.

I'm going to attach two patches to this issue which brought the serialization time down to roughly the same speed as AMFphp.

Any comments on whether these patches help others would be appreciated.

Posted by Mark Reidenbach (mreiden) on 2009-12-30T08:47:57.000+0000

These two patches were tested on php 5.3 and are against Zend AMF 1.9.6. After applying both the serialization times for our project containing many multi-dimensional arrays decreased from 25.5 seconds to 2.3 seconds which matches the speed of using the older AMFphp serialization. Comments on whether these patches help others would be appreciated.


This patch changes Zend_Amf to always write strings without doing an array_search to see if a string has already been written. This is how AMFphp handled strings and did result in the download size increasing from around 179kb to 287kb for our project, but that's acceptable to us for the increased serialization speed. Perhaps a better approach would be to use some sort of hash for string lookups to decrease the number of array elements array_search must search through.

This patch passes data by reference rather than by value in the serialization routines. This prevents making copies of large data while doing serialization.

Posted by Wade Arnold (wadearnold) on 2010-01-04T11:38:21.000+0000

I will run this throught the unit tests tonight. If all passes I will submit a patch for the next mini release. Mark thanks for your patch!!

Posted by Mark Reidenbach (mreiden) on 2010-01-04T22:57:29.000+0000

After running through the AllTests.php unit tests from…, the attached patch provides the same results as testing against Zend_Amf 1.9.6 with the data-as-refs patch applied.

The noref-writeString patch causes 3 errors, but I believe this is due to not using references to some strings (larger payload, much reduced response time).

Posted by Wade Arnold (wadearnold) on 2010-01-06T12:34:00.000+0000

Zend Framework requires that we php strict mode standards. If we apply this patch I get all kinds of "Only variables should be passed by reference" so I guess we need to jump into this and figure out were we don't need to explicitly pass by reference because php is already passing it that way.

Posted by Mark Reidenbach (mreiden) on 2010-01-06T15:00:22.000+0000

Wade, did you apply patch #3 [Amf.Response_Body_By_Value.diff] when you ran your tests? I'm not seeing any of the "Only variables should be passed by reference" errors when I run the tests with all three patches applied.

Without patch #3 I get many of the passed by reference errors and a summary of "Tests: 160, Assertions: 289, Failures: 2, Errors: 72."

With all three patches I get a summary of "OK (160 tests, 416 assertions)" which matches what I get when run against 1.9.6.

Posted by Mark Reidenbach (mreiden) on 2010-01-06T17:00:49.000+0000

Wade, if you're still getting the "Only variables should be passed by reference" errors with the previous 3 patches, here's an initial try at passing non-objects by reference as the first parameter and php objects (objects, dates, xml, etc) as a new third parameter of the writeTypeMarker serialization methods.

I haven't gotten to do much testing with this latest patch, but it does pass the unit tests and initial testing of our project.

Posted by Wade Arnold (wadearnold) on 2010-01-17T12:11:43.000+0000

So your suggesting apply both Amf.Response_Body_By_Value.diff and Amf.Combined-NoObjectsByRef.diff patches?

Posted by Wade Arnold (wadearnold) on 2010-01-17T12:14:43.000+0000

Nope just Amf.Combined-NoObjectsByRef.diff it looks like

Posted by Mark Reidenbach (mreiden) on 2010-01-18T08:50:17.000+0000

Wade, I had some time to do more testing on this over the weekend.

I created a unit test for serializing a large array and after using this test, I believe the only patch that is needed is the one to not reference strings.

My previous patch [Amf.noref-writeString.diff] causes some test failures due to not referencing any strings which some of the tests expect, so here's a new patch based on a recommendation that in_array can be very slow as more elements are added to an array while checking if an array key exists is roughly constant.

Hopefully this much simpler patch will fix this performance issue and it passes the unit tests.

Patches: Amf.perform.ref-writeString.diff : Use the string as the array key and store the reference number as the value for much quicker lookup performance.

Amf.ResponseTest.php.diff : Unit test to make sure the large array serialization time hasn't ballooned by a factor of 10. (Is there a better way of testing the speed other than comparing against a "high enough" number that works on today's hardware?)

largeArrayData.bin : This is simply my test dataset compressed with gzcompress that consists of several large arrays containing almost . It is 624kB in size though, so maybe this isn't acceptable to include for unit testing.

Posted by Matthias Steinböck (rocksocke) on 2010-01-18T18:43:47.000+0000

Using the latest zendfw beta and the patch provided on your side, i get the following error using zamfbrowser

<pre class="highlight">

There was an error loading the server's info.  Error: (
  bubbles = false
  cancelable = true
  currentTarget = (mx.rpc.remoting.mxml::RemoteObject)#1
    channelSet = (mx.messaging::ChannelSet)#2
      authenticated = false
      channelIds = (Array)#3
        [0] (null)
      channels = (Array)#4
        [0] (mx.messaging.channels::AMFChannel)#5
          authenticated = false
          channelSets = (Array)#6
            [0] (mx.messaging::ChannelSet)#2
          connected = true
          connectTimeout = -1
          enableSmallMessages = true
          endpoint = "http://............../gateway.php"
          failoverURIs = (Array)#7
          id = (null)
          mpiEnabled = false
          netConnection = (
            client = (mx.messaging.channels::AMFChannel)#5
            connected = false
            maxPeerConnections = 8
            objectEncoding = 3
            proxyType = "none"
            uri = "http://............../gateway.php"
          piggybackingEnabled = false
          polling = false
          pollingEnabled = true
          pollingInterval = 3000
          protocol = "http"
          reconnecting = false
          recordMessageSizes = false
          recordMessageTimes = false
          requestTimeout = -1
          uri = "http://............../gateway.php"
          url = "http://............../gateway.php"
          useSmallMessages = false
      clustered = false
      connected = true
      currentChannel = (mx.messaging.channels::AMFChannel)#5
      initialDestinationId = (null)
      messageAgents = (Array)#9
        [0] (mx.rpc::AsyncRequest)#10
          authenticated = false
          autoConnect = true
          channelSet = (mx.messaging::ChannelSet)#2
          clientId = (null)
          connected = true
          defaultHeaders = (null)
          destination = "AMF"
          id = "2D1CCD62-E448-16FD-79B8-4465C9CE0DDB"
          reconnectAttempts = 0
          reconnectInterval = 0
          requestTimeout = -1
          subtopic = ""
    concurrency = "multiple"
    destination = "AMF"
    endpoint = "http://............../gateway.php"
    getServices = (mx.rpc.remoting.mxml::Operation)#11
      argumentNames = (Array)#12
      arguments = (Object)#13
      concurrency = "multiple"
      lastResult = (null)
      makeObjectsBindable = true
      name = "getServices"
      service = (mx.rpc.remoting.mxml::RemoteObject)#1
      showBusyCursor = true
    makeObjectsBindable = true
    operations = (Object)#14
      getServices = (mx.rpc.remoting.mxml::Operation)#11
    requestTimeout = -1
    showBusyCursor = true
    source = "ZendAmfServiceBrowser"
  eventPhase = 2
  fault = (mx.rpc::Fault)#15
    content = (Object)#16
    errorID = 0
    faultCode = "Server.Acknowledge.Failed"
    faultDetail = "Was expecting mx.messaging.messages.AcknowledgeMessage, but received null"
    faultString = "Didn't receive an acknowledge message"
    message = "faultCode:Server.Acknowledge.Failed faultString:'Didn't receive an acknowledge message' faultDetail:'Was expecting mx.messaging.messages.AcknowledgeMessage, but received null'"
    name = "Error"
    rootCause = (null)
  headers = (null)
  message = (mx.messaging.messages::ErrorMessage)#17
    body = (Object)#16
    clientId = (null)
    correlationId = "06927F94-0A5F-A7AC-EAA6-4465C9CFB388"
    destination = ""
    extendedData = (null)
    faultCode = "Server.Acknowledge.Failed"
    faultDetail = "Was expecting mx.messaging.messages.AcknowledgeMessage, but received null"
    faultString = "Didn't receive an acknowledge message"
    headers = (Object)#18
    messageId = "A2AAC85B-DA13-57AE-45A9-4465D0C4D16C"
    rootCause = (null)
    timestamp = 0
    timeToLive = 0
  messageId = "A2AAC85B-DA13-57AE-45A9-4465D0C4D16C"
  statusCode = 0
  target = (mx.rpc.remoting.mxml::RemoteObject)#1
  token = (mx.rpc::AsyncToken)#19
    message = (mx.messaging.messages::RemotingMessage)#20
      body = (Array)#21
      clientId = (null)
      destination = "AMF"
      headers = (Object)#22
        DSEndpoint = (null)
        DSId = "nil"
      messageId = "06927F94-0A5F-A7AC-EAA6-4465C9CFB388"
      operation = "getServices"
      source = "ZendAmfServiceBrowser"
      timestamp = 0
      timeToLive = 0
    responders = (null)
    result = (null)
  type = "fault"

print_r-ing the result from ->handle() i get this:

                            [body] => <methods>....correctxml....</methods>
    [_headers:protected] => Array
    [_outputStream:protected] => Zend_Amf_Parse_OutputStream Object
            [_stream:protected] => 



Posted by Matthias Steinböck (rocksocke) on 2010-01-19T06:55:48.000+0000

please ignore my previous post. the problems i encountered can be traced back to errors i did myself and with the usage of zamfbrowser.

allthough, i found another issue - i'm not sure if this is a bug or a behaviour: <>

thanks for your brilliant work!



Posted by Mark Reidenbach (mreiden) on 2010-03-04T12:09:45.000+0000

This issue still affects ZendFramework 1.10.2.

The only change needed to greatly increase performance is to use an associative array and array\_key\_exists instead of array\_search when writing referenced strings in the AMF3 serializer.

The strict type checking used in array\_search is unnecessary since only string data is ever passed to the writeString method and checking if an array key exists is much faster than searching an array as the array becomes increasingly large.

Patch is below:

--- Parse/Amf3/Serializer.orig.php 2010-01-18 12:34:23.000000000 -0600 +++ Parse/Amf3/Serializer.php 2010-03-04 13:56:36.000000000 -0600 @@ -212,31 +212,31 @@ /\*\* \* Send string to output stream \* \* @param string $string \* @return Zend\_Amf\_Parse\_Amf3\_Serializer \*/ public function writeString($string) { $len = strlen($string); if(!$len){ $this->writeInteger(0x01); return $this; }

- $ref = array\_search($string, $this->\_referenceStrings, true);
- $ref = array\_key\_exists($string, $this->\_referenceStrings) ? $this->\_referenceStrings[$string] : false; if($ref === false){
- $this->\_referenceStrings[] = $string;
- $this->_referenceStrings[$string] = count($this->_referenceStrings);
     } else {
         $ref <<= 1;
     return $this;



  - Send ByteArray to output stream
  - @param string|Zend\_Amf\_Value\_ByteArray $data
  - @return Zend\_Amf\_Parse\_Amf3\_Serializer


Posted by Marc Bennewitz (private) (mabe) on 2010-03-08T15:13:51.000+0000

What about moving out the serialization to Zend\_Serializer\_Adapter\_Amf[0|1] ? -> This could minimize include files for Zend\_Amf and speed up AMF\* serializer of Zend\_Serializer ?

For now Zend\_Serializer\_Adapter\_Amf needs instantiate 3 classes for serializing: \* Zend\_Serializer\_Adapter\_Amf\* \* Zend\_Amf\_Parse\_OutputStream \* Zend\_Amf\_Parse\_Amf\*\_Serializer

    <pre class="highlight">
    $stream     = new Zend_Amf_Parse_OutputStream();
    $serializer = new Zend_Amf_Parse_Amf*_Serializer($stream);
    return $stream->getStream();

and on unserializing, too: \* Zend\_Serializer\_Adapter\_Amf\* \* Zend\_Amf\_Parse\_InputStream \* Zend\_Amf\_Parse\_Amf\*\_Deserializer

    <pre class="highlight">
    $stream       = new Zend_Amf_Parse_InputStream($value);
    $deserializer = new Zend_Amf_Parse_Amf*_Deserializer($stream);
    return $deserializer->readTypeMarker();

-> This could be handled similar to the PythonPickle serializer.



Posted by Matthew Weier O'Phinney (matthew) on 2010-04-28T07:58:45.000+0000

Patches reviewed and applied to trunk and 1.10 release branch.



Posted by Philippe Bettler (pbettler) on 2010-05-21T11:56:07.000+0000

Matthew, I just downloaded 1.10.4 and it seems that the patch from Mark Reidenbach hasn't been applied. I am looking at Serializer.php line 231, function writeString. The string is not stored as a key but as a value. Can you please check it out? Thanks



Posted by James Spurin (funkjames) on 2010-05-23T08:28:19.000+0000

I also downloaded 1.10.4 today.

As Philippe mentioned, the patch from Mark Reidenbach mentioned at 04/Mar/10 12:09 PM does not seem to have been applied. I manually made the changes to 1.10.4 using the details provided by Mark and noticed substantial performance improvements.

On a 70,000 row table the query took 13 seconds as opposed to 60+ seconds prior to the patch being applied.



Posted by Otto (wedge) on 2010-06-03T01:10:10.000+0000

I also tried Mark's patch from 04/Mar/10. It's only slightly faster here though. Getting 400 objects I go from 2300ms -> 2000ms. Is the count part of his patch really necessary? I've modified it, and it appears to work and also appears to be slightly faster.

    <pre class="literal">
    -        $ref = array_search($string, $this->_referenceStrings, true);
    +        $ref = array_key_exists($string, $this->_referenceStrings) ? $this->_referenceStrings[$string] : false;
             if($ref === false){
    -            $this->_referenceStrings[] = $string;
    +            $this->_referenceStrings[$string] = 1;



Posted by Otto (wedge) on 2010-06-03T03:04:42.000+0000

Upon further testing it looks like the count part of Mark's patch is indeed needed so please disregard my addition.



Posted by Mark Reidenbach (mreiden) on 2010-06-03T07:44:01.000+0000

Patch from March 4, 2010 updated against 1.10.5 as an attachment.



Posted by Henry Tran (newbs) on 2010-07-26T06:59:34.000+0000

Is the patch applied in 1.10.6?



Have you found an issue?

See the Overview section for more details.


© 2006-2021 by Zend by Perforce. Made with by awesome contributors.

This website is built using zend-expressive and it runs on PHP 7.