Zend Framework

Zend_XmlRpc_Generator: memory leak

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Critical Critical
  • Resolution: Fixed
  • Affects Version/s: 1.10.0, 1.10.1, 1.10.2, 1.10.3
  • Fix Version/s: 1.10.4
  • Component/s: Zend_XmlRpc_Server
  • Labels:
    None

Description

With Zend Framework 1.9.5 we've any problems with our xmlrpc-server.
Now (with 1.10.2) we've got this error:
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 220715 bytes) in /usr/local/zend/share/ZendFramework/library/Zend/XmlRpc/Generator/GeneratorAbstract.php on line 127

I've tried both:

  • Zend_XmlRpc_Value::setGenerator(new Zend_XmlRpc_Generator_XmlWriter());
  • Zend_XmlRpc_Value::setGenerator(new Zend_XmlRpc_Generator_DomDocument());

Best Regards
Sascha Wojewsky

Issue Links

Activity

Hide
David Abdemoulaie added a comment -

I have been able to duplicate this memory leak as well, please view the attached xdebug trace. (45M, 1.2M compressed).

Things to note:

The return value from this xmlrpc call is an array in the following format:

array(
    'values' => array(
        0 => array(
            'id' => 1,
            'name' => 'Foobar',
            'group_id' => 7,
            'priority' => 0,
        ),
       // ...
       740 => array(/* ... */),
    ),
    'last_modified' => '2010-01-01 00:00:00',
)

The entire resultset is loaded into the PHP array format between lines 5577348 and 6837560. This increases script memory usage by 1.2M for a total of 6.6M.

Line 63810 Zend_XmlRpc_Response->__toString(), invoked marking beginning of copying the PHP array into the corresponding Zend_XmlRpc_Value object graph - current usage 6.5M

Line 86148 - Zend_XmlRpc_Value object graph construction completes - current usage 8.7M

Line 86245 - Zend_XmlRpc_Generator_GeneratorAbstract->__toString() invoked, leak begins - current usage 9072432 bytes

Now continue searching the document for invocations of Zend_XmlRpc_Generator_GeneratorAbstract->__toString():

Line 86283 - Usage: 9072744 (312 byte increase)
Line 86321 - Usage: 9073144 (400 byte increase)
Line 86349 - Usage: 9073612 (468 byte increase)
...
Line 247124 - Usage: 726055636
Line 247162 - Usage: 726371428 (315792 byte increase)
Line 247200 - Usage: 726687288 (315860 byte increase)

The memory usage continues to increase exponentially. You can view this easily by simply grepping for Zend_XmlRpc_Generator_GeneratorAbstract->stripDeclaration() and noting the increasing deltas in column 3. By the time the script crashes, each call to saveXml() is increasing the memory usage by over 300KB.

There is clearly something wrong here, though I haven't yet figured out yet.

The memory usage is increasing exponentially, but the rate of increase is linear. This leads me to believe it's a problem similar to:

<?php

$o = new stdClass();
$o->str = 'abc';

$arr = array($o);
for ($i = 0; $i < 100; $i++) {
    $o = clone $o; 
    $o->str .= 'abc';
    $arr[] = $o; 
}
Show
David Abdemoulaie added a comment - I have been able to duplicate this memory leak as well, please view the attached xdebug trace. (45M, 1.2M compressed). Things to note: The return value from this xmlrpc call is an array in the following format:
array(
    'values' => array(
        0 => array(
            'id' => 1,
            'name' => 'Foobar',
            'group_id' => 7,
            'priority' => 0,
        ),
       // ...
       740 => array(/* ... */),
    ),
    'last_modified' => '2010-01-01 00:00:00',
)
The entire resultset is loaded into the PHP array format between lines 5577348 and 6837560. This increases script memory usage by 1.2M for a total of 6.6M. Line 63810 Zend_XmlRpc_Response->__toString(), invoked marking beginning of copying the PHP array into the corresponding Zend_XmlRpc_Value object graph - current usage 6.5M Line 86148 - Zend_XmlRpc_Value object graph construction completes - current usage 8.7M Line 86245 - Zend_XmlRpc_Generator_GeneratorAbstract->__toString() invoked, leak begins - current usage 9072432 bytes Now continue searching the document for invocations of Zend_XmlRpc_Generator_GeneratorAbstract->__toString(): Line 86283 - Usage: 9072744 (312 byte increase) Line 86321 - Usage: 9073144 (400 byte increase) Line 86349 - Usage: 9073612 (468 byte increase) ... Line 247124 - Usage: 726055636 Line 247162 - Usage: 726371428 (315792 byte increase) Line 247200 - Usage: 726687288 (315860 byte increase) The memory usage continues to increase exponentially. You can view this easily by simply grepping for Zend_XmlRpc_Generator_GeneratorAbstract->stripDeclaration() and noting the increasing deltas in column 3. By the time the script crashes, each call to saveXml() is increasing the memory usage by over 300KB. There is clearly something wrong here, though I haven't yet figured out yet. The memory usage is increasing exponentially, but the rate of increase is linear. This leads me to believe it's a problem similar to:
<?php

$o = new stdClass();
$o->str = 'abc';

$arr = array($o);
for ($i = 0; $i < 100; $i++) {
    $o = clone $o; 
    $o->str .= 'abc';
    $arr[] = $o; 
}
Hide
David Abdemoulaie added a comment -

I have confirmed my suspicions from the example above. Each Zend_XmlRpc_Value node has an $_xml property. During the _generateXml() methods the Generator is reused, thus a full copy of the DOM as xml is saved at each node.

e.g.

First Node

<?xml version="1.0"?>
<methodResponse>
  <params>
    <param>
      <value>
        <struct>
          <member>
            <name>updates</name>
            <value>
              <array>
                <data>
                  <value>
                    <struct>
                      <member>
                        <name>id</name>
                        <value>
                          <string>1</string>
                        </value>
                      </member>
                    </struct>
                  </value>
                </data>
              </array>
            </value>
          </member>
        </struct>
      </value>
    </param>
  </params>
</methodResponse>

Second Node

<?xml version="1.0"?>
<methodResponse>
  <params>
    <param>
      <value>
        <struct>
          <member>
            <name>updates</name>
            <value>
              <array>
                <data>
                  <value>
                    <struct>
                      <member>
                        <name>id</name>
                        <value>
                          <string>1</string>
                        </value>
                      </member>
                      <member>
                        <name>name</name>
                        <value>
                          <string>Relocate / Remove</string>
                        </value>
                      </member>
                    </struct>
                  </value>
                </data>
              </array>
            </value>
          </member>
        </struct>
      </value>
    </param>
  </params>
</methodResponse>

Third Node

<?xml version="1.0"?>
<methodResponse>
  <params>
    <param>
      <value>
        <struct>
          <member>
            <name>updates</name>
            <value>
              <array>
                <data>
                  <value>
                    <struct>
                      <member>
                        <name>id</name>
                        <value>
                          <string>1</string>
                        </value>
                      </member>
                      <member>
                        <name>name</name>
                        <value>
                          <string>Relocate / Remove</string>
                        </value>
                      </member>
                      <member>
                        <name>description</name>
                        <value>
                          <string/>
                        </value>
                      </member>
                    </struct>
                  </value>
                </data>
              </array>
            </value>
          </member>
        </struct>
      </value>
    </param>
  </params>
</methodResponse>
Show
David Abdemoulaie added a comment - I have confirmed my suspicions from the example above. Each Zend_XmlRpc_Value node has an $_xml property. During the _generateXml() methods the Generator is reused, thus a full copy of the DOM as xml is saved at each node. e.g. First Node
<?xml version="1.0"?>
<methodResponse>
  <params>
    <param>
      <value>
        <struct>
          <member>
            <name>updates</name>
            <value>
              <array>
                <data>
                  <value>
                    <struct>
                      <member>
                        <name>id</name>
                        <value>
                          <string>1</string>
                        </value>
                      </member>
                    </struct>
                  </value>
                </data>
              </array>
            </value>
          </member>
        </struct>
      </value>
    </param>
  </params>
</methodResponse>
Second Node
<?xml version="1.0"?>
<methodResponse>
  <params>
    <param>
      <value>
        <struct>
          <member>
            <name>updates</name>
            <value>
              <array>
                <data>
                  <value>
                    <struct>
                      <member>
                        <name>id</name>
                        <value>
                          <string>1</string>
                        </value>
                      </member>
                      <member>
                        <name>name</name>
                        <value>
                          <string>Relocate / Remove</string>
                        </value>
                      </member>
                    </struct>
                  </value>
                </data>
              </array>
            </value>
          </member>
        </struct>
      </value>
    </param>
  </params>
</methodResponse>
Third Node
<?xml version="1.0"?>
<methodResponse>
  <params>
    <param>
      <value>
        <struct>
          <member>
            <name>updates</name>
            <value>
              <array>
                <data>
                  <value>
                    <struct>
                      <member>
                        <name>id</name>
                        <value>
                          <string>1</string>
                        </value>
                      </member>
                      <member>
                        <name>name</name>
                        <value>
                          <string>Relocate / Remove</string>
                        </value>
                      </member>
                      <member>
                        <name>description</name>
                        <value>
                          <string/>
                        </value>
                      </member>
                    </struct>
                  </value>
                </data>
              </array>
            </value>
          </member>
        </struct>
      </value>
    </param>
  </params>
</methodResponse>
Hide
Matthew Weier O'Phinney added a comment -

Patch created by hobodave, and applied by matthew in trunk and 1.10 release branch; saw improvements of 1GB memory usage -> 8.8MB after patch, and > 300% decrease in time needed to execute.

Will release with 1.10.4.

Show
Matthew Weier O'Phinney added a comment - Patch created by hobodave, and applied by matthew in trunk and 1.10 release branch; saw improvements of 1GB memory usage -> 8.8MB after patch, and > 300% decrease in time needed to execute. Will release with 1.10.4.

People

Vote (2)
Watch (2)

Dates

  • Created:
    Updated:
    Resolved: