Zend_Memory is a component that offers API for managing data in limited memory mode.
2. References
3. Component Requirements, Constraints, and Acceptance Criteria
Module must effectively manage large amount of data.
Memory management must be based on access statistics.
Data load/unload process must be transparent for an application.
4. Dependencies on Other Framework Components
Zend_Exception
5. Theory of Operation
User creates Zend_Memory objects to store binary (string) value by using appropriate backend (Zend_Cache backends).
Stored value can be accessed by 'value' property of Zend_Memory class.
6. Milestones / Tasks
Milestone 1: Proposal is reviewed and approved by community
<p>If I'm understanding the manual section on Zend_Cache and the proposal for Zend_Memory correctly, Zend_Memory could theoretically be built on top of Zend_Cache. Where Zend_Cache backends are restricted to non-volatile storage, Zend_Memory would provide memory management which would rely on non-volatile backends only when necessary (utilizing Zend_Cache) and would swap data in and out of memory on the basis of request frequency.</p>
<p>The functionality is not the same. <br />
Zend_Cache is intended to store data between script executions (requests), and Zend_Memory should store values during script execution (to free used memory while value is not used).</p>
<p>But Zend_Cache backends can (and should) be used as storage backends.</p>
<p>This proposal should also include functionality to leech the data from a file pointer (http stream etc) without consuming much memory to do it if its too large for the memory limit and I would be inclined to have Zend_Memory_Manager too that forked out Zend_Memory instances and monitored how much ram each Zend_Memory was using to keep a global ram limit in place for all of the containers.</p>
<p>Something to handle substr() of the dataset would also be useful to me. </p>
<p>I would also be inclined to use Zend_Cache in a way to have intelligently split the dataset into multiple smaller parts and use the cache on each of the pieces instead of the whole string.</p>
<p>This module would be VERY useful to me in quite a few projects where I could have anything upto 500meg of data loaded at once if it wasn't such a chore to do with php without overloading servers.</p>
<p>I do agree, that it's necessary to add Zend_Memory_Manager class.<br />
I planned to use some static methods to request memory usage information, but memory manager conception is more common and clear.</p>
<p>File access management looks interesting. Do you mean something like "memory mapped files"? It may be future enhancement of Zend_Memory.</p>
<p>As to long strings management. I don't see now, how it may be implemented without complex API. I don't know way to hook access to the part of string.</p>
<ac:macro ac:name="code"><ac:plain-text-body><![CDATA[$memObject->value[$offset] = $someChar;]]></ac:plain-text-body></ac:macro>
<p>invokes __get() method, which returns string or string reference. We don't know, which offset will be used.</p>
<ac:macro ac:name="note"><ac:parameter ac:name="title">Zend comments</ac:parameter><ac:rich-text-body>
<p>This proposal is approved for inclusion in Zend Framework.</p>
<p>The file access feature does sound interesting, but it is not a requirement for the first release of Zend_Memory. Like you say, it may be a future enhancement.</p></ac:rich-text-body></ac:macro>
7 Comments
comments.show.hideJul 21, 2006
Thomas Weidner
<p>Zend_Cache seems to have the same functionality as this proposal.</p>
<p>Where are the benefits from this proposal to the existing Zend_Cache ?</p>
<p>Greetings <br />
Thomas</p>
Aug 02, 2006
Matthew Turland
<p>If I'm understanding the manual section on Zend_Cache and the proposal for Zend_Memory correctly, Zend_Memory could theoretically be built on top of Zend_Cache. Where Zend_Cache backends are restricted to non-volatile storage, Zend_Memory would provide memory management which would rely on non-volatile backends only when necessary (utilizing Zend_Cache) and would swap data in and out of memory on the basis of request frequency.</p>
Sep 07, 2006
Alexander Veremyev
<p>Thanks! It's very good idea.</p>
<p>As I have chacked all Zend_Cache backends can be used here and I just made necessary API changes.</p>
Sep 07, 2006
Alexander Veremyev
<p>The functionality is not the same. <br />
Zend_Cache is intended to store data between script executions (requests), and Zend_Memory should store values during script execution (to free used memory while value is not used).</p>
<p>But Zend_Cache backends can (and should) be used as storage backends.</p>
<p>Thanks for your comment!</p>
Sep 08, 2006
Cameron Brunner
<p>This proposal should also include functionality to leech the data from a file pointer (http stream etc) without consuming much memory to do it if its too large for the memory limit and I would be inclined to have Zend_Memory_Manager too that forked out Zend_Memory instances and monitored how much ram each Zend_Memory was using to keep a global ram limit in place for all of the containers.</p>
<p>Something to handle substr() of the dataset would also be useful to me. </p>
<p>I would also be inclined to use Zend_Cache in a way to have intelligently split the dataset into multiple smaller parts and use the cache on each of the pieces instead of the whole string.</p>
<p>This module would be VERY useful to me in quite a few projects where I could have anything upto 500meg of data loaded at once if it wasn't such a chore to do with php without overloading servers.</p>
Sep 11, 2006
Alexander Veremyev
<p>I do agree, that it's necessary to add Zend_Memory_Manager class.<br />
I planned to use some static methods to request memory usage information, but memory manager conception is more common and clear.</p>
<p>File access management looks interesting. Do you mean something like "memory mapped files"? It may be future enhancement of Zend_Memory.</p>
<p>As to long strings management. I don't see now, how it may be implemented without complex API. I don't know way to hook access to the part of string.</p>
<ac:macro ac:name="code"><ac:plain-text-body><![CDATA[$memObject->value[$offset] = $someChar;]]></ac:plain-text-body></ac:macro>
<p>invokes __get() method, which returns string or string reference. We don't know, which offset will be used.</p>
Nov 30, 2006
Bill Karwin
<ac:macro ac:name="note"><ac:parameter ac:name="title">Zend comments</ac:parameter><ac:rich-text-body>
<p>This proposal is approved for inclusion in Zend Framework.</p>
<p>The file access feature does sound interesting, but it is not a requirement for the first release of Zend_Memory. Like you say, it may be a future enhancement.</p></ac:rich-text-body></ac:macro>