Skip to end of metadata
Go to start of metadata

<ac:macro ac:name="unmigrated-inline-wiki-markup"><ac:plain-text-body><![CDATA[

<ac:macro ac:name="unmigrated-inline-wiki-markup"><ac:plain-text-body><![CDATA[

Zend Framework: Zend_Db_Mapper Component Proposal

Proposed Component Name Zend_Db_Mapper
Developer Notes http://framework.zend.com/wiki/display/ZFDEV/Zend_Db_Mapper
Proposers Benjamin Eberlei
Zend Liaison TBD
Revision 1.0 - 25 January 2009: Initial Draft.
2.0 - 01 February 2009: Draft Ready for Recommendation (wiki revision: 41)

Table of Contents

1. Overview

Discontinuing Zend Entity

*As a note for everyone finding this proposal: Zend Entity will be discontinued in favour of Doctrine 1 and 2 integration into ZF, see http://www.mail-archive.com/fw-general@lists.zend.com/msg25412.html*

This component willl be an implementation of DataMapper pattern that tries to seperate Objects from their persistent separation to let you focus on the application more.

It will offer a huge range of features for handling all types of objects, versioning, relations, cascadading operations. Extension points are provided to make the component as flexible as possible. The data mapping will be based on a mapping syntax that has to be written out explicitly (or be autogenerated). Data Mapping is a crucial point for developing applications that are developed Domain Driven. This component will contrast the Table-Row-Data-Gateway pattern which is more useful for Transaction Script or Table Module patterns of the model. Its core feature is the focus on the Entity as datastructure in contrast to SQL as datastructure. DataMapper makes heavy use of LazyLoading for relations that the actual related objects with Proxies that act as if they were the real objects and only upon request load the required data from the database. This lazyloading can be done for single related entities, collections or for fields of an entity that contain huge data (BLOBs and Text fields). The lazyloading replacements make sure that every relation of an object can be traversed/accessed inside the application without thinking about if it was loaded before.

Why is a data mapper a good extension to ZF? Because Data mapper is an enterprise pattern, and gets really useful in domain driven design. Zend Framework is an enterprise application framework. Other data mapper implementations in PHP are ezComponents Persistent Object (which leans to much to SQL Tables imho rather than on a Entity Class as the central datatype) and the under development FLOW3 and Doctrine 2.0 Frameworks.

The proposed data mapper solution will offer a generic solution and the possibility to extend certain parts of the mapper to match your needs perfectly. It will offer more flexibility for applications with complex business logic where a Table-Row-Data-Gateway will reach its limits faster. Using the DataMapper for an application as simple as a blog is nice, but rather overhead. This solution shines when you have more complex stuff going on.

This proposal is a first step, a second might be a proposal for Zend_Entity_Repository which implements the repository and specification patterns to completly encapsualte domain logic from persistence. Zend_Repository might use adapter to offer access to Zend_Db_Table instances, Zend_Db_Mapper persistence sessions and even an adapter for In Memory Repositories which would significantly enhance Unit Testing of Domain Logic. It could also include adapters for another current propsal Zend_CouchDb. Implementing the repository on top of a functional mapper is peanouts though

Additional further support to enhance this component would be integration with a Database Schema Component and different Tooling providers.

Clarification of terms

Entity An entity is a class/object that encapsulates Domain Logic. It uses all object-oriented reference types like composition, aggregation and such to represent the state of business objects in memory. It has onlly little relation with the relational entity definition.

Data Mapper A class that maps entities into a relational database based on mapping defintions. It does that behind an API that hides the relational database. In short words: It maps the memory representation of objects into the database.

Entity Manager Single point of entry for calls to the underlying mapping persistence engine. It controls the entity definitions, unit-of-work, the mapping engine and it allows CRUD operations on those entities via load, save and delete methods. These methods accept SQL through the Zend_Db_Select Query object (only).

Collections Sets of related entities are saved in collections. These are ArrayAccess and Iterator implementations. Implementing those as objects rather than simple PHP arrays allows for Lazy Loading of collections behind the scenes and offer a way for the data mapper to recognize which related objects have changed, are new or were deleted in a session.

Repository A repository completly hides the underlying persistence from the business domain, it does not allow for sql or query objects. It offers access to objects via Criteria objects that specifiy which objects may be retrieved. An extended repository allows saving and deleting of objects. Criteria are transformed into the concrete selecting language, SQL in the case of relational databases.

Loader Based on the entity metadata definitions a loader implementation is selected for each entity to load the data from the database in the most efficient way.

Persister Inverted principle of the loader, based on the definition the persister knows how to persist an entitiy and its related entities.

Zend Package/Namespace Discussion

Some of the interfaces and implementations of this propsal are that generic, that an ORM based on for example CouchDb could also use them. The overlap is fairly small though and in general I think the use-case of migrating from CouchDb to a RDMS or the other way around is rather small.

For this proposal to be really useful in a generic context it should follow its primary terminology Entity for its frontline interfaces. The question is wheater separation into two namespaces "Zend_Entity" and "Zend_Db_Mapper" are required, or if everything goes into subpackages of Zend_Entity. Personally I would put it all under a new Zend_Entity namespace although that might be misleading.

What is definately required in my opinion is the definition of the following interfaces and subpackages in a new Zend_Entity package that other persistence layers might be using (to name a few CouchDB, XML Databases, Remote Webservice that manage models).

  • Zend_Entity_Manager_Interface
  • Zend_Entity_Interface
  • Zend_Entity_Collection_Interface
  • Zend_Entity_IdentityMap
  • Zend_Entity_Query_*
  • Zend_Entity_MetadataFactory_*
  • Zend_Entity_LazyLoad_*

All the stuff that currently resides inside "Zend_Entity_Mapper_*" is database specific.

2. References

3. Component Requirements, Constraints, and Acceptance Criteria

  • Zend_Db_Mapper is NOT a model proposal. It merely allows to save Entities (classes) persistent. See "Domain Model, Table Module or Transaction Script" Patterns in Fowler PEAA for a model.
  • Zend_Db_Mapper will put the Entity (A Record Class) as Datastructure into the focus, not the where and how it is saved for persistance.
    • An Entity is defined via a metadata syntax, by default a programmatic PHP implementation is shipped. Extension points for Annotations, XML or YAML configs are provided.
    • Metadata allows to specifiy: Property Names, Column Names, Property Types for example and many other details regarding the mapping.
    • Simple entity objects will be supported via an interface, which breaks encapsulation for PHP 5.2 objects
    • For PHP 5.3 usage a Reflection API will be implemented that directly sets properties (even private and protected ones).
  • Zend_Db_Mapper will lean to a Data Mapper like Hibernate, it will provide the most basic functionality but offer rich interfaces to extend it.
    • It will take a Hibernate Mapping like syntax as basis to profit from the rich experience.
    • It will have reasonable defaults, only deviations should be configured.
    • It will provide a generic solution for the DataMapper pattern based on generic per entity data-mappers.
    • It will provide full functionality to define relations between tables that are translated into object composition behind the scenes.
    • Collections and object composition will make heavy use of behind the scences lazy loading to encapsulate object creation and domain logic without hurting performance.
    • It will implement IdentityMap
    • It will not implement UnitOfWork so far. It can be added as a decorator to the Entity Manager Session with hooks into the event API at a later point.
    • It will provide a Query API for RDMS mappers that bases on Zend_Db_Select
    • It will provide a Query Object API that is sql-likeish but speaks in terms of the domain model (properties, entities).
  • Zend_Db_Mapper will provide extension points through modularity that allow to overwrite specific functionality of the mapper.
    • It will strictly give responsibility of query building to subcomponents of the entity definitions such that developers can overwrite behaviour themselves (This compares to Doctrine behaviours)
    • Collection and lazy load classes can be configured to deviate from their defaults.
    • An event API will publish events on all entities.

4. Dependencies on Other Framework Components

  • Zend_Db_Adapter_Abstract
  • Zend_Db_Select
  • Zend_Loader_PluginLoader
  • Zend_Loader_Autoloader_Resource

5. Theory of Operation

Steps of configuration:

  • For each object that is an Entity you have to create a definition file that describes the process of mapping this entity to the database.
  • Each object that is an Entity has to be implemented. An interface can be implemented that breaks up encapsulation of the entity for the datamapper.
  • The Entity Manager will have to be initialized to be aware of the definitions.

Steps of a session:

  • A new Entity Manager is initializing with a Database connection and the mapping definitions.
  • The Entity Manager is a single point of entry that encapsulates the Mapping and Loading of objects.
  • The Entity Manager delegates all calls to the underlying concrete/generic mappers and the Query API.
  • You can load Entities through the Entity Manager via Primary Key or a derived Zend_Db_Select object.
  • The Entity Manager return and accept Entity objects that are controlled a very simple entity interface
  • The Entity Manager delegates saving, selecting, updating and inserting.
  • When Entities are retrieved the Mapping Engine decides how relations and large fields should be handled in termns of LazyLoading.
  • Entities do not include any database related code (except hidden lazy loading mechanisms through an Inner/Outer Iterator schema).
  • Besides more or less transactionless work, the session allows access to a Unit Of Work that handles a complete transaction based on the users need.

Additional detail on my currenct concrete implementation proposal ideas:

1.) Concrete Type Mapper vs Generic Mapper: It will be allowed to overwrite the lowlevel mapper for a particular entity. By default all entities will use a powerful generic entity mapper, so that only when performance issues occour it should be relevant to write hard-coded sql by overwriting a mapper.

2.) The Entity Manager keeps track of all dependencies such as Db Adapter, the unit of work, identity map, entity definitions and the underying mappers. They are enforced to be "singletons" with small "s" inside that session. This allows to have a hand on memory management which is necessary with UoW and Identity Map.

3.) LazyLoading will primarily be implemented at the Collection level, which under circumstances can lead to the N+1 query problem. This can be prevented by using outer joins which might be generated when loading the root object. You could also define a "formula" field can also be used to inject subselect values into an object field.

4.) Cascaded saving, updating and deleting of objects will be supported by configuration, allowing to save an entity and automatically saving all or some of its related entities.

5.) The generic implementation of the mapper is possible by using the Visitor pattern with the definition objects of a table. Depending on the state and action the mapper injects all the related data into a visitor accept function of the definition property. That way complicated lazy loading, association and other stuff is encapsulated at the point where its information is stored. This pattern allows for the great flexibility for developers to build their own properties with special handling that only need to be attached to the table definition and work without changes on the session and mapper code.

6.) As for every tool that tries to make you think less about the database this mapper implementation could cause considerable Database overhead, when used wrong. Therefore the documentation should by default include a section about pitfalls, performance and best-practices for special cases. To investigate the mapper behaviour I would like you to propose some object designs that we have to test in their persistent form.

6. Milestones / Tasks

  • Milestone 1: Proposal (Done)
  • Milestone 2: Early Prototype and add more Use-Cases from it (Done)
  • Milestone 3: Reviews and Zend acceptance.
    • Milestone 3a: Integrate community feedback (lots of that is already done, more to come)
    • Milestone 3b: Integrate Zend feedback
  • Milestone 4: Beta Phase with Iterative Feature Enhancements, Testing, Documentation and Collection of Scenarios
  • Milestone 5: Release

7. Class Index

Why two namespaces? I discuss some requirements of that in the Component Overview section.

Core classes (Public API)

  • Zend_Entity_Manager_Interface
  • Zend_Entity_IdentityMap
  • Zend_Entity_Query_*
  • Zend_Entity_MetadataFactory_*
  • Zend_Entity_Manager_Interface

Entity and Collection classes

  • Zend_Entity_Interface
  • Zend_Entity_List (Simple list 1...n of related objects)
  • Zend_Entity_Map (Objects accessiable via map-key => object association)
  • Zend_Entity_Set (Unique objects)
  • Zend_Entity_Collection_Interface
  • Zend_Entity_LazyLoad_*

The namespace "Zend_Db_Mapper_" is currently "Zend_Entity_Mapper_*" in the svn and git repositories.

Database specific implementations of core interfaces

  • Zend_Db_Mapper_EntityManager
  • Zend_Db_Mapper
  • Zend_Db_Mapper_Select
  • Zend_Db_Mapper_Query_*

Loader and Persister classes

  • Zend_Db_Mapper_Persister_Interface
  • Zend_Db_Mapper_Loader_Interface
  • A bunch of implementations based on the requirements of the entity definition

Definition classes (Tooling Providers would greatly enhance work with this)

  • Zend_Db_Mapper_DefinitionMap
  • Zend_Db_Mapper_Definition_Entity (Core Definition)
  • Zend_Db_Mapper_Definition_Property (Simple column property mapped to entity field)
  • Zend_Db_Mapper_Definition_PrimaryKey (Required Id field of entity)
  • Zend_Db_Mapper_Definition_CompositeKey
  • Zend_Db_Mapper_Definition_Formula (Read only properties that are SQL formulas, for example group formulas)
  • Zend_Db_Mapper_Definition_Collection (Has Many collections of entities or value objects)
  • Zend_Db_Mapper_Definition_Join (Join a second table that holds properties of entity)
  • Zend_Db_Mapper_Definition_Component (Nested value object that is generated from table columns)
  • Zend_Db_Mapper_Definition_Discriminator (Property that decides which sub-class to instantiate)
  • Zend_Db_Mapper_Definition_Version (field to use for optimitistic locking)
  • Zend_Db_Mapper_Definition_Timestamp (Update timestamp on saving)
  • Zend_Db_Mapper_Definition_Date (Zend_Date Serializer)
  • Zend_Db_Mapper_Definition_DateTime (DateTime Serializer)
  • Zend_Db_Mapper_Definition_Id_Interface
  • Zend_Db_Mapper_Definition_Id_AutoIncrement
  • Zend_Db_Mapper_Definition_Id_Sequence
  • Zend_Db_Mapper_Definition_Relation_Interface
  • Zend_Db_Mapper_Definition_Relation_OneToMany
  • Zend_Db_Mapper_Definition_Relation_ManyToOne
  • Zend_Db_Mapper_Definition_Relation_OneToOne
  • Zend_Db_Mapper_Definition_Relation_ManyToMany

8. Use Cases

Scenarios (Integration Tests)

I have implemented two scenarios for integration testing already, which are listed in the SVN repository:

http://framework.zend.com/svn/framework/standard/branches/user/beberlei/ZendEntity/tests/Zend/Entity/IntegrationTest

These show lots of use-cases and how the component works.

9. Class Skeletons

]]></ac:plain-text-body></ac:macro>

]]></ac:plain-text-body></ac:macro>

Labels:
None
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.
  1. Jan 25, 2009

    <p>I think a data mapper implementation would be a very good addition to ZF, so I'm looking forward to seeing this one progress.</p>

    <p>Some initial feedback:</p>

    <p>By calling the component 'Zend_Db_Mapper' you are tying its usage to that of the database. I think it would be better to have the component as 'Zend_Mapper' (or Zend_Data_Mapper), and then have a Zend_Mapper_Db as your database implementation of this. This would allow other implementations such as using web services as a data source/store.</p>

    <p>I'm slightly confused by the name and purpose of the Zend_Db_Mapper_Session class. Is this naming inherited from Hibernate (which I'm not familiar with)? I think having 'session' in the name of anything not connected with Zend_Session could be confusing for users. It encourages (as in your example) variables such as $session for something that has nothing to do with what most would consider to be the 'session' in the context of a web application.</p>

    <p>This class appears to be just an instance of the data mapper itself, so why not have this functionality in the Zend_Db_Mapper class? Your first use case could then become something like this:</p>

    <ac:macro ac:name="code"><ac:plain-text-body><![CDATA[
    $dbAdapter = Zend_Db::factory(..);
    $dataMapper = Zend_Db_Mapper::factory($dbAdapter, array(
    'mapperClassPath' => array('My_Application_Mapper' => 'My/Application/Mapper')
    ));

    $customer = $dataMapper->findByKey(1, 'Customer');
    [...]
    ]]></ac:plain-text-body></ac:macro>

    <p>You might want to take a look at the data mapper implementation in Ruby if you've not already - <a class="external-link" href="http://datamapper.org">http://datamapper.org</a></p>

    <p>Also one nitpick - the protected vars in your example class definitions should start with an underscore.</p>

  2. Jan 25, 2009

    <p>I have also been exploring Data Mapper in ZF recently, and agree that a generic component could make a nice addition.</p>

    <p>My initial thought though, is that you are trying to push too much into this proposal. I think a generic Mapper that isn't tied to any persistence but allows for manual implementation of the load/save and populate functions would be a better start.<br />
    Then add different persistence (DB, session, web service, whatever...) options through either extension or adapters.</p>

  3. Jan 25, 2009

    <p>Hi Benjamin,</p>

    <p>I agree with Tim in abstracting the mapper from the DB. Let the mapper subclasses decide the type of the objects being created, using your definition interface for example.</p>

    <p>Your created objects would ideally know nothing about where the data is coming from. Why would you want your domain object to implement an interface? Your domain model should be able to focus on the business logic and let the data be managed by the mapper.</p>

    <p>I suppose you could integrate the existing db gateway classes or adapt them to your mapper so that the mapper could use them by default. I have used Zend_Db_Table_Rowset as collection, adding ArrayAccess methods with lazy loading capabilities. But you don't want to restrict the mapper to this type of collection. Your collection can come from a file, a feed, a service, etc...</p>

    <ac:macro ac:name="code"><ac:default-parameter>php</ac:default-parameter><ac:plain-text-body><![CDATA[
    class My_Application_Customer extends Zend_Db_Mapper_Abstract
    {
    protected function fetchDefinition()
    {
    $def = new Zend_Db_Mapper_Definition_Mapper_Record();
    [...]
    $def->setRecordClass("My_Db_Table_Row");
    // or
    $def->setRecordClass("My_Test_Mock_Row");
    $def->setCollectionClass("My_Db_Table_Rowset");
    // or
    $def->setCollectionClass("My_Test_Mock_Rowset");
    [...]
    ]]></ac:plain-text-body></ac:macro>

    <p>However Zend_Db classes would have to be reviewed in the unit of work context, so that the mapper session encapsulate their save method nicely.</p>

    <p>I would leave the user the option of having the domain objects separate from the framework and have perhaps in my application library the mappers, tables, services and collection classes extending Zend classes accordingly.</p>

    <p>Looking forward to see this implementation.</p>

  4. Jan 25, 2009

    <p>Like others have said, I also think that this is a great idea, but would prefer a more generic solution rather than one that was so explicitly tied to relational databases (athough, I think that might be where you're actually trying to go with this).<br class="atl-forced-newline" /></p>

    <p>You should check out the <a href="http://xyster.devweblog.org/">Xyster</a> project (a component library built on top of the ZF), as their <a href="http://xyster.devweblog.org/documentation/guide/xyster.orm.html">Xyster_Orm</a> component has very similar goals to this proposal. Here are a few more links:</p>
    <ul>
    <li> SVN: <a class="external-link" href="https://xyster.devweblog.org/svn/trunk/library/Xyster/">https://xyster.devweblog.org/svn/trunk/library/Xyster/</a></li>
    <li>An in-progress, from-scratch rewrite of Xyster_Orm: <a class="external-link" href="https://xyster.devweblog.org/svn/trunk/incubator/library/Xyster/Orm/">https://xyster.devweblog.org/svn/trunk/incubator/library/Xyster/Orm/</a></li>
    </ul>

  5. Jan 25, 2009

    <p>Too much but not generic enough? This doesnt match. <ac:emoticon ac:name="smile" /> In my opinion a data mapper belongs to the OO <-> Relational world. There is no sense in really "mapping" Webservices or XML, because they can serialize objects (almost) perfectly.</p>

    <p>@Tim: The data mapper has to be nested inside a repository pattern to become really useful. Which is why I am going to propose a Repository pattern, when this proposal is accepted. In this the data mapper will be ONE adapter that can be plugged in, webservices or CouchDb or XML whatever are other options. The mapper itsself will be database centric, but the Repositories offer the possibility to switch between different sources.</p>

    <p>Your idea about Zend_Db_Mapper is an option, but its not really supporting the notion that you have a session with your persistent objects which the mapper provides, (which is the Hibernate notion).</p>

    <p>@Brenton: Allowing only for manual load and save methods would make no sense, everyone could write that himself using Zend_Db_Select. What my proposal offers as value add to mapping yourself is the fact that you can rewire your objects easily if you feel the need in your domain layer.</p>

    <p>Its view is record centric. If you have to implement the Load and Save methods yourself the process becomes SQL centric again, and you don't refactor your domain objects often enough, because the methods have to be rewritten completely.</p>

    <p>@Marcello: If you allow the subclasses to decide on the persistence you can't really do references between objects anymore. Additionally when its all tied together, you can do highly optimized joins that fetch all One-To-One and Many-To-One relations with the data in one single select.</p>

    <p>A further negative point on subclasses that are separated from each other: In an OneToMany or ManyToMany reference relationship between different objects: How are you going to specify where the "relation" itsself is saved? There is no generic solution for this problem and you become stuck with a maintenance nightmare of solution.</p>

    <p>Using the Zend_Db_Table_Row and Zend_Db_Table_Rowset as base classes is a bad idea. They are tied to the database by default allowing for save() methods and stuff on them. The concept of a data mapper completly separates a Record Object as datasource from its persistence. That is why my record only has getState(), setState() methods which the mapper uses for creation and persistence.</p>

    <p>Because the Mapper is Record centric, you have to implement each record yourself implementing the interface. There cannot be a generic Zend class to extend, because the records are pure domain and business logic, where no abstract class would now what to offer.</p>

    <p>@Aaron: No i am aiming at databse only <ac:emoticon ac:name="smile" /> Your stuff looks good, i can't make a hold of many things but the concepts look like the ones I am trying to implement. I see many familiar interfaces <ac:emoticon ac:name="smile" /></p>

    1. Jan 25, 2009

      <p>@Ben: Just to clarify, I am in no way involved with the <a href="http://xyster.devweblog.org">Xyster</a> project. It was just something I found a while back that I thought might interest you.</p>

  6. Jan 26, 2009

    <p>IMHO, this is one of the most well-thought out ORM-related ZF proposals I've seen. Good job! I think you've taken a good approach to a difficult problem.</p>

    <p>With respect to naming, I agree with Tim's original comment that it might be a good idea to avoid the term "session", if possible. Also, the term "Entity" might be a bit better than "Record" (despite the DB-centric nature of this proposal), since this is all about how to map your business Entities to some persistent store.</p>

    <p>The definition builder API could be simplified by making all the "add*()" methods direct proxies to the constructors of their respective classes. This would have the side benefit of removing those hard class dependencies. To maintain flexibility, you could introduce interfaces for each of the definition types, and allow the builder methods to accept either an instance of the appropriate interface, or the normal constructor args of the default implementation.</p>

    <p>A have a few other questions about the design floating around in my head, but no time to verbalize them here now. Again, I like your overall approach, though.</p>

  7. Jan 26, 2009

    <p>Entities is indeed the better name. I will update the proposal accordingly, maybe i should also add a section with short term definitions.</p>

    <p>I had the greatest idea about the definition API. It should use the Zend_Loader_PluginLoader and be proxied by name through the add() Methods as you suggested. This would really simplify it and make it extremly easily extensible with selfmade properties that extend functionality of the basic ones.</p>

    1. Jan 26, 2009

      <p>additionally maybe it would be a wise idea to integrate the EJB 3.0 specification into this propsal since that one is exactly about Object-Relational Mapping via Data Mappers (and Hibernate is implementing it).</p>

  8. Jan 27, 2009

    <p>Since loading data into the Entity objects is just a piece of cake i guess one could offer 3 strategies to do so: Either setState/getState, or set/get Methods for each individual property, or inject into $objet->var which could either be public or _<em>get/</em>_set managed variables.</p>

    <p>Would that be a good idea on a per Entity basis?</p>

    1. Jan 30, 2009

      <p>This is a lot clearer now with the changed terminology. Thank you!</p>

      <p>I'm completely unfamiliar with the EJB spec, so I can't usefully comment on that.</p>

      <p>Personally, I very much dislike getter/setter methods (they're just terribly ugly to me). I'd say stick with your current (get|set)State API, or just write directly to public variables. Also, your generic Mapper won't be able to know all the correct method names for many different Entities, so that couldn't really be part of the Entity interface. (Doing something like iterating over all "set*" or "get*" methods would be bad idea for several reasons.)</p>

      <p>The second definition example ("Article") is using a completely different class and API, and it's not clear exactly how that code relates to the proposed API. I'd suggest either explaining the differences in detail, or removing it entirely, as it creates confusion.</p>

      1. Jan 30, 2009

        <p>well you could say that a property with name "myProperty" translates into getter and setter setMyProperty and getMyProperty by default.</p>

        <p>Article is using the same API as the Customer Definition. Its just that in the Article example the "Table" and "Class" properties are set through the constructor. Its following ZF standards in allowing both constructor injection of options and via setter methods.</p>

        <p>Since all elements always need a unique identifier the constructor API is:</p>

        <ac:macro ac:name="code"><ac:plain-text-body><![CDATA[
        public function __construct($name=null, array $options=array());
        ]]></ac:plain-text-body></ac:macro>

        <p>and in options any "key" translates to setKey($value);</p>

        1. Mar 08, 2009

          <p>I'd like to see the mapper defaulting to using getters and setters. If Zend_Entity_Interface is implemented, the mapper would use the methods provided by the interface. My main argument for this is that I don't want to have to update two places when my class interface changes: the mapper AND the getState() method. That would kind of defeat the Data Mapper implementation and turn it into an Active Record/Data Mapper hybrid. <ac:emoticon ac:name="smile" /></p>

          <p>Minor detail: you're still use the WW_ namespace in some examples <ac:emoticon ac:name="wink" /></p>

  9. Jan 27, 2009

    <p>Looks very promising. Do you have any code samples available, perhaps a repository?</p>

    1. Jan 28, 2009

      <p>hallo Matt,</p>

      <p>i do have some code, i am testing with the idea only though. Its under heavy refactoring and the only thing that is currently stable is the public API. I have overthrown the internal query engine several times already to cope with problems that appeared. Additionally Insert, Update and Delete is not working at all.</p>

      <p>What would you like to look at, how you are using it later and what actions trigger what queries? Or the internal code for Selecting and Persisting?</p>

  10. Feb 01, 2009

    <p>I have finished a prototype that is lacking functionality in definitions, loader and persister optimizations.</p>

    <p><a class="external-link" href="http://github.com/beberlei/data-mapper-proposal-prototype/tree/master">http://github.com/beberlei/data-mapper-proposal-prototype/tree/master</a></p>

    <p>There is a Scenario TestCase class that shows some functionality of the Entity Manager in the context of an example with rich entity definitions:</p>

    <p><a class="external-link" href="http://github.com/beberlei/data-mapper-proposal-prototype/tree/88cf07e2deadbd4f68065be1f879da08e3ecb216/tests/WW/Mapper/ScenarioData/Clinic">http://github.com/beberlei/data-mapper-proposal-prototype/tree/88cf07e2deadbd4f68065be1f879da08e3ecb216/tests/WW/Mapper/ScenarioData/Clinic</a></p>

    1. Feb 01, 2009

      <p>I added a large example now, that shows everything in combination and how the seperation of domain logic and database persistence is achieved. I hope this isn't to big to be grasped <ac:emoticon ac:name="smile" /> </p>

  11. Feb 02, 2009

    <p>I forgot to mention earlier, I like the addition of the Zend_Entity namespace. Those things don't really belong under Zend_Db.</p>

    <p>Are the Zend_Entity_(List|Set|Map) classes really necessary? Having "interface Zend_Entity_Collection extends ArrayAccess, Iterator, Countable" would seem to encompass all of those. Then you could have "Zend_Db_Mapper_LazyLoad_Collection implements Zend_Entity_Collection" as a concrete implementation that lazy loads elements on access.</p>

    <p>Another thing to consider is caching. It might be a good idea to think about how a caching object can be given to the entity manager. Since the entity manager handles the mappers' dependencies, it can also inject a cache object as an optional dependency, if present. As you mentioned, these kinds of mappers are prone to the N+1 query problem, and lazy loading can sometimes exasperate the issue. Caching would be a good way to mitigate some of that as an easy first step before going in an optimizing how the mappers build SQL.</p>

    1. Feb 02, 2009

      <p>i think the difference in collections is necessary. You have to allow the programmer of the domain logic to do fancy algorithms with the domain objects. It might be very necessary for this to distinguish between a map where the key => object pairs can be accessed, a list where you go from 0..n and a set where you make sure that one and only one entity of the same id is added. Also there might be use-cases where an ordered list or map is required.</p>

      <p>Caching is a must, i forgot to mention it here. There are some problems with it though that i encountered when building the prototype. This is probably an issue i have to discuss at length with others, since there are many possibilities and some stones in the way.</p>

      <p>The component is quite huge, so what i think would be best would be a lengthy development of to a stable version in the incubator. I would make sure that all the included functionality would be tested in depth, in every phase of development. This would ensure stable usage of a subset of functionality, while adding new ones piece by piece.</p>

      <p>By choosing the way of a loader and a persister interface on a per entity basis i think the proposal is very flexible to allow for this kind of development and people using the incubator implementation for their own tests and making suggestions about use-cases.</p>

      1. Feb 05, 2009

        <p>I see your point about supporting different algorithm types. However, those, by definition, would be application-domain issues. If an application writer needed a specialized collection for those purposes, he could easily extend the generic collection. Also, those kinds of specialized sets are somewhat "un-PHP"-like; we're used to having our associative arrays, and that could follow naturally into the Entity collection class. Thus, I imagine that most users (ok, maybe just me <ac:emoticon ac:name="wink" /> ) would want to simply use the Entity collection the same way we currently use arrays. Either way, this is just my $0.02; I'm not strongly opposed.</p>

      2. Jun 02, 2009

        <p>Caching should be pretty easy to add. The UnitOfWork handles CRUD functionality, so this one can easely manage (CRUD) the needed cache. </p>

        <p>I currently use a combination of collection objects with lazy loading, data mappers, and when needed identity maps within the data mappers with caching on identity maps en datamapper levels. The advantage of this double caching is that entities are cached seperatly and function results are cached based on entity id's. This way all entities are only cached a single time and used for all queries that need them. Also the identity maps have an internal (static) cache, so for every request the real cache is only called once for each entity. These are purely speed optimalizations but the proposal should keep in mind that optimalizations should still be possible, else the proposal makes no sense for enterprise applications.. and that's where we're all working on, right? <ac:emoticon ac:name="smile" /> (or atlease ZF/Zend is focusing on)</p>

        <p>+1, a version into the incubator would be great</p>

  12. Mar 08, 2009

    <p>Hi Benjamin!<br />
    I'm thrilled to see this proposal, solid components to support models is something I've missed in ZF.<br />
    If this proposal is turned into a component that would be truly great!</p>

    <p>Keep up the good work!</p>

  13. Mar 20, 2009

    fc

    <p>Very interesting Benjamin. I like FLOW3's approach, it's simple. Shouldn't Zend_Entity be part of the ZendX_Persistence package, like javax.persistence? And yes, a Repository would be nice.</p>

  14. Apr 12, 2009

    <p>Hi Benjamin - nice idea!</p>

    <p>If you're interested, there's a basic implementation of something similar in a book called Professional PHP5. It uses Generic Objects and Generic Object Collections, implements lazy loading and allows you to define callbacks to load data when required. It seems to be similar to what you're proposing; check it out, if you can.</p>

  15. Apr 12, 2009

    <p>Hi Benjamin,</p>

    <p>I really like this proposal in general. More sophisticated persistence tools are pretty much needed in the PHP world in general and specifically for the Zend Framework.</p>

    <ul class="alternate">
    <li>I really like the possibility to find by example in hibernate. Behind the scenes it's nothing more than a findByQuery() but with a focus the domain, not on the database. Basically you pass an blueprint object with just a few properties initialized and let the mapper do the work.</li>
    <li>I would not couple it so tight to Zend_Db. With document oriented databases becoming more popular, I guess there is a lot of room for them. And nevertheless it is easier to store objects in DODB systems, there is a lot of room for actual mapping.</li>
    <li>What about a Criteria object which is domain centric and not DB centric like Zend_Db_Select? As far as I can see, the mapping definitions should contain all the information to create Zend_Db_Select instances from Criteria objects.</li>
    <li>How do Specification objects fit into that picture? Also it is a huge performance burden to use them to find business objects, for a few hundred it might be a good enough solution, especially for rapid prototyping. Maybe we can have a findBySpecification() and a Specification interface simply requiring a isSatisfiedBy()-method to be implemented.</li>
    </ul>

    1. Apr 12, 2009

      <p>Hello Lars,</p>

      <p>My general plan is to build an ORM mapper with this solution, which is a big enough task anyways, considering the advanced features I want it to have.</p>

      <p>Then extend upon that in the future to implement a very generic Repository pattern, which might allow for Criteria, Specification and Query By Example searching of objects.</p>

      1. May 10, 2009

        <p>Thanks for your answer. However, what about the issue with Zend_Db_Mapper being strongly coupled to Zend_Db. I would again suggest not to do it.</p>

        1. May 11, 2009

          <p>Zend_Db_Select offers VERY convenient functionality to build queries. Additionally ::insert, ::update and ::delete functions of Zend_Db_Adapter_Abstract are also very convenient for a Mapper. It saves me alot of time in optimizing against database sql dialects, which i am no expert in.</p>

          <p>Coupling with Zend_Db is not that bad, since you can implement your own adapter easily that for example supports the old mysql_ functions, or a Doctrine, Propel, symonfy or any other Database connection.</p>

          1. Jun 11, 2009

            <p>A data mapper is for abstracting data access. The data is not strictly limited to being in a database.</p>

  16. Jun 11, 2009

    <p>It seems like the direction of this component is moving away from being strictly object-to-relational DB, which is good!</p>

    <p>As such, maybe a better top-level namespace for this would be Zend_DataMapper. All the Core classes could be renamed from Zend_Entity_* to Zend_DataMapper_<strong>. The Entity and Collection classes would be at home in Zend_DataMapper_Entity_</strong>. I also personally think that most of the classes under Zend_Db_Mapper would better belong under Zend_DataMapper. This would keep the implementations and interfaces under the same namespace. The DB-specific mapping definition classes could probably be put into Zend_DataMapper_Db_Definition_*.</p>

    <p>The Entity Manager is already not too far from being a Repository; making it depend on Zend_Db_Select doesn't seem like a good idea. Writing a basic search criteria specification class would not be too hard, and would make it that much easier to get it all the way to being a Repository.</p>

    1. Jun 12, 2009

      <p>I have to say again this proposal is outside of the scope of the repository pattern. It is an implementation of an object relational mapper. This task is really complex on its own and wrapping a Repository around it with Criteria that translate to Zend_Db_Select statements have to be written anyways to implement repository. This proposal only does the ORM stuff. Most people want to use Zend_Db_Select or Raw SQL in the end so it has to be supported for the relational database backends.</p>

      <p>A repository/criteria proposal might follow after it and wrap around the Entity Manager/Session.</p>

      <p>I really understand you all with the need for a generic repository implementation, i really want one too. But what besides Zend_Db_Table and the Mapper would you want to wrap in it for now? There is nothing currently <ac:emoticon ac:name="smile" /></p>

      <p>I opt for going step by step! If anyone wants to seriously help to implement the bigger picture feel free to drop in #zftalk.dev and poke me.</p>

      <p>The current proposal state is out of sync to the incubator prototype. I have to invest some time to get it up to speed and make it a bit shorter also to focus on the important points.</p>

  17. Jun 12, 2009

    <p>This looks very promising, and I think it would be a great addition to Zend Framework. Is there any chance that the ZF core team would contribute to this, speeding things up?</p>

  18. Jun 19, 2009

    <p>I'm glad to see that ZF is going to aid the development of applications using Domain-Driven Design.<br />
    Recently I've been using Doman-Driven Design (or rather what I learned after reading Doman-Driven Design Quickly from infoQ.com <ac:emoticon ac:name="smile" /> ) in a project and I stumbled in an endless series of problems regarding Entity storage making a real pain working with DDD on a Zend Framework project.</p>

    <p>By the way, I'd suggest putting the DataMapper, Entity, Repository,Collection and so forth part of a Domain namespace in such way:</p>

    <ul>
    <li>Zend_Domain_Entity</li>
    <li>Zend_Domain_Entity_Interface</li>
    <li>Zend_Domain_Entity_Manager_Interface</li>
    <li>Zend_Domain_Entity_Collection_Interface</li>
    <li>Zend_Domain_Entity_Repository_Interface</li>
    <li>Zend_Domain_Data_Mapper</li>
    <li>Zend_Domain_Data_Loader</li>
    <li>Zend_Domain_Data_Persister<br class="atl-forced-newline" /></li>
    </ul>

  19. Jun 20, 2009

    <p>Although I am a database designer at heart, through my work I have seen the way that programmers operate. When a complex program has a very tight deadlines, and performance is not a problem, programmers prefer to take the solution that is the quickest. As database design is not the main skill (rather a secondary skill) of programmers, they seem to prefer to solve problems at the programming language end. </p>

    <p>I think that this Zend_Db_Mapper Component Proposal would greatly help in such situations. I also agree with some of the above comments that this does not belong under Zend_Db. I would like to see this developed into an object persistence tool. There seems to be no reason for it to restrict persistence to only databases.</p>

    <p>I think that it should also support persisting entities to files and even MemCache systems.</p>

  20. Jul 11, 2009

    <p>I came up with a way to decouple the EntityManager interface from SQL and Zend_Db_Select. This is going along the lines of the JSR/JDO specification and adds two methods: "createNativeQuery" and "createQuery", which are either a Zend_Db_Select alike object or an object which allows queries based on object properties.</p>

    <p>Also the "select", "find" and "findOne()" methods from the manager interface should be removed and moved onto the Query objects.</p>

    <p>I'll post an update of this proposal soon, which hopefully gets much clearer (after dozen refactorings of the API).</p>

    <p>Thanks so far for the great feedback, i have been incorporating lots of stuff <ac:emoticon ac:name="smile" /></p>

  21. Jul 13, 2009

    <p>I like that "createNativeQuery" and "createQuery" idea; it sounds like that would take care of my Zend_Db_Select coupling concerns.</p>

    <p>Since there's now code both in ZF's SVN, and in your github, which one is the "authoritative" source? Which should I use if I want to help work on this code? BTW, the above link to SVN appears to be wrong; should maybe be: <a class="external-link" href="http://framework.zend.com/svn/framework/standard/branches/user/beberlei/Zend_Entity/">http://framework.zend.com/svn/framework/standard/branches/user/beberlei/Zend_Entity/</a> Note the underscore in "Zend_Entity".</p>

  22. Jul 28, 2009

    <ac:macro ac:name="note"><ac:parameter ac:name="title">Zend Framework Acceptance</ac:parameter><ac:rich-text-body>
    <p>The Zend Framework Team is pleased to accept this proposal for immediate development in the Standard Incubator.</p>

    <p>We ask that you follow the requirements below when developing the component:</p>

    <h4>Ensure that DataMapper is not coupled to Zend_Db</h4>

    <p>Allow for data mappers that map to non-RDBMS data sources, such as document databases, web service protocols, etc.</p>

    <p>I see two potential paths.</p>

    <p>First, and perhaps the more "pure" solution would be to create interfaces for Storage and Criteria that define the minimal functionality necessary for each. These would borrow ideas from Zend_Db_Adapter and Zend_Db_Select, respectively – insert, update, delete, select::from, select:;where, etc. The base implementations would wrap the Zend_Db components and implement these interfaces.</p>

    <p>The second potential solution is to document ways to create Zend_Db Adapters, Statements, and Select objects that proxy to web services or other classes. This would allow you to keep Zend_Db_Adapter as the standard Storage type and Zend_Db_Select as the standard Criteria object.</p>

    <p>Whichever way is chosen, there <strong>must</strong> be documented examples, so that developers have a path for using these alternate data sources.</p>

    <h4>Naming conventions</h4>

    <p>Please update the code to follow the new coding standards shipping with 1.9; in particular, pay attention to the naming of abstract classes and interfaces.</p>

    <p>Regarding the actual component names, I'd like to see the base interfaces and/or abstract classes for the Mapper subcomponent moved under Zend_Entity, including the DefinitionMap, with Zend_Db-specific implementaions then living under Zend_Db. This will better indicate that DataMappers need not be specific to RDBMS systems.</p>

    <h4>Please create an outline of work</h4>

    <p>Create a work outline indicating what tasks may be developed in parallel, and indicating such things as the interface requirements and desired concrete implementations. This will allow others to assist in the development.</p></ac:rich-text-body></ac:macro>

    1. Dec 19, 2009

      <p>As a note for everyone finding this proposal: Zend Entity will be discontinued in favour of Doctrine 1 and 2 integration into ZF</p>

      <p><a class="external-link" href="http://www.mail-archive.com/fw-general@lists.zend.com/msg25412.html">http://www.mail-archive.com/fw-general@lists.zend.com/msg25412.html</a></p>

  23. Aug 14, 2009

    <p>Great nice one, I look forward to seeing this develop. All we need now is some dependency injection....</p>

  24. Sep 07, 2009

    <p>One thing I think that could simplify some of this, is telling the back end where to put values, rather than ask for them and having to juggle them about. Especially when have to deal with joins and particularly self joins, as each set of fields can be bound to their own scope.</p>

  25. Oct 05, 2009

    <p>How feasible would it be to rename Zend_Entity_Interface getState and setState methods to something purposefully unambiguous? For example, an entity may have getState to refer to application logic such as a persons US-State code or a active registration for example. Perhaps getEntityState and setEntityState ?</p>

    1. Oct 05, 2009

      <p>Come to think of it, why not push the property export to another class altogether? This could be done as-is. For example, having a Blog class with the domain logic and a Blog_Entity class which accepts a Blog instance and implements getState and setState methods. Even a StdClass could be used within that design.</p>

      <p>The only advantage to having the actual entity implement the interface would be scope rules allowing the getState method to retrieve directly from protected or private members but that begs the question of whether getState should be retrieving otherwise protected values.</p>

      <p>The disadvantage being that code not developed for Zend_Entity would require extending the base classes and implementing the interface.</p>

      <p>Just a thought.</p>

      1. Oct 05, 2009

        <p>Hello Aaron,</p>

        <p>The Interface is not necessary anymore to enhance to "persistence ignorance", however using PHP 5.2 getState/setState are the default. It is possible to extend the specific state-transformer, an object that does just what you described, and change the set and get access methods.</p>

        <p>However your request to change to getEntityState()/setEntityState() is valid and I will think about changing it.</p>

        1. Dec 19, 2009

          <p>Ideally fetching and setting state is completely separate from the domain object. Since the latter is not possible with Reflection, currently the only method I am aware of is serializing and unserializing. Sebastian Bergmann has a working example of this with his "Object Freezer":</p>

          <p><a class="external-link" href="http://github.com/sebastianbergmann/php-object-freezer">http://github.com/sebastianbergmann/php-object-freezer</a></p>

  26. Oct 15, 2009

    <p>Can entities be persisted in different database tables / servers based on a condition (including a condition of entity itself)?</p>

    <p>For example, we have two "items" tables, one in a "global" database and another in a "site" database (each site has their own database). Users have access to the items that belong to their site along with global items. To help us differentiate between the two types of items, we have a rule where all global items have an id greater than 1,000,000.</p>

    <p>So, when finding or saving an item, the mapper needs to know which database to work with. Is this something that the mapper would be able to handle?</p>