Added by Rob Allen, last edited by Thomas Weidner on Nov 29, 2007  (view change)

Labels

 
(None)

Zend Framework: Zend_Db_Schema_Manager Component Proposal

Proposed Component Name Zend_Db_Schema_Manager
Developer Notes http://framework.zend.com/wiki/display/ZFDEV/Zend_Db_Schema_Manager
Proposers Rob Allen
Thomas Weidner
Revision 0.1 - 25 October 2006: Initial release.
0.2 - 30 November 2007: Additional features (wiki revision: 11)

Table of Contents

1. Overview

ACTUAL BEING REWORKED

Zend_Db_Schema_Manager maintains a database schema in a database agnostic format.
This ensures that the same change can be applied to multiple databases and all changes can be stored in
a version control system.
The schema itself can be given in several formats like XML, PHP. Additional input formats could be integrated.

2. References

3. Component Requirements, Constraints, and Acceptance Criteria

  • Zend_Db_Schema_Manager requires a database specific adapter for each supported database.

4. Dependencies on Other Framework Components

  • Zend_Exception
  • Zend_Db

5. Theory of Operation

Zend_Db_SchemaManager enables easy changes to a database schema without destroying
data. They can also be rolled back if required. The schema manager executes database
migration scripts which are collections of database operations such as adding a field to
a table.

Each migration script is a class extending from Zend_Db_Schema_Change which contains two
abstract methods: up() and down(). Migration script files are named like 001_initialSchema.php
so that they can be operated upon in order. The class within 001_initialSchema.php is called
initialSchema.

Note that for Zend_Db_Schema_Manager to work, a new table is required within the the database
being managed. This table, schema_info, holds the current version of the schema and is used
by the Manager to decide if the current operation is an upgrade or a rollback.

In order to work, a database independent structure to describe a table is used.

6. Milestones / Tasks

  • Milestone 1: Completed design on wiki
  • Milestone 2: Working prototype with unit tests checked into the incubator.
  • Milestone 3: Documentation exists

7. Class Index

  • Zend_Db_Schema_Exception - exception handler
  • Zend_Db_Schema_Manager - class to manage migrating from one schema version to another
  • Zend_Db_Schema_Change - abstract class to inherit each migrations script from
  • Zend_Db_Schema_Table - database independant table creation and altering
  • Zend_Db_Schema_Adapter - abstract class for database specific schema-change adapters
  • Zend_Db_Schema_Adapter_Mysql - MySQL adapter for adding/deleting/editing tables
  • Zend_Db_Schema_Adapter_Mssql - MSSQL adapter for adding/deleting/editing tables

8. Use Cases

UC-01

The main use case is when you need to change the database schema of a live application.
Obviously this is done on the development machine first and the Schema_Manager will
ensure that the same changes to the database will occur on the live site.

Example migration script: 001_AddSecondEmailAddressColumn.php

Example CLI script to run the Manager: updateSchema.php

9. Class Skeletons

Why not to send this proposal for review? It is very useful.

While I love this idea, I wonder if it's necessary to name files with revision numbers in front? It breaks the naming conventions because the classes can't be named properly, and it just looks weird.

Why not make revisionNum a property of Zend_Db_Schema_Change and then use that to check against the db table?

Other than that, I think it's a great idea and long-overdue for the PHP community.

To have revisionNum as a property of Zend_Db_Schema_Change would it make very unhandy to get the latest revision - You have to open every Schema Change File to know what the latest change was. In this case it`s also needed that Zend_Db_Schema_Manager parses and initiates all Zend_Db_Schema_Change Classes first to get the right order. Thats not a real problem but i think to name the files is a good convention for fast development.

But anyway - Is there any active work on this proposal? It is a must-have for web applications.

I have a few point from the perspective of one who has used rails db:migrations in the past. BTW, correct me if I'm wrong, but I see this as a relatively straightforward adaptation of migrations in Rails to PHP; it would help me immensely if you summarized the similarities and differences so I can see how we might differentiate ZF's solution.

1) I guess I just never saw the value in creating a whole abstraction around DDL statements for non-trivial schemas. I mean, yeah, they're portable to different databases, but then a lot of the simple ddl that you might cover with such a system is often portable in its regular string DDL format. That said, for a non-trivial schema in Rails, one generally (by necessity) disposes of this abstraction fairly quickly, which fortunately still allows the developer to take advantage of the versioning without necessarily using the built-in database agnosticism. At the very least, this proposal would have to include such a mechanism, since it would be nearly impossible to support all database DDL/features in the abstraction. In fact, these different feature sets make complete db agnosticism completely impossible, so I'm left wondering 'Why can't we just use DDL again?' I can see the value in simple cases, but is it enough to add a significant abstraction to the framework to expose?
2) I like database versioning, which I fully agree that we need to support in some way for framework. I haven't done a whole lot of complex versioning with db:migrations, but I'd want to be sure that it would work well even in less-than-straightforward circumstances. This versioning is the real value in db:migrations from my point of view.
3) This should definitely reuse [Zend_Build - Wil Sinclair].

Also, I believe Cake has a db:migrations port to PHP, and I don't see much point in redoing their work. I would challenge you to see where we can add value to these tools- especially in the context of non-trivial or legacy schemas.

Hope this all makes sense given my limited background on this proposal.

Thanks.
,Wil

Until now I did not have time to complete the rework of this proposal.
It will still take me some time to finish it, based on how much work I will have with the other proposals.

to 1)
DB Abstraction has it's benefits if you are porting an application from one server to another, or change the RDBMS, or even if you are importing a new version which has another schema. I have created my own in the past ( pre-ZF ), and it's very usefull when you have a working environment and want to update it.

Generally there are two set's of work...
1.) The DB-Layer itself... here documented as set of PHP files. This one I would like to have seperated in XML or any other text-based format. It simplifies the port between systems.
2.) The Update procedure... from version x to y or backwards... these are sets of works... could be textbased (limited) or other.

But I have not been ready to write down the whole thing.

to 2)
This is a must for update/downgrade. But the way should not be fixed but loosely coupled.

to 3)
I did not know this component for now nor did I see it end of november when I had some time for rethinking of this component. Is it already finished ?

to Cake)
Do we want to say "use Cake for database versioning", we don't have one ?
I will have to look what they have done before I can say what's the difference. But I think there is one. I am basically an enterprise application developer so my point of view is possibly not that strict as a normal web-dev. And I think especially this component has much value for enterprise applications.

Generally speaking:
This proposal has not been forgotten... there's just much work to do and to rethink before doing it.

Greetings
Thomas

I'd like to reserve judgment on the DDL abstraction layer until this proposal is fully fleshed out, so Thomas/Rob- the ball is fully in your court.
RE:Cake- What we really want to say is "Zend Framework has database versioning, and it is the only logical choice for you given your needs". My team at Zend has heard it a lot lately, and it's about time we started spreading the message to the whole community- we're looking for best-by-far solutions for a set of users that has not been adequately served by other frameworks in the very crowded field of PHP frameworks. The fact is, we can't be the best framework for every application, but I believe that we are the best framework for non-trivial apps in the Enterprise or other environments where "opinionated frameworks" just don't cut it and that we should continue building our components to further this lead. So, in that context, I would ask myself for any proposal- including this one- "Can we bring value to functional areas and more demanding users where other frameworks have not yet ventured or fallen short?"
I personally feel that RoR migrations are great in some ways but fall short in more complex cases. I can't wait to see how your proposal will address some of those shortcomings.

I think the primary purpose should not be that the application is portable to different databases. I think the benefit is that the schema-change-files are easy to maintain and seperate through different versions of your application. I can see this at my job, where we have every month a new release of our application. In most releases we do not have much DDL and less non-trivial DDL, but although we see that its not the best way to maintain the plain sql files through different versions. So i fully agree with Thomas Weidner.

Regarding to Zend_Build and this proposal. Perhaps it would be good to have different adapters for the DDL. Which allows to be more compatible to existing build tools.
Although adapters for converting or use of existing db-change files, for instance from phing (I know phing is using creole).

Cakes tools is php4.

kind regards,
Thomas (the other one) ;o)

Hmm... for some reason, the wiki has lost my watch on this page

I had two key reasons for writing this proposal, before all my time got sucked away:

1. Version control of database schema changes.
2. Database independence so that I can manage both MSSQL and MySQL schemas.
3. Bonus points for being able to use the same set of schema changes in both MSSQL and MySQL.

I still have these needs and I'm sorry that I haven't been able to push this forward as I had intended.

Regards,

Rob...

That's exactly what I have developed for my company.
But that's closed source and this was the reason why I wanted to help you.

My own time is also melted away... too much opened proposals or ideas to implement

I think after 1.5 I will have time to add my ideas to this proposal.
My ideas cover:

  • Database Versioning
  • Schema independence and portability
  • Schema checking and repairing
  • Integrating of default values
  • Adding of an portable schema format (xml) others are also possible
  • Support of multiple RDBMS through Zend_Db

With regards to the use of DDL, I imagine Enterprise Applications as such would require much finer tweaking of the database than an abstraction layer could provide. I struggled with Rails Migrations because I wanted to store large integers (billions) in my MySQL database, at the time there was no way of specifying BIGINT, so I had to use strings and cast in the model.

Versioning is very useful, I use phing and dbdeploy at the minute and find it very effective. The only problems I've had to overcome is managing the migrations between branches of the applications.

Hi there,

The last comments here were in Jan saying that people didn't want to forget this proposal...

May I offer my help in a general 'do lots of rubbish jobs' kind of way? My thought is that if someone other than you were to put some effort in it might get things rolling, i.e. inspire you.

I'm happy to do whatever...

Regards,
Carlton

This proposal is not forgotten...
Actually I have 6 proposals in queue which are worked on additionally some issues.

And I am forcing to integrate a coding standard with ralph which is in my eyes more important to be finished as it has effect to the complete framework.

Related to this proposal my idea was to have it more abstract.
I want it to be more generic so you can eighter use a sort of db description file which could be xml what I would prefer, or php what rob has proposed original.

What I also had in mind was a way to have default values integrated, and checked.
Also the db structure should be able to be checked and repaired.

I've already done such a class for my company but it's only mssql and it's closed source.

So what we want to do is a generic class which uses Zend_Db and is able to handle different layout input files.

Some of my other ideas which shall be integrated were already described by me in my previous comments.

Regards,
Thomas, I18N Team Leader

Hi Thomas,

"This proposal is not forgotten..." – I never thought for a moment that it was

My thought was just that if you and Rob are both busy, which given all you activity on ZF I imagine you are, it might be that there were tasks waiting to be done which weren't that hard but would still take time.

I am still beginning with ZF - I have only been using PHP for 2 years - but I want to help. So IF THERE IS such a task on this proposal, please let me know and I will be happy to put my thought into it.

In the meantime I will think on the proposal myself and perhaps start to come up with some test cases and the like.

All the best,
Carlton

+1
Very much looking forward to this!

I think a slightly different approach should be taken to where/how DDL is generated. I'd suggest placing a number of abstract methods into Zend_Db_Adapter_Abstract (createTable, dropTable, addColumn, etc.), and each database adapter should implement those methods.

If methods are the same across (some of) the adapters (such as dropping a table), they could even be implemented in Zend_Db_Adapter_Abstract and those that are different can override them.

All in all, I would really love to see this functionality in the Zend Framework.

Looks good! Can't wait to get my hands on it
Is there an implementation available somewhere?

You're not the only one.

Actually myself is working on the file transfer component and the new coding standards.
I had no time until now to complete this proposal.

It's far away from being ready for review.
There are necessary features missing and some are not defined complete.

For example:
There will be a sort of definition schema which will allow to define a db schema per xml or ini file. Also there will be a way of defining default values in tables.

These are just a few of what I've already done for my company where I created a DB Update manager which allows what Rob and I created here and a lot more.

So all I can say for now is:
... Please be patient and stay tuned ...