Added by Ralph Schindler, last edited by Ralph Schindler on Aug 04, 2008  (view change)

Labels

 
(None)

Zend Framework: Zend_Tool_CodeGenerator Component Proposal

Proposed Component Name Zend_Tool_CodeGenerator
Developer Notes http://framework.zend.com/wiki/display/ZFDEV/Zend_Tool_CodeGenerator
Proposers Ralph Schindler
Zend Liaison TBD
Revision 1.0 - 2 June 2008: Initial Draft. (wiki revision: 4)

Table of Contents

1. Overview

Zend_Tool_CodeGenerator is a component that will facilitate the generation of new code based files as well as the modification of existing code based files.

2. References

3. Component Requirements, Constraints, and Acceptance Criteria

  • This component will use PHP's Reflection classes.

4. Dependencies on Other Framework Components

  • Zend_Reflection (proposed)
  • Zend_Exception

5. Theory of Operation

Goals

It is the goal of the Zend_Tool_CodeGenerator is to provide an object oriented set of abstracts and interfaces to facilitate the tasks related to and around the task of generating structured code. Currently in PHP, developers are able to (through Reflection) disassemble object oriented code into finite objects. Zend_Tool_CodeGenerator will attempt to take finite objects and construct and modify existing files (at the top most level) to produce completely working PHP files, classes, methods, properties, and docblocks.

Zend_Tool_Codegenerator will supply a set of interfaces and abstracts that will be able to be used to produce other flavors of "code", such as apache config files, php ini files, and potentially other languages such as javascript.

Zend_Tool_CodeGenerator_Abstract

Zend_Tool_CodeGenerator_Abstract is the simplest interface. Essentially, the only methods that are required are toString() and fromString(). This will ensure that any generators that use this component can read existing code and break it down into a finite object representation of the code, as well as take that object representation and write it out as code.

This interface also enforces the passing of options to the constructor that will allow for the generation of

Zend_Tool_CodeGenerator_Apache

TO BE COMPLETED AT A LATER DATE

Zend_Tool_CodeGenerator_Php

The Zend_Tool_CodeGenerator_Php sub-component will be responsible for generating PHP based files. At current the following language constructs that can be retrieved through reflection will be implemented

  • Classes
  • Methods/Functions
  • Properties
  • Parameters (for methods and functions)

The following elements will be able to be generated by the PHP CodeGenerator, but are not explicitly modeled in the PHP Reflection API:

  • Docblocks
    • File
    • Class
    • Method
    • Property
    • Required Files
  • Method Bodies
  • File

There exist the possibility that in the future, "body" elements/language constructs can be implemented such as: foreach loops, switch statements, for loops, etc.

6. Milestones / Tasks

Describe some intermediate state of this component in terms of design notes, additional material added to this page, and / code. Note any significant dependencies here, such as, "Milestone #3 can not be completed until feature Foo has been added to ZF component XYZ." Milestones will be required for acceptance of future proposals. They are not hard, and many times you will only need to think of the first three below.

  • Milestone 1: design notes will be published here
  • Milestone 2: Working prototype checked into the incubator supporting use cases #1, #2, ...
  • Milestone 3: Working prototype checked into the incubator supporting use cases #3 and #4.
  • Milestone 4: Unit tests exist, work, and are checked into SVN.
  • Milestone 5: Initial documentation exists.

If a milestone is already done, begin the description with "[DONE]", like this:

  • Milestone #: [DONE] Unit tests ...

7. Class Index

  • Zend_Tool_CodeGenerator_Apache
  • Zend_Tool_CodeGenerator_Apache_File
  • Zend_Tool_CodeGenerator_CodeGeneratorAbstract
  • Zend_Tool_CodeGenerator_Php
  • Zend_Tool_CodeGenerator_Php_Body
  • Zend_Tool_CodeGenerator_Php_Body
  • Zend_Tool_CodeGenerator_Php_Class
  • Zend_Tool_CodeGenerator_Php_Docblock
  • Zend_Tool_CodeGenerator_Php_Docblock_Class
  • Zend_Tool_CodeGenerator_Php_Docblock_DocblockAbstract
  • Zend_Tool_CodeGenerator_Php_Docblock_File
  • Zend_Tool_CodeGenerator_Php_Docblock_Method
  • Zend_Tool_CodeGenerator_Php_Docblock_Property
  • Zend_Tool_CodeGenerator_Php_File
  • Zend_Tool_CodeGenerator_Php_MemberAbstract
  • Zend_Tool_CodeGenerator_Php_Method
  • Zend_Tool_CodeGenerator_Php_Parameter
  • Zend_Tool_CodeGenerator_Php_Property

8. Use Cases

UC-01

UC-02

9. Class Skeletons

Hi Ralph, I really like this work, just saw the webinar and I see tons of potential in this component, I'm worry about something, I've been playing around with code generation for a while and I've found the best way you can generate anything is by using templates, I see you are using procedural code for generate stuff, it does the job when you need to generate something which is not big, but when things go big, to find a line where you need to make a change can be really messy, have you consider to implement something like Zend_View as a templating system for Zend_Tool?

I think it could be better have a template like:

well, obviously it could vary some, but IMHO, it is easier to read than writing code inside a method, btw, you said in the webinar django and after checking it out, it uses templates, but it has a 'flaw', which I've already experienced, I used to create my templates using some xml tags for common operations like, foreach, print, etc, but templates went bigger and bigger, to use Zend_View as templating system is to use the very same PHP as template language, which is what I'm using right now for templating (not Zend_View, PHP).

Hope it helps,

Aldemar

Posted by Aldemar Bernal at Jun 21, 2008 00:20 Updated by Aldemar Bernal

Specifics:

  • Putting this in the Zend_Tool namespace indicates that it will execute in the tooling execution environment. This is an unnecessary limitation. If we decide to implement Dojo integration that provided client-side validation, for example, we might reuse this code to generate the appropriate JavaScript. In that case, it would be awkward if this code resided in this package. I suggest Zend_CodeGenerator- or even Zend_Generate or Zend_Generator, depending on whether you like verbs or nouns in your namespaces. This will also reduce namespace bloat.
  • I like the simple Zend_Tool_CodeGenerator_Abstract interface, but have a few reservations. First, I'm not sure there's enough here to justify an abstract class. Plus, the options functionality is peripheral to the main abstraction here, and it could be argued that this form of options handling is not ideal for some systems; maybe an interface would be better with an abstract base class for the code generators that we provide as part of this proposal. Secondly, I'd consider method names like 'generate()' and 'reflect()', or maybe 'serialize()' and 'deserialize()' to avoid name collision with the very semantically fixed __toString() method.
  • I'd like to see the Apace config file generation soon, since it is a very different type of file from the php files that you go in to detail on. Also consider naming it 'ApacheConfig' or something similar.
  • What does "It will take cue from Reflection in what parts and bits of can become represented in object oriented notation" mean?
  • How would this class be instantiated if you wanted to modify an already existing class? Will it work with Zend_Loader? Will (can?) it work for plugins?
  • I agree with Aldemar about the templates and why we'd need them, and I think it's interesting that he reused PHP as we had discussed in the past.

General:

  • Again with the confusing typos.
  • Nurse, we need use cases, stat.

Wil, Ralph,

in the case we use Zend_View to generate the code, maybe there wouldn't be needed to create a Zend_Generate, Zend_CodeGenerator, or equivalent since Zend_View and its render() method would be the generator itself, if you like we could have a conference with Ralph in order to show you how I use PHP to generate code and wow, I would really like to help you guys even with code in this proposal =D.

Aldemar

  • Putting this in the Zend_Tool namespace indicates that it will execute in the tooling execution environment. This is an unnecessary limitation. If we decide to implement Dojo integration that provided client-side validation, for example, we might reuse this code to generate the appropriate JavaScript. In that case, it would be awkward if this code resided in this package. I suggest Zend_CodeGenerator- or even Zend_Generate or Zend_Generator, depending on whether you like verbs or nouns in your namespaces. This will also reduce namespace bloat.

This is gonna be a continuing problem if we don't resolve our perspectives. In my perspective, ZF is the package, Zend_Tool is just a namespace describing the primary realm of the components found within it. This means that the tasks related to "Code Generation" live most aptly in the "Tooling" space. That said, I agree, yes, this could be a top level component .. but then the next question is, is this another component that is flattening the top level a bit more, and is it worth it? I could go either way, but as I mentioned before: Code Generation is a tooling task, as you are generating server code.

Actually, I am not quite following what you mean by generating Javascript. Thats an interesting use case, but I would need you to expand on it as I dont completely see it. If we allow for client side language generation, why stop there? Why not move all Zend_View Helpers into Zend-Generator as well since they generate HTML? Please expand on the JS example.

  • I like the simple Zend_Tool_CodeGenerator_Abstract interface, but have a few reservations. First, I'm not sure there's enough here to justify an abstract class. Plus, the options functionality is peripheral to the main abstraction here, and it could be argued that this form of options handling is not ideal for some systems; maybe an interface would be better with an abstract base class for the code generators that we provide as part of this proposal. Secondly, I'd consider method names like 'generate()' and 'reflect()', or maybe 'serialize()' and 'deserialize()' to avoid name collision with the very semantically fixed __toString() method.

I completely agree here. I am revising the interfaces to reflect this.. ANother reason is that fromString() implies that a parser of some sort will tokenize the steam of PHP which definintely will not happen. reflect() is a better method and ill explore it for getting stuff into the code generator. And yes, generate() is a much better method for this class.

  • I'd like to see the Apace config file generation soon, since it is a very different type of file from the php files that you go in to detail on. Also consider naming it 'ApacheConfig' or something similar.

On it.

  • What does "It will take cue from Reflection in what parts and bits of can become represented in object oriented notation" mean?

Removed, will update with better verbage when the interfaces are updated.

  • How would this class be instantiated if you wanted to modify an already existing class? Will it work with Zend_Loader? Will (can?) it work for plugins?

What are Plugins in this sense? Basically, the idea is that anything that can be reflected by Zend_Reflection is a candidate for modifiation via Zend_Tool_CodeGenerator.

  • I agree with Aldemar about the templates and why we'd need them, and I think it's interesting that he reused PHP as we had discussed in the past.

I agree completely that we should perhaps include Zend_View here. The only problem with allowing usage of Zend_View is subsequent modification of files created with Zend_View. Will the syntax be Reflectable, will the Reflection-based modification resemble the original Zend_View based template.

All in all, Zend_View (and php based template approach) solves a very specific problem that we currently have.. And that is producing code at the language construct .. variable level. So yes, I am exploring how best to include Zend_View for templating.

  • Nurse, we need use cases, stat.

They are coming within the day or two.

I agree with Ralph on the Zend_Tool namespace meaning. The namespace contains tools to aid with project development and provides tools that applications can use themselves. I do think it is important to define a few aspects of code contained in Zend_Tool:

  • It is typically focused on functionality, not performance
  • It should not use the project config/context but rather have it supplied explicitly if called from the application (See my comment on Zend_Tool_Rpc proposal for more details)
  • It should not be executed by default in production environments but be limited to debug/development activities as well as specific uses such as generating code for dojo. In this case the result should be cached and the generation should only happen if required.

I love this component (along with Zend_Reflection) and believe it is a critical component for furthering meta-driven development. My development work focuses on building frameworks that are meta-driven and extensible. Most frameworks define an architecture which is abstracted to admin tools to different extents but require the developer to build business logic and other application-specific code by hand. Some frameworks can generate scaffolding code to get you started fast, but once the code is modified it falls outside of the framework's ability to update it. On one hand you have frameworks that provide a lot of end-user functionality out of the box but typically strap the developer to an architecture that does not lend itself well for extensive customization. One the other hand you have frameworks that provide components but require the developer to wire everything together cuasing lots of repetitive work. The challenge is to marry the two with the aim of reducing grunt work and at the same time maintain flexibility.

The only way I see this happening is through code reflection, parsing and generating. This involves several phases of abstraction and development:

  1. Be able to parse any file into an object structure and save it again
  2. Be able to isolate patterns in the object structure
  3. Be able to compare and map the object structure to defined standards
  4. Be able to manipulate the object structure in an intelligent way
  5. Be able to manipulate the object structure based on meta-data
  6. Be able to test the resulting file using unit testing driven by the mate-data

This will provide the required functionality to:

  • generate scaffolding
  • generate completely functional modules
  • enable the developer to modify code while still being able to maintain it automatically
  • provide detailed information to advanced IDE's that can write code-blocks based on meta programming
  • enable the developer to swap out sections of generated code for custom code that is faster, different etc... without breaking automatic maintainability of the entire module.

Generating code is easy compared to parsing and re-saving it with some changes without changing previous formatting. Who is to say we need to preserve formatting? Standards exist for a reason. They allow two or more people to easily work on the same code interchangeably. Thus we need standards for writing PHP code, .htaccess files, apache configs etc... It will enable the developer to stay in sync with the system and not get frustrated when things get reformatted.

To me code parsing is just as important as code generation. I would love to see the parsing functionality be built at the same time as the generation functionality to ensure a seamless flow. It would also make unit testing of generated code much easier.

I think this component is so important that it warrants a separate project with a C library that can be used by any programming language. We need to start somewhere and ZF is a great place, but I only see the concept reaching it's true potential if we can get support from other camps who can provide parsers and generators to and from a generic (XML?) format. They can at the same time define strict formatting conventions for their files which will help everyone.

The Zend_Tool_CodeGenerator name is great for the "generating" aspect but I think the parsing should be put into Zend_Tool_CodeParser and the two components should use the same underlying data structure for representing the code in tokenized form similar to DOM. If CodeGenerator is too specific we could call it something like StructuredParser and StructuredGenerator implying that it parses and generated data in a structured way based on standards.

The possibilities for such a system are endless and I would love to see ZF pioneer it for all structured data a typical PHP system encounters. Maybe a Zend_Structured namespace would be best as I see many uses of this functionality within applications. One application would be the parsing of the XML returned from Service calls via a chain of parsers and filters to generate desired object structures that can be used directly in other ZF components.