Skip to end of metadata
Go to start of metadata

<ac:macro ac:name="unmigrated-inline-wiki-markup"><ac:plain-text-body><![CDATA[

<ac:macro ac:name="unmigrated-inline-wiki-markup"><ac:plain-text-body><![CDATA[

Zend Framework: Zend_Controller Testing Infrastructure Component Proposal

Proposed Component Name Zend_Controller Testing Infrastructure
Developer Notes http://framework.zend.com/wiki/display/ZFDEV/Zend_Controller Testing Infrastructure
Proposers Matthew Weier O'Phinney
Zend Liaison Ralph Schindler
Zend Liaison TBD
Revision 1.0 - 27 May 2008: Community Draft.
1.1 - 27 May 2008: Proposal draft for Zend review (wiki revision: 11)

Table of Contents

1. Overview

Since I originally undertook the MVC refactoring project in the fall of 2006, I've been touting the fact that you can test ZF MVC projects by utilizing the Request and Response objects; indeed, this is what I actually did to test the Front Controller and Dispatcher. However, there is not currently an easy way to do so; the default request and response objects make it difficult to easily and quickly setup tests, and the methods introduced into the front controller to make it testable are largely undocumented. The only resource developers currently have for this sort of testing is the unit tests themselves.

Testing against the response object is somewhat tedious; you have to be regex guru in many cases, and the testing itself includes many calls to the response object in order to grab the values to test against.

This proposal aims to simplify testing MVC applications by providing tools to allow assertions against the DOM structure of your response, specialized request/response objects that mimic the HTTP environment and also prevent spurious headers and content from being generated, and more.

2. References

3. Component Requirements, Constraints, and Acceptance Criteria

The following application areas should be testable:

  • Response status: HTTP response codes (assume 200, but several methods will set the response code in the response object)
  • Response headers: presence of headers and content of headers
  • Content:
    • what nodes are present, selected by DOM id and/or CSS class selectors
    • content of selected nodes, both straight string comparisons and by regex

4. Dependencies on Other Framework Components

  • Zend_Controller
  • Zend_Exception
  • PHPUnit

5. Theory of Operation

In most cases, you don't need to go into terribly fine-grained detail, but you need to verify that particular elements or types of elements are present in the response, as well as potentially match small segments of the content. The response status is generally desirable, and often you need to test where an action may redirect.

When selecting nodes in XML or XHTML, the "gold standard" is represented by Prototype's $$() function, and Dojo Toolkit's dojo.query() method. Each of these utilize CSS selectors to qualify the nodes to select; CSS selectors allow us to specify specific node types, node ids, as well as the values contained in the 'class' attribute of nodes; additionally, it has a concept of inheritance, to indicate parent/child relations. This kind of specificity can allow a developer to check that types of content are present, instead of the specific content – which may be variable, due to having unreliable sources (database, for instance).

Additionally, if a developer wishes to use straight XPath, they should be able to do so as well.

I'm proposing several items. First, a component for selecting against DOM artifacts:

  • Zend_Dom_Query

This class is named to indicate first, its function, and second, an affinity for dojo.query() (since $$ is a name that's hard to translate to PHP). This component would be utilized to parse response body content. It would allow querying via CSS selectors or XPath.

Second, a test case for controllers that extends PHPUnit_Extensions_Database_TestCase:

  • Zend_Test_PHPUnit_ControllerTestCase (extends PHPUnit_Extensions_Database_TestCase)

This test case would provide scaffolding for bootstrapping the MVC application being tested, methods for dispatching a request and retrieving the response, and a number of assertions for running against the request.

The assertions include:

  • assertSelect() (assert that nodes specified by CSS selectors exist)
  • assertNotSelect()
  • assertSelectContentContains() (assert that nodes found by CSS selector contain the text specified)
  • assertNotSelectContentContains()
  • assertSelectContentRegex() (assert that nodes found by CSS selector match the pattern specified)
  • assertNotSelectContentRegex()
  • assertSelectCount() (assert that exactly the number of nodes found by CSS selector are found)
  • assertNotSelectCount()
  • assertSelectCountMin() (assert that at least the specified number of nodes are found by CSS selector)
  • assertSelectCountMax() (assert that at most the specified number of nodes are found by CSS selector)
  • assertXpath() (assert that nodes specified by XPaths exist)
  • assertNotXpath()
  • assertXpathContentContains() (assert that nodes found by XPath contain the text specified)
  • assertNotXpathContentContains()
  • assertXpathContentRegex() (assert that nodes found by XPath match the pattern specified)
  • assertNotXpathContentRegex()
  • assertXpathCount() (assert that exactly the number of nodes found by XPath are found)
  • assertNotXpathCount()
  • assertXpathCountMin() (assert that at least the specified number of nodes are found by XPath)
  • assertXpathCountMax() (assert that at most the specified number of nodes are found by XPath)
  • assertResponseCode() (assert that the given response code was given)
  • assertNotResponseCode()
  • assertHeader() (assert that a given header type was provided)
  • assertNotHeader()
  • assertHeaderContains() (assert that a given header type was provided and contains the text specified)
  • assertNotHeaderContains()
  • assertHeaderRegex() (assert that a given header type was provided and matches the pattern specified)
  • assertNotHeaderRegex()
  • assertRedirect() (assert that the response is a redirect)
  • assertNotRedirect() (assert that the response is a redirect)
  • assertRedirectTo() (assert that the response redirects to the given url)
  • assertNotRedirectTo()
  • assertRedirectRegex() (assert that the response redirects to a url matching the given pattern)
  • assertNotRedirectRegex()
  • assert(Not)AttributeExists($domNode, $attr)
  • assert(Not)AttributeEquals($domNode, $attr, $value)
  • assertDataSetInsert($table)
  • assertDataSetDelete($table, $id)
  • assertDataSetUpdated($table, $id)

Finally, custom request/response objects would also be created to facilitate testing (in particular, adding accessors to set request data), with corresponding interfaces for the functionality required by testing. This latter would faciliate custom request/response objects provided by the developer.

  • Zend_Controller_Request_HttpTestCase
  • Zend_Controller_Response_HttpTestCase

bootstrap() Method

By default, checks if $bootstrapFile is set, and if so, includes that file.

Override the method to provide specific functionality.

If using a bootstrap file, the file should not dispatch. Additionally, users should be aware that custom request and response objects will be set in the front controller if none are set in the TestCase.

reset() Method

Reset's object state of front controller, and clears request/response objects.

setUp() Method

By default, calls reset() and bootstrap(). When overriding, should call parent::setUp().

dispatch($uri) Method

Sets $uri in request object, and then dispatches front controller.

Sets returnResponse() in front controller to ensure response is returned.

Sessions

May need to create a Zend_Session replacement that can be used within tests. Must be compatible with Zend_Session_Namespace. Will be an object store to allow you to check the contents of the session to see if data has been set.

New Assertions

assertSelect*() family

Basically, these operates similar to Prototype's $$() function; you pass them CSS selector notation, which allows for ids, classes, and inheritance. Based on the assertion type and arguments you pass, you assert different things:

  • assertSelect('#foo') - foo id exists
  • assertNotSelect('#foo') - foo id does not exist
  • assertSelectContentContains('#foo', 'bar') - content of foo contains 'bar'
  • assertNotSelectContentContains('#foo', 'bar') - content of foo does not contain 'bar'
  • assertSelectContentRegex('#foo', '/bar/i') - regex match on content of foo
  • assertNotSelectContentRegex('#foo', '/bar/i') - negative regex match on content of foo
  • assertSelectCountMin('td#foo>tr', 3) - at least three rows in the foo table
  • assertSelectCountMax('td#foo>tr', 3) - no more than three rows in the foo table

All assertions would accept an optional extra argument, a $message to return on failure.

assertXpath*() family

All assertSelect* assertions would have corresponding assertXpath assertions that accept an XPath expression as an argument.

assertResponse(), assertRedirect()

Tests various properties of the response:

  • assertResponseCode(200) - 200 OK response code
  • assertResponseCode(404) - assert page not found
  • assertNotResponseCode(404) - assert NOT page not found
  • assertResponseCode(500) - assert application error
  • assertHeader('X-App-Id') - X-App-Id header in response
  • assertNotHeader('X-App-Id') - X-App-Id header not in response
  • assertHeaderContains('X-App-Id', 'bar') - X-App-Id header contains 'bar'
  • assertHeaderRegex('X-App-Id', '/bar/i') - regex match on X-App-Id header
  • assertRedirect() - response is a redirect
  • assertNotRedirect() - response is not a redirect
  • assertRedirectTo('/foo') - response redirects to /foo
  • assertNotRedirectTo('/foo') - response does not redirect to /foo
  • assertRedirectRegex('|^/foo|') - regex match on response redirect
  • assertNotRedirectRegex('|^/foo|') - negative regex match on response redirect

6. Milestones / Tasks

  • Milestone 1: [DONE] Submit proposal for review
  • Milestone 2: Complete working tests and code for implementation
  • Milestone 3: Documentation, use cases, and tutorials

7. Class Index

  • Zend_Controller_Request_HttpTestCase
  • Zend_Controller_Response_HttpTestCase
  • Zend_Dom_Exception
  • Zend_Dom_Query
  • Zend_Dom_Query_Css2Xpath
  • Zend_Dom_Query_Result
  • Zend_Test_PHPUnit_ControllerTestCase
  • Zend_Test_PHPUnit_Constraint_Exception
  • Zend_Test_PHPUnit_Constraint_DomQuery
  • Zend_Test_PHPUnit_Constraint_Redirect
  • Zend_Test_PHPUnit_Constraint_ResponseHeader

8. Use Cases

UC-01

9. Class Skeletons

]]></ac:plain-text-body></ac:macro>

]]></ac:plain-text-body></ac:macro>

Labels:
None
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.
  1. May 27, 2008

    <p>I think that something like this would be incredibly useful!</p>

    <p>Regarding Zend_Dom_Query:</p>

    <p>You may want to take a look at <a href="http://code.google.com/p/phpquery/">phpQuery</a> (at least as a point of reference). It seems to use the same CSS --> XPath transformation that you want to do.</p>

    1. May 28, 2008

      <p>I've actually written most of the query functionality already, and our two approaches are quite different, which will be evident once I have the code in the repository.</p>

      <p>Thanks for the link, though!</p>

  2. May 27, 2008

    <p>This would be excellent.</p>

    <p>I have really struggled to get my tests operating properly using direct code calls so I resort to using CURL operations which are slow and require a database and mail server setup. The end result is it's very slow and I can't get code coverage reports.</p>

    <p>The main hurdles I have faced are headers, sessions, mail and database commands.</p>

    <p>Could this testing infrastructure include stubbing of Zend_Mail, Zend_Session, Zend_Response and Zend_Db in such a way that they would operate as normal to the application but be able to be queried by the test to ensure everything is correct. There may be other classes that need to be stubbed (such as services where you may want to test static responses), I'm not sure how easy it would be to create a framework for stubbing.</p>

    1. May 28, 2008

      <p>Stubbing Zend_Mail is not much of an issue, and is a good idea; it's pretty trivial to provide a 'testing' transport, and we do so already in the Zend Framework test suite. This is also true of most service classes, as you can provide a mock HTTP client adapter.</p>

      <p>The response object will already be stubbed, per the proposal.</p>

      <p>Stubbing Zend_Db is less necessary, as recent versions of PHPUnit include DBUnit-style assertions and scaffolding; we will be recommending usage of this for testing with databases.</p>

      <p>Finally, regarding Zend_Session, this is the one sticky point of the proposal. The current implementation makes it very difficult, if not impossible, to test against in unit tests. I will either provide an alternate implementation of it that is included directly by the testcase, or attempt to refactor Zend_Session in such a way that is backwards compatible yet allowing for stubbing.</p>

  3. May 29, 2008

    <p>This is a great idea. It appears to be focused on html/xhtml content being returned, are there plans to have something that could work with service (SOAP, REST, AMF, etc) or CLI responses? </p>

    <p>Also, the assert methods appear to be pretty generalized, would it be better to have separate asserts for the varying functionality? ie, assertSelect, assertSelectMatch, assertSelectCount, etc?</p>

    1. May 29, 2008

      <p>For this first pass, I'm primarily looking at XML/HTML response payloads; later iterations may look for other payload types. As such, SOAP ad REST payloads should be testable immediately.</p>

      <p>In doing the implementation, I've already decided on a more fine-grained approach to the assertions:</p>

      <ul>
      <li>assertSelect() (are there any nodes matching the selector)</li>
      <li>assertNotSelect()</li>
      <li>assertSelectContentContains() (are there any nodes matching the selector containing the content)</li>
      <li>assertNotSelectContentContains()</li>
      <li>assertSelectContentRegex() (are there any nodes matching the selector with contents matching the regex)</li>
      <li>assertNotSelectContentRegex()</li>
      <li>assertSelectCount() (are there X number of nodes matching the selector, exactly)</li>
      <li>assertNotSelectCount()</li>
      <li>assertSelectCountMin()</li>
      <li>assertSelectCountMax()</li>
      </ul>

      <p>I'll be updating the proposal later today to reflect this.</p>

  4. May 29, 2008

    <p>related to above: access to the view object from tests would enable more fine-grained testing. then I could write a test that certain Module_Controller::Action() methods assign certain pieces of data to the view, which is the controller action's only job.</p>

    <p>right now I build up a request and send it thru routing, dispatching, helper, plugin, and view code and then test the final response. it's useful testing, for sure, but if I want to write more granular unit-tests that cover only my controller's action code, having access to the view object after the controller action executes is necessary.</p>

    <p>but that might be out of scope for this proposal?</p>

    1. May 29, 2008

      <p>You can, of course, pull the view object and test against it separately. However, what you're suggesting, pulling just the object state immediately following dispatching a controller action, would be next to impossible to do without greatly modifying the MVC components. Not going to happen for this iteration.</p>

      <p>With the selectors, however, its easy to write assertions against your specific action view scripts – simply look for specific CSS selectors from those view scripts. You should be able to accomplish much, if not all, your testing using those.</p>

  5. May 30, 2008

    <p>I think the testing support will be more than welcome. Similer to Luke though, my main objective is object isolation. I should be able to write unit tests in some format for Models, Controllers and Views separately. The main reason being that when applying TDD to a Controller, I won't have a View to assert against.<br />
    Testing the View is unavoidably linked to functional testing of course, but doing controller level testing where a real database and view is required makes for a very slow and fragile test suite which doesn't support TDD/BDD that much. The main direction of functional testing, in my process at least, is as acceptance tests where fragility is limited since the test captures an actual client requirement not a development requirement.</p>

    <p>That all said, I understand the scope of the proposal is limited. With some luck hopefully it grows in depth for the next iteration.</p>

    <p>As to the proposal here, I just had a few comments.</p>

    <p>1. The proposal shows a dependency on PHPUnit which is entirely understandable, but won't be everyone's preferred unit testing library. Would proposals falling into line with SimpleTest (I can't survive without Mock Objects <ac:emoticon ac:name="wink" />), for example, be acceptable?</p>

    <p>2. I won't repeat my entire first paragraph <ac:emoticon ac:name="wink" />. But worth highlighting somewhere the definition of functional testing for those unfamiliar and just getting started.</p>

    <p>3. Will it be possible to set custom error messages on assertions?</p>

    <p>4. It's not entirely clear from the use cases and examples whether XPath is integrated or not. It's mentioned earlier, but then later references only note CSS Selectors. It would be nice to have both since I'll never claim to be a great CSS person whereas I know XPath very well.</p>

    <p>5. You already updated for negative assertions so there goes my complaint about that optional "false" parameter <ac:emoticon ac:name="wink" />.</p>

    <p>6. There's no detailed mention of how Zend_PHPUnit_ControllerTestCase including a bootstrap file works. It includes the file, but will it do anything else? My own bootstrap files are classes, not procedural scripts. So would I, in theory, have to add a setup() override to call Bootstrap::prepare() for example, while skipping my dispatch method run()? You can see my Bootstrap example over on my blog. Would we need to follow a prescribed format of any kind?</p>

    1. May 30, 2008

      <p>Paddy – I was hoping you'd chime in!</p>

      <p>The intention of this proposal is, in fact, functional testing: does a given URL return the expected content? As such, it is more geared towards acceptance testing. I disagree that this is not in line with TDD/BDD, but that's an argument for another day.</p>

      <p>This test suite is intended to test primarily controllers and views; model unit tests would still be necessary, and typically can be written using normal testing strategies. </p>

      <p>Controllers and views, however, are often dependent on a variety of other processes that occur within the front controller dispatch cycle – plugins, helper registration, etc. – which make a functional testing approach likely the approach most likely to ensure the entire environment is prepared.</p>

      <p>Regarding your specific, annotated arguments:</p>

      <p>1. I target PHPUnit for two reasons: one, I'm most familiar with it, because two, it's the test suite used by the ZF project. It makes most sense to me to utilize the same testing framework we utilize in the project itself. I'm completely open to proposals for other testing frameworks, but these will likely need to be in the Extras repository and supported by community developers. The most important work, however, is in the Zend_Dom_Query classes, and this could easily be consumed by other test scaffolding.</p>

      <p>2. I'll make sure that a definition of functional testing is included in the documentation – thanks for pointing this out. <ac:emoticon ac:name="smile" /></p>

      <p>3. Yes, custom error messages will be possible, and you can see this in the current SVN tree (<a href="http://framework.zend.com/svn/framework/standard/branches/user/matthew/mvcTesting/library/Zend/PHPUnit/ControllerTestCase.php">http://framework.zend.com/svn/framework/standard/branches/user/matthew/mvcTesting/library/Zend/PHPUnit/ControllerTestCase.php</a>).</p>

      <p>4. XPath is not integrated currently; Zend_Dom_Query translates CSS selectors to XPath notation. I can definitely see a case for using straight XPath, however, and it would be fairly trivial to add a separate group of assertXpath*() assertions to support this. I'll definitely consider this for the final implementation.</p>

      <p>5. The original assertSelect() implementation was lifted from conversations I had with Mike Naberezny, but as I started working on the implementation, I realized it was incredibly unintuitive, and made providing custom error messages next to impossible. The new design is much cleaner. <ac:emoticon ac:name="smile" /></p>

      <p>6. bootstrap() will either include a bootstrap file via include() or call a callback; both are specified via the public $bootstrap property. So, you can either set that property in your class, or via the setUp() method. I'll make sure this is documented; in the meantime, you can check out how it works in the test suites I've already written (see the POC link in the References section). Using a callback, for instance, you could specify another method in the test suite itself, allowing the ability to encapsulate all other setup in one place, including calls to your custom Bootstrap class.</p>

      1. Jun 05, 2008

        <p>Think you pretty much busted up my points and left them floating in tiny pieces <ac:emoticon ac:name="wink" />. It's great to see the component is already ahead of me! Good job.</p>

  6. Jun 12, 2008

    <ac:macro ac:name="note"><ac:parameter ac:name="title">Zend Team Offical Recommendation</ac:parameter><ac:rich-text-body>
    <p>As is this propsoal is accepted for Standard Incubator development.</p>

    <p>At this time, we will reserve the opinion as per the location of some class files (such as Zend_Controller_Request_HttpTestCase and Zend_Controller_Response_HttpTestCase) until further development of this component is availabe in the incubator.</p>
    </ac:rich-text-body></ac:macro>

  7. Jul 25, 2008

    <p>What about PHPUnit_Extensions_Database_TestCase?</p>

    <p>In "Theory of operation" it's suggested as parent class of Zend_Test_PHPUnit_ControllerTestCase, but code sample here and in 1.6.0 RC1 use PHPUnit_Framework_TestCase as parent class. PHPUnit_Extensions_Database_TestCase is really usefull, and would provide easy-to-set-up database fixture generation infrastructure for ZF.</p>

    1. Jul 25, 2008

      <p>The author of the PHPUnit Database extensions is actually refactoring such that the assertions and database setup/population methods can be called via the base PHPUnit test case class as dynamic extensions – which will give us native capabilities. These should be ready for PHPUnit 3.3, iirc.</p>

  8. Sep 01, 2008

    <p>I'm probably a bit late with this since I only just started using it (and loving it already) - but shouldn't it be namespaced Zend_Test_PhpUnit instead of Zend_Test_PHPUnit in line with the <a href="http://framework.zend.com/manual/en/coding-standard.naming-conventions.html#coding-standard.naming-conventions.classes">coding standard</a>? (I'd even go as far as to suggest Zend_Test_Phpunit, but seeing as Zend_OpenId uses this form that would probably be most appropriate)</p>

    1. Sep 01, 2008

      <p>We have been debating whether or not it makes sense to have strict MixedCase conventions. Our own plugin loaders do not require it, nor does Zend_Loader need it to resolve class names to file names. Our feeling is that in some important areas, it actually leads to confusion when there is a mismatch between a known acronym/project and the class name. For this reason, we decided on retaining the original case of PHPUnit.</p>

      1. Sep 01, 2008

        <p>Does that mean that (for 2.0 I suppose) Zend_OpenId will become Zend_OpenID? And if you're not going to stick to this already, it might also make sense to remove it from the coding standard <ac:emoticon ac:name="smile" /></p>