Added by Justin Plock, last edited by Alexander Veremyev on Mar 18, 2009  (view change)

Labels

 

Zend Framework: Zend_Service_Amazon_S3 Component Proposal

Proposed Component Name Zend_Service_Amazon_S3
Developer Notes http://framework.zend.com/wiki/display/ZFDEV/Zend_Service_Amazon_S3
Proposers Justin Plock
Alexander Veremyev (Zend liaison)
Stanislav Malyshev
Revision 1.0 - 10 March 2008: Initial revision.
1.1 - 31 December 2008: Revised code committed into laboratory SVN
2.0 - 25 February 2009: Revised the API and finalized the laboratory implementation (wiki revision: 14)

Table of Contents

1. Overview

Zend_Service_Amazon_S3 is an implementation of the service API and PHP user-stream wrapper for Amazon's Simple Storage Service (S3)

2. References

3. Component Requirements, Constraints, and Acceptance Criteria

  • This component will implement the Amazon S3 API
  • This component will use Zend_Http_Client to communicate with S3
  • This component will provide the API to use Amazon S3 directly
  • This component will implement a user-based stream wrapper for S3 (s3://)
  • This component will allow using multiple S3 accounts

4. Dependencies on Other Framework Components

  • Zend_Http_Client

5. Theory of Operation

This component will provide Amazon S3 API allowing to store and retrieve data from the Amazon S3 service.

Also, this component will be utilized using the stream_wrapper_register() PHP method to register a new user-based stream interface to S3. This allows the user to utilize existing fread(), fwrite(), fclose(), mkdir(), rmdir(), and stat() methods to directly interface with S3 buckets and objects.

6. Milestones / Tasks

  • Milestone 1: [DONE] Design notes will be published here
  • Milestone 2: [DONE] Working prototype checked into the laboratory supporting use cases
  • Milestone 3: [DONE] Unit tests exist, work, and are checked into SVN.
  • Milestone 4: [DONE] Initial documentation exists.
  • Milestone 5: Proposal accepted and moved to incubator
  • Milestone 6: The component is accepted as part of ZF core

7. Class Index

  • Zend_Service_Amazon_S3
  • Zend_Service_Amazon_S3_Stream

8. Use Cases

9. Class Skeletons

At some point, I'd like to take out the kind of hacky self::$debug stuff and integrate with Zend_Log. I just found it useful to see how the stream was actually being called to help with building the methods. This class attempts to support GET Ranges as well because user-based streams limit to 8192 bytes being returned, even though it seemed Amazon was returning the full object size and ignoring my Range headers. Not too sure what was going on, but the class works regardless.

+1 (great work!)

Zend Comments

This proposal has been approved for laboratory development with the following suggestion:

  • Watch use of static members, underscores in variable names and class members, hardcoded behavior for mime-type detection based on filename, opportunity for generalized filesystem storage component (e.g., with Nirvanix and S3 as available backends), and integration with Zend_Log.

Once Zend_Mime becomes available, I'd like to replace my getMimeType() method with something from that. I've changed all of the variable names to be camelCase. I'm not quite sure how we would generalize this as a filesystem storage component considering it's intended usage is as a PHP stream wrapper. Unless a large portion of it is made static, it's sort of meant to be a standalone class. What are your thoughts? How do I migrate this class from the laboratory to the standard incubator? Thanks.

Hi,

first of all I'd like to say I appreciated your work on this component and I'll use it as soon as it made it to the core. Anyhow I would prefer an API close to the S3-SOAP-Interface because it would be much easier to port existing code to use Zend_Service_Amazon_S3 and I don't like to create aliases for already existing method names (like mkdir vs. putBucket). I can only hope you reconsider the interface of your class

I just refactored the stream class to Zend_Service_Amazon_S3_Stream which interfaces with a more standardized interface on Zend_Service_Amazon_S3.  It's missing some of the functionality that Amazon provides (like copying objects, interfacing with CloudFront, etc), but it should be a good base.

hey, EXCELLENT WORK. Why is this not on the zend framework ? Everybody needs this !

Posted by vcx at Nov 27, 2008 12:31

It's in the incubator. It is still missing test cases and documentation. I haven't had much time to contribute that yet.  I also haven't heard from Zend much in regards to what is the future of this class.  Thanks for the support though.

Hey, It's just bad to let this in the incubator. I've posted a few mails in the mailing list in order to get this project going. There are 2 proposals for an s3 component and dozens of messages arround from guys asking for this.
You've done an excellent work and it's been nearly a year since it's there lying in the incubator. It's not fair !

Posted by vcx at Dec 29, 2008 07:12

Tell me about it.  I wouldn't mind some help finishing up the component (documentation, unit tests, more refactoring, etc).  I just don't know who to talk to at Zend who can assist with granting more privileges on the laboratory SVN tree, but I'd be more than interested in having some help.  I've personally been really busy lately and also don't like seeing this component linger.

Thanks for the support.

Justing, just ping me if you need additional svn privilegies.

The component is in the Ready for Recommendation state now, so it should be additionally reviewed by Zend Team really soon.

Amazon S3 support is a great idea! I think this service is much needed, and people have been asking for it. But I have concerns for the proposed API.

I am a massive proponent for not creating OO interface for OO's sake, but I am also a massive proponent of consistent API's. To me, the primary interface for this service component should consistent with the rest of the framework. This means that there should be accessors & mutators (setters & getters), as well as verb'd action names that "do work".

I am all about using php's strengths, and I think a stream wrapper should definitely be part of this component, but not as the primary api.

I feel like the primary class Zend_Service_Amazon_S3 should look and feel like the other service classes with respect to the api, and stay true to the concepts exposed by the API:

Beyond that, I think its perfectly acceptable to create a stream wrapper class to expose this service as a custom stream:

As you can see, the stream wrapper class (Zend_Service_Amazon_S3_StreamWrapper) would be used to be the "proxy" Between the actual s3 object and php's stream internals.

I guess thats all for now

-ralph

Regarding streams - I like this idea, but it's not as easy to do as one would think. The problem is that stream_open doesn't pass you enough context to find your object. It may be worked around with some creative play with stream contexts, etc. maybe but I don't see any API that binds contexts to wrappers - though API implies it in some places but it doesn't specifically says you can create context that would always be linked to certain wrapper. Maybe with more research it's possible to find the right magic.

Hi Guys,

Congratulations to everyone for their work on this component so far. I'd have to say though that I agree with Ralph. I feel the stream wrapper is definitely important - in fact required for this component, but it should not be the primary API. Also as Ralph mentioned the $s3->registerAsStream method would allow for multiple accounts, which in itself may be fairly important for some users as well.

Looking at the repository it seems we still need to finish the tests?

Cheers,

Rob

I'd really like someone to take this over and finish it up.  I just feel bad delaying this even more due to my schedule.  Any takers?

Thanks.

-Justin

I think I could help with it.

Justin,

Ralph pinged me about helping with this. What is left to do?

Jon

tests and documentation mostly.

Thanks for helping out.

-Justin

Justin,

Will look at this weekend then.

Jon

Justin,

I have just created a proposal for Zend_Service_Amazon_Abstract so that all the Amazon Services can have the same key handling code.

After talking with a few people in the chat room i have moved this to a JIRA issue.

http://framework.zend.com/issues/browse/ZF-5791

Let me know what you think.

I have the unit test code done and checked into my sqs repo on gethub

http://github.com/sidhighwind/zend_service_amazon_sqs/tree/master

Jon

Posted by Jon Whitcraft at Feb 15, 2009 06:56 Updated by Jon Whitcraft

It would be nice, to add method like this:

getQueryString($object, $expire);

to return url like this:
''http://bucket.s3.amazonaws.com/object?AWSAccessKeyId=44CF9590006BF252F707&Expires=1177363698&Signature=vjbyPxybdZaNmGa%2ByT272YEAiv4%3D''

or

''http://s3.amazonaws.com/bucket/object?AWSAccessKeyId=44CF9590006BF252F707&Expires=1177363698&Signature=vjbyPxybdZaNmGa%2ByT272YEAiv4%3D''

link to doc:
http://docs.amazonwebservices.com/AmazonS3/latest/S3_QSAuth.html

pch

I have problem with upload big files to s3. Maybe good idea will be add method to do POST request?

putObject is using PUT, which should be working fine.

Yes, it is using PUT, but for this object is read to memory. It's a problem for very large files.

Zend Comments

This proposal has been approved for standard library.