ZF-8911: wddx_serialize and utf-8 data will not work correctly

Issue Type:
Bug
Created:
2010-01-23T14:32:02.000+0000
Last Updated:
2010-04-02T06:27:38.000+0000
Status:
Closed
Fix version(s):
  • Next Major Release ()
Reporter:
Are Pedersen (are)
Assignee:
Marc Bennewitz (private) (mabe)
Tags:
  • Zend_Serializer
Related issues:
Attachments:

Description

The wddx_serialize function always encodes to utf-8...even when the data is already utf-8. Therefore you must decode to iso-8859-1 before wddx_serialize, or else your data gets "double" utf-8 encoded. This means wddx_serialize can't really serialize utf-8 encoded strings. If this is a PHP bug I don't know...but ZF should be aware of this and maybe implement this workaround.

Try wddx_serialize with a string in utf-8 encoding containing some special characters that are not in the standard english alphabet. The resulting wddx xml document will not be correct.

Comments

Posted by Marc Bennewitz (private) (mabe) on 2010-01-23T17:51:00.000+0000

I tested it with the following test but it pass:


    public function testSerializeStringUtf8() {
        $value    = "\xce\x9e"; // Ξ
        $expected = ''
                  . "\xce\x9e";
        $data = $this->_adapter->serialize($value);
        $this->assertEquals($expected, $data);
    }

Please tell me your php version. Can you create a test that failes please.

Posted by Are Pedersen (are) on 2010-01-24T08:36:16.000+0000

I tested a bit more and it seems that it works fine on PHP 5.2.9 but fails on PHP 5.2.6 However it's on two different servers, so it could still be other things than PHP. I can try to upgrade the server that runs 5.2.6 and see if resolves the issue.

I used the string "\xc2\xbf" to test with, it is a reverse questionmark.

Posted by Are Pedersen (are) on 2010-01-24T08:49:23.000+0000

it seems this is a bug in PHP that is already fixed. Sorry for the inconvenience. http://bugs.php.net/bug.php?id=46496

Maybe a comment in the ZF sourcecode would be helpfull for people still running PHP versions having this bug.

Posted by Marc Bennewitz (private) (mabe) on 2010-01-24T14:00:04.000+0000

Now I ran this test with different php version (5.2.4, 5.2.5, 5.2.6, 5.2.8). It fails on all php versions < 5.2.8 (I think it was fixed within 5.2.7 - but this version isn't available for download)

Within PHP-Docs I found the node: Note: If you want to serialize non-ASCII characters you have to convert your data to UTF-8 first (see utf8_encode() and iconv()). But this doesn't work correctly for php < 5.2.7.

Posted by Marc Bennewitz (private) (mabe) on 2010-04-02T06:27:38.000+0000

Now I close this issue because it can't fixed for php < 5.2.8 without. -> ZF 2.0 will need php 5.3.x as minimum version where the problem was fixed in ext/wddx.