ZF-4071: Lucene doesn't work on 64-bit

Description

I tried Zend_Search_Lucene on a 32 bit and all was fine. Trying on a 64 bit simply doesn't work. The same index correctly opened on the 32 bit machine raises an exception on 64 bit. Looking in the code I discovered that there's a problem with integer precision: on line 422 of Zend/Search/Lucene.php the code:

$numField = $segmentsFile->readInt();

gives -1 on 32 bit, 4294967295 on 64 bit, reading the same index.

moreover, on line 425 the code

if ($numField != (int)0xFFFFFFFF)

fails, since 0xFFFFFFFF has a different value on 32 bit and 64 bit. As workaround I changed numField to -1 and skipped the check and it works. So I think it's sufficent to change the test and the readInt function.

Comments

Hello guys,

I've just experienced this issue, quick-looked into the code and just made a patch for this. It's changing -1 to (int)0xFFFFFFFF in more places, like in other places it is done already.

The patch (looks like this is reverse patch, but you'll deal with it):

Index: library/Zend/Search/Lucene.php

--- library/Zend/Search/Lucene.php (revision 13711) +++ library/Zend/Search/Lucene.php (working copy) @@ -413,7 +413,7 @@ if ($this->_formatVersion == self::FORMAT_2_3) { $docStoreOffset = $segmentsFile->readInt();

  • if ($docStoreOffset != -1) {
  • if ($docStoreOffset != (int)0xFFFFFFFF) { $docStoreSegment = $segmentsFile->readString(); $docStoreIsCompoundFile = $segmentsFile->readByte();

Index: library/Zend/Search/Lucene/Index/Writer.php

--- library/Zend/Search/Lucene/Index/Writer.php (revision 13711) +++ library/Zend/Search/Lucene/Index/Writer.php (working copy) @@ -496,7 +496,7 @@ if ($srcFormat == Zend_Search_Lucene::FORMAT_2_3) { $docStoreOffset = $segmentsFile->readInt();

  • if ($docStoreOffset != -1) {
  • if ($docStoreOffset != (int)0xFFFFFFFF) { $docStoreSegment = $segmentsFile->readString(); $docStoreIsCompoundFile = $segmentsFile->readByte();

This is patch to hopefully fix the Lucene addDocument on 64bit systems. It seems it is a reverse patch.

Any information on this, ZF team?

Fixed

This: http://framework.zend.com/code/viewrep/… seams to include what I suggested.

I'll test it. Thanks!

Hello guys, I need your help.

I've installed Drupal (http://drupal.org/), module Search Lucene API (http://drupal.org/project/luceneapi) with Zend Framework components (http://drupal.org/node/521604) and module Search Lucene DidYouMean (http://drupal.org/project/luceneapi_dym). First partial indexings were successful, but now "Index statistics" for Search Lucene DidYouMean is: "Number of documents in index: 13500; Number of terms: 90598", and I'm getting "Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 71 bytes) in /var/www//data/www//modules/luceneapi_dym/luceneapi_dym.index.inc on line 64" when I'm trying to perform indexing. Replacing -1 with (int)0xFFFFFFFF, as you suggested, doesn't solve a problem. Applying fixes from http://framework.zend.com/code/viewrep/… makes search inoperative. Replacing library folder with one, extracted from http://framework.zend.com/download/latest, too.

Please upload to somewhere an archive, which I should to use instead of Zend Framework components (http://sourceforge.net/projects/luceneapi) to get things working. Or give me to know what I need to change in code. I'll be very grateful!