Zend Framework

Lucene doesn't work on 64-bit

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 1.6.0RC2
  • Fix Version/s: 1.8.4
  • Component/s: Zend_Search_Lucene
  • Labels:
    None

Description

I tried Zend_Search_Lucene on a 32 bit and all was fine. Trying on a 64 bit simply doesn't work.
The same index correctly opened on the 32 bit machine raises an exception on 64 bit.
Looking in the code I discovered that there's a problem with integer precision:
on line 422 of Zend/Search/Lucene.php the code:

$numField = $segmentsFile->readInt();

gives -1 on 32 bit, 4294967295 on 64 bit, reading the same index.

moreover, on line 425 the code

if ($numField != (int)0xFFFFFFFF)

fails, since 0xFFFFFFFF has a different value on 32 bit and 64 bit.
As workaround I changed numField to -1 and skipped the check and it works. So I think it's sufficent to change the test and the readInt function.

Issue Links

Activity

Hide
Piotr Gabryjeluk added a comment -

Hello guys,

I've just experienced this issue, quick-looked into the code and just made a patch for this. It's changing -1 to (int)0xFFFFFFFF in more places, like in other places it is done already.

The patch (looks like this is reverse patch, but you'll deal with it):

Index: library/Zend/Search/Lucene.php
===================================================================
— library/Zend/Search/Lucene.php (revision 13711)
+++ library/Zend/Search/Lucene.php (working copy)
@@ -413,7 +413,7 @@
if ($this->_formatVersion == self::FORMAT_2_3) {
$docStoreOffset = $segmentsFile->readInt();

  • if ($docStoreOffset != -1) {
    + if ($docStoreOffset != (int)0xFFFFFFFF) {
    $docStoreSegment = $segmentsFile->readString();
    $docStoreIsCompoundFile = $segmentsFile->readByte();

Index: library/Zend/Search/Lucene/Index/Writer.php
===================================================================
— library/Zend/Search/Lucene/Index/Writer.php (revision 13711)
+++ library/Zend/Search/Lucene/Index/Writer.php (working copy)
@@ -496,7 +496,7 @@
if ($srcFormat == Zend_Search_Lucene::FORMAT_2_3) {
$docStoreOffset = $segmentsFile->readInt();

  • if ($docStoreOffset != -1) {
    + if ($docStoreOffset != (int)0xFFFFFFFF) {
    $docStoreSegment = $segmentsFile->readString();
    $docStoreIsCompoundFile = $segmentsFile->readByte();
Show
Piotr Gabryjeluk added a comment - Hello guys, I've just experienced this issue, quick-looked into the code and just made a patch for this. It's changing -1 to (int)0xFFFFFFFF in more places, like in other places it is done already. The patch (looks like this is reverse patch, but you'll deal with it): Index: library/Zend/Search/Lucene.php =================================================================== — library/Zend/Search/Lucene.php (revision 13711) +++ library/Zend/Search/Lucene.php (working copy) @@ -413,7 +413,7 @@ if ($this->_formatVersion == self::FORMAT_2_3) { $docStoreOffset = $segmentsFile->readInt();
  • if ($docStoreOffset != -1) { + if ($docStoreOffset != (int)0xFFFFFFFF) { $docStoreSegment = $segmentsFile->readString(); $docStoreIsCompoundFile = $segmentsFile->readByte();
Index: library/Zend/Search/Lucene/Index/Writer.php =================================================================== — library/Zend/Search/Lucene/Index/Writer.php (revision 13711) +++ library/Zend/Search/Lucene/Index/Writer.php (working copy) @@ -496,7 +496,7 @@ if ($srcFormat == Zend_Search_Lucene::FORMAT_2_3) { $docStoreOffset = $segmentsFile->readInt();
  • if ($docStoreOffset != -1) { + if ($docStoreOffset != (int)0xFFFFFFFF) { $docStoreSegment = $segmentsFile->readString(); $docStoreIsCompoundFile = $segmentsFile->readByte();
Hide
Piotr Gabryjeluk added a comment -

This is patch to hopefully fix the Lucene addDocument on 64bit systems. It seems it is a reverse patch.

Show
Piotr Gabryjeluk added a comment - This is patch to hopefully fix the Lucene addDocument on 64bit systems. It seems it is a reverse patch.
Hide
Piotr Gabryjeluk added a comment -

Any information on this, ZF team?

Show
Piotr Gabryjeluk added a comment - Any information on this, ZF team?
Hide
Alexander Veremyev added a comment -

Fixed

Show
Alexander Veremyev added a comment - Fixed
Hide
Piotr Gabryjeluk added a comment -
Show
Piotr Gabryjeluk added a comment - This: http://framework.zend.com/code/viewrep/Zend_Framework/standard/trunk/library/Zend/Search/Lucene.php?r1=15946&r2=16167 seams to include what I suggested. I'll test it. Thanks!
Hide
John added a comment -

Hello guys, I need your help.

I've installed Drupal (http://drupal.org/), module Search Lucene API (http://drupal.org/project/luceneapi) with Zend Framework components (http://drupal.org/node/521604) and module Search Lucene DidYouMean (http://drupal.org/project/luceneapi_dym). First partial indexings were successful, but now "Index statistics" for Search Lucene DidYouMean is: "Number of documents in index: 13500; Number of terms: 90598", and I'm getting "Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 71 bytes) in /var/www/**/data/www/**/modules/luceneapi_dym/luceneapi_dym.index.inc on line 64" when I'm trying to perform indexing.
Replacing -1 with (int)0xFFFFFFFF, as you suggested, doesn't solve a problem. Applying fixes from http://framework.zend.com/code/viewrep/Zend_Framework/standard/trunk/library/Zend/Search/Lucene.php?r1=15946&r2=16167 makes search inoperative. Replacing library folder with one, extracted from http://framework.zend.com/download/latest, too.

Please upload to somewhere an archive, which I should to use instead of Zend Framework components (http://sourceforge.net/projects/luceneapi) to get things working. Or give me to know what I need to change in code. I'll be very grateful!

Show
John added a comment - Hello guys, I need your help. I've installed Drupal (http://drupal.org/), module Search Lucene API (http://drupal.org/project/luceneapi) with Zend Framework components (http://drupal.org/node/521604) and module Search Lucene DidYouMean (http://drupal.org/project/luceneapi_dym). First partial indexings were successful, but now "Index statistics" for Search Lucene DidYouMean is: "Number of documents in index: 13500; Number of terms: 90598", and I'm getting "Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 71 bytes) in /var/www/**/data/www/**/modules/luceneapi_dym/luceneapi_dym.index.inc on line 64" when I'm trying to perform indexing. Replacing -1 with (int)0xFFFFFFFF, as you suggested, doesn't solve a problem. Applying fixes from http://framework.zend.com/code/viewrep/Zend_Framework/standard/trunk/library/Zend/Search/Lucene.php?r1=15946&r2=16167 makes search inoperative. Replacing library folder with one, extracted from http://framework.zend.com/download/latest, too. Please upload to somewhere an archive, which I should to use instead of Zend Framework components (http://sourceforge.net/projects/luceneapi) to get things working. Or give me to know what I need to change in code. I'll be very grateful!

People

Vote (1)
Watch (3)

Dates

  • Created:
    Updated:
    Resolved: