ZF-2470: If two long-running threads are doing multiple updates on the index at the same time, the index can be left in a corrupted/unusable.
E-mailed bug report:
I have found a lurking bug which is critical (applies to the most revision 7584 in trunk SVN): If two long-running threads are doing multiple updates on the index at the same time, the index can be left in a corrupted/unusable.
Specific situations I have found to cause index corruption: - two different long-running threads adding and/or deleting many different documents at the same time - calling the $index->optimize() command while another thread is adding and/or deleting documents.
An exception thrown: 'Zend_Search_Lucene_Exception' with message 'File '[some_index_file_in_the_data_directory]' is not readable.'... The index remains unusable/corrupted for all future requests.
I believe there to be a bug in how flock is being used somewhere in the framework code (possibly due to race conditions). You might suggest that my filesystem doesn't support the flock feature. I checked this out and it shouldn't be an issue. I am using this on a PHP's flock() function works as expected (testing on updated CentOS with most recent official Redhat backported security patched PHP 5.1.6).
It should be noted that certain aspects of the locking do work: - if a single thread is writing (updating) the index, and other threads are just reading (searching) the index, the writing/reading threads seem to block each other properly, without any problems.
I hope this problem isn't too difficult to fix. To you help reproduce it, using a fresh copy of the same test script I sent before (see attachment) do the following:
- on the command line, run the script once to create the initial index 2. once #1 is complete, modify the script and comment out line 24 which contains the call to create the index:
createInitialIndex($indexPath); (this is so that we don't re-create the index) 3. once #2 is complete, open up two (or more) console windows and run the script at the same time in each window
- after a few (or several seconds depending on your machine) an Exception should be thrown regarding missing index files