I created an index of approximately 1.58M records with 10,000 records per segment. Searching that index was yielding far fewer results than I expected. Upon calling Zend_Search_Lucene::count(), I got a number close to what I expected to be the segment size. I traced the issue to the following lines from Zend/Search/Lucene.php
140 // read counter
141 $segmentsFile->readInt();
142
143 $segments = $segmentsFile->readInt();
Adding lines to echo both of these values yielded that the first readInt() call was returning what I expected to be the number of segment files in the index, which the second call was simply returning 1, causing my search to be applied only to the first segment in the index.
See the modifications to this block as shown below. Outputting $segName and $segSize within the loop that follows this block and _docCount following that loop yielded the expected results after I applied these modifications.
140 // read counter
141 $segments = $segmentsFile->readInt();
142
143 $segmentsFile->readInt();
Code
------------
140 // read counter
141 $segmentsFile->readInt();
142
143 $segments = $segmentsFile->readInt();
------------
is correct.
First int is a segment name counter, second is a current number of segments in the index.
Bug was in a segment counter updating in case of several commit() operations within one script execution.
Fixed in SVN early.