We have written a crawler based on code we found on the net. The code has run smoothly, however on numerous occassions it now hangs on certain pages - which have been changed and I guess it's the content of the pages that influences the behaviour. The script dies when the php max exec time is reached. An example page with which the statement "$index->addDocument($doc);" hangs is: http://www.febe.be/nl_BE/page/show/id/41

Please note that it seems most likely that the issue is not caused by the content of the page, because all other pages from that point onwards fail.

/**
 * SE siteCrawler v0.1
 * 
 * Generates a Lucene index per supported language
 * 
 * Uses:
 * - Zend_HTTP
 * - Zend_Search_Lucene
 * 
 * Todo:
 * - Currently deletes and rebuilds, should be dynamic
 * - Index pdf files & other
 * - Check for redirects
 * 
 * @author patrick @ studioemma.com
 */

header('Content-Type: text/html; charset=utf-8');
include("prepend.php");
set_time_limit(3600);

// Update these constants for correct usage.
define("_crawler_pathToLucenIndex",$application->getPath("Lucene"));
define("_crawler_indexName","FEBE");

require_once 'Zend/Search/Lucene.php';
require_once 'Zend/Http/Client.php';
require_once 'Zend/Log.php';
require_once 'Zend/Log/Writer/Stream.php';

foreach ($application->getLanguages() as $supportedLanguage) {

    
    $start_uri = $application->getUrl("Absolute").$supportedLanguage."/index/index2";
    $match_uri = $application->getUrl("Absolute").$supportedLanguage."/";
    
    $logFile = _crawler_pathToLucenIndex._crawler_indexName.".".$supportedLanguage.'.log';
    $indexFile = _crawler_pathToLucenIndex._crawler_indexName.".".$supportedLanguage.'.index';
    
    // Should be rebuilt, but currently we remove the index
    system("rm -rf $indexFile");
    system("rm -rf $logFile");
//die($start_uri);
    
    // Set up log
    //$log = new Zend_Log(new Zend_Log_Writer_Stream($logFile));
    ob_implicit_flush(true);
    $log = new Zend_Log(new Zend_Log_Writer_Stream('php://output'));
    $log->info('Crawler starting up');

    // Set up Zend_Http_Client
    $client = new Zend_Http_Client();
    $client->setConfig(array('timeout' => 30));


    // Open a Lucene index, or create it if it does not exist
    // Make it possible to create more than one index:
    $indexpath = $indexFile;

    try {
        $index = Zend_Search_Lucene::open($indexpath);
        $log->info("Opened existing index in $indexpath");
    // If can't open, try creating
    } catch (Zend_Search_Lucene_Exception $e) {
        try {
            $index = Zend_Search_Lucene::create($indexpath);
            $log->info("Created new index in $indexpath");
        // If both fail, give up and show error message
        } catch(Zend_Search_Lucene_Exception $e) {
            $log->err("Failed opening or creating index in $indexpath");
            $log->err($e->getMessage());
            echo "Unable to open or create index: {$e->getMessage()}";
            exit(1);
        }
    }

    // Set up the targets array
    $targets = array($start_uri);
    // Start iterating
    for($i = 0; $i < count($targets); $i++) {
        // Fetch content with HTTP Client
        $client->setUri($targets[$i]);
        $response = $client->request();
        if ($response->isSuccessful()) {

            // Possibly check for redirects, don't know if this is currently done automaticaly

            $body = $response->getBody();
            $log->info("Fetched " . strlen($body) . " bytes from {$targets[$i]}");

            // Create document
            $body_checksum = md5($body);
            try {
                $doc = Zend_Search_Lucene_Document_Html::loadHTML($body);
            }
            catch (Exception $e) {
                // It's possibly another document
                echo "

Another document