Indexing hangs endlessly

[ThomasLandauer]

I was hoping that #483 (comment) fixed it, but obviously it didn't :-(

Status: Indexing starts; when I change something in a file, it restarts. Sometimes it even (re)starts by itself; this goes on for hours...

[Gert-dev]

Could you post the output of the developer tools console? Don't forget to enable verbose mode there. There may be some useful output there.

[ThomasLandauer]

Before posting some endless output... ;-)
#483 (comment) mentioned the Atom version. I have 1.31.2 - cause that's what kubuntu gives me. Do you think this might be the reason?

[Nykilor]

I have the same problem I'm working on:

atom -v output

Atom : 1.43.0
Electron: 4.2.7
Chrome : 69.0.3497.128
Node : 10.11.0

Windows 10 1903 OS
php-ide-serenata 5.2.0

The message I get that is repeating itself

Received response message without id: Error is: 
{
    "code": -32000,
    "message": "Could not index file due to an internal exception. This likely means an exception should be caught at a deeper level (if it is acceptable) or there is a bug. The file is \"file://D:/xampp/htdocs/job-board-crawler/app/lib/JobBoardCrawler/DataProvider/PaginableWebsiteInterface.php\" and the exact exception message: \"Entity of type 'Serenata\\Indexing\\Structures\\File' for IDs id(5e2ed693e1d8a7.72926987) was not found\"",
    "data": {
        "line": 136,
        "file": "phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Indexing/StorageFileIndexer.php",
        "backtrace": "→ Message 1\nCould not index file due to an internal exception. This likely means an exception should be caught at a deeper level (if it is acceptable) or there is a bug. The file is \"file://D:/xampp/htdocs/job-board-crawler/app/lib/JobBoardCrawler/DataProvider/PaginableWebsiteInterface.php\" and the exact exception message: \"Entity of type 'Serenata\\Indexing\\Structures\\File' for IDs id(5e2ed693e1d8a7.72926987) was not found\"\n \n→ Location 1\nphar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Indexing/StorageFileIndexer.php:136\n \n→ Backtrace 1\n#0 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Indexing/TextDocumentContentRegistryUpdatingIndexer.php(39): Serenata\\Indexing\\StorageFileIndexer->index(Object(Serenata\\Utility\\TextDocumentItem))\n#1 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Indexing/UnmodifiedFileSkippingIndexer.php(51): Serenata\\Indexing\\TextDocumentContentRegistryUpdatingIndexer->index(Object(Serenata\\Utility\\TextDocumentItem))\n#2 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Indexing/Indexer.php(164): Serenata\\Indexing\\UnmodifiedFileSkippingIndexer->index(Object(Serenata\\Utility\\TextDocumentItem))\n#3 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Indexing/Indexer.php(107): Serenata\\Indexing\\Indexer->indexFile('file://D:/xampp...', Array, Array, true)\n#4 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/UserInterface/JsonRpcQueueItemHandler/IndexJsonRpcQueueItemHandler.php(59): Serenata\\Indexing\\Indexer->index('file://D:/xampp...', true, Object(Serenata\\Sockets\\JsonRpcConnectionHandler))\n#5 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/UserInterface/JsonRpcQueueItemHandler/IndexJsonRpcQueueItemHandler.php(48): Serenata\\UserInterface\\JsonRpcQueueItemHandler\\IndexJsonRpcQueueItemHandler->handle('file://D:/xampp...', Object(Serenata\\Sockets\\JsonRpcConnectionHandler))\n#6 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Sockets/JsonRpcQueueItemProcessor.php(135): Serenata\\UserInterface\\JsonRpcQueueItemHandler\\IndexJsonRpcQueueItemHandler->execute(Object(Serenata\\Sockets\\JsonRpcQueueItem))\n#7 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Sockets/JsonRpcQueueItemProcessor.php(75): Serenata\\Sockets\\JsonRpcQueueItemProcessor->handle(Object(Serenata\\Sockets\\JsonRpcQueueItem))\n#8 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/UserInterface/JsonRpcApplication.php(176): Serenata\\Sockets\\JsonRpcQueueItemProcessor->process(Object(Serenata\\Sockets\\JsonRpcQueueItem))\n#9 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/UserInterface/JsonRpcApplication.php(140): Serenata\\UserInterface\\JsonRpcApplication->processNextQueueItem()\n#10 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/vendor/react/event-loop/src/Timer/Timers.php(96): Serenata\\UserInterface\\JsonRpcApplication->Serenata\\UserInterface\\{closure}(Object(React\\EventLoop\\Timer\\Timer))\n#11 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/vendor/react/event-loop/src/StreamSelectLoop.php(184): React\\EventLoop\\Timer\\Timers->tick()\n#12 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/UserInterface/JsonRpcApplication.php(112): React\\EventLoop\\StreamSelectLoop->run()\n#13 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/vendor/symfony/console/Command/Command.php(253): Serenata\\UserInterface\\JsonRpcApplication->runEventLoop(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))\n#14 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/vendor/symfony/console/Application.php(1011): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))\n#15 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/vendor/symfony/console/Application.php(272): Symfony\\Component\\Console\\Application->doRunCommand(Object(Symfony\\Component\\Console\\Command\\Command), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))\n#16 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/vendor/symfony/console/Application.php(148): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))\n#17 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/UserInterface/JsonRpcApplication.php(63): Symfony\\Component\\Console\\Application->run()\n#18 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Main.php(24): Serenata\\UserInterface\\JsonRpcApplication->run()\n#19 C:\\Users\\PC-ADMIN\\AppData\\Roaming\\php-ide-serenata\\distribution.phar(10): include('phar://C:/Users...')\n#20 {main}\n \n→ Message 2\nEntity of type 'Serenata\\Indexing\\Structures\\File' for IDs id(5e2ed693e1d8a7.72926987) was not found\n \n→ Location 2\nphar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/vendor/doctrine/orm/lib/Doctrine/ORM/EntityNotFoundException.php:47\n \n→ Backtrace 2\n#0 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/vendor/doctrine/orm/lib/Doctrine/ORM/Proxy/ProxyFactory.php(166): Doctrine\\ORM\\EntityNotFoundException::fromClassNameAndIdentifier('Serenata\\\\Indexi...', Array)\n#1 R:\\appdata_temp\\__CG__SerenataIndexingStructuresFile.php(219): Doctrine\\ORM\\Proxy\\ProxyFactory->Doctrine\\ORM\\Proxy\\{closure}(Object(DoctrineProxies\\__CG__\\Serenata\\Indexing\\Structures\\File), 'getUri', Array)\n#2 R:\\appdata_temp\\__CG__SerenataIndexingStructuresFile.php(219): Closure->__invoke(Object(DoctrineProxies\\__CG__\\Serenata\\Indexing\\Structures\\File), 'getUri', Array)\n#3 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Analysis/Conversion/NamespaceConverter.php(22): DoctrineProxies\\__CG__\\Serenata\\Indexing\\Structures\\File->getUri()\n#4 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Analysis/DoctrineNamespaceListProvider.php(79): Serenata\\Analysis\\Conversion\\NamespaceConverter->convert(Object(Serenata\\Indexing\\Structures\\FileNamespace))\n#5 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Analysis/DoctrineNamespaceListProvider.php(50): Serenata\\Analysis\\DoctrineNamespaceListProvider->mapNamespaces(Array)\n#6 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Analysis/NamespaceListRegistry.php(91): Serenata\\Analysis\\DoctrineNamespaceListProvider->getAll()\n#7 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Analysis/NamespaceListRegistry.php(82): Serenata\\Analysis\\NamespaceListRegistry->initializeRegistry()\n#8 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Analysis/NamespaceListRegistry.php(41): Serenata\\Analysis\\NamespaceListRegistry->initializeRegistryIfNecessary()\n#9 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Mediating/NamespaceIndexingNamespaceRegistryMediator.php(57): Serenata\\Analysis\\NamespaceListRegistry->add(Array)\n#10 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/vendor/evenement/evenement/src/Evenement/EventEmitterTrait.php(123): Serenata\\Mediating\\NamespaceIndexingNamespaceRegistryMediator->Serenata\\Mediating\\{closure}(Object(Serenata\\Indexing\\Structures\\FileNamespace))\n#11 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Indexing/EventEmittingStorage.php(153): Serenata\\Indexing\\EventEmittingStorage->emit('namespaceUpdate...', Array)\n#12 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Indexing/EventEmittingStorage.php(123): Serenata\\Indexing\\EventEmittingStorage->dispatchScheduledEvents()\n#13 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Indexing/StorageFileIndexer.php(194): Serenata\\Indexing\\EventEmittingStorage->commitTransaction()\n#14 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Indexing/StorageFileIndexer.php(125): Serenata\\Indexing\\StorageFileIndexer->indexNamespacesWithUseStatements(Array, Object(Serenata\\Indexing\\Structures\\File), Object(Serenata\\Utility\\TextDocumentItem))\n#15 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Indexing/TextDocumentContentRegistryUpdatingIndexer.php(39): Serenata\\Indexing\\StorageFileIndexer->index(Object(Serenata\\Utility\\TextDocumentItem))\n#16 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Indexing/UnmodifiedFileSkippingIndexer.php(51): Serenata\\Indexing\\TextDocumentContentRegistryUpdatingIndexer->index(Object(Serenata\\Utility\\TextDocumentItem))\n#17 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Indexing/Indexer.php(164): Serenata\\Indexing\\UnmodifiedFileSkippingIndexer->index(Object(Serenata\\Utility\\TextDocumentItem))\n#18 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Indexing/Indexer.php(107): Serenata\\Indexing\\Indexer->indexFile('file://D:/xampp...', Array, Array, true)\n#19 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/UserInterface/JsonRpcQueueItemHandler/IndexJsonRpcQueueItemHandler.php(59): Serenata\\Indexing\\Indexer->index('file://D:/xampp...', true, Object(Serenata\\Sockets\\JsonRpcConnectionHandler))\n#20 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/UserInterface/JsonRpcQueueItemHandler/IndexJsonRpcQueueItemHandler.php(48): Serenata\\UserInterface\\JsonRpcQueueItemHandler\\IndexJsonRpcQueueItemHandler->handle('file://D:/xampp...', Object(Serenata\\Sockets\\JsonRpcConnectionHandler))\n#21 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Sockets/JsonRpcQueueItemProcessor.php(135): Serenata\\UserInterface\\JsonRpcQueueItemHandler\\IndexJsonRpcQueueItemHandler->execute(Object(Serenata\\Sockets\\JsonRpcQueueItem))\n#22 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Sockets/JsonRpcQueueItemProcessor.php(75): Serenata\\Sockets\\JsonRpcQueueItemProcessor->handle(Object(Serenata\\Sockets\\JsonRpcQueueItem))\n#23 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/UserInterface/JsonRpcApplication.php(176): Serenata\\Sockets\\JsonRpcQueueItemProcessor->process(Object(Serenata\\Sockets\\JsonRpcQueueItem))\n#24 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/UserInterface/JsonRpcApplication.php(140): Serenata\\UserInterface\\JsonRpcApplication->processNextQueueItem()\n#25 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/vendor/react/event-loop/src/Timer/Timers.php(96): Serenata\\UserInterface\\JsonRpcApplication->Serenata\\UserInterface\\{closure}(Object(React\\EventLoop\\Timer\\Timer))\n#26 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/vendor/react/event-loop/src/StreamSelectLoop.php(184): React\\EventLoop\\Timer\\Timers->tick()\n#27 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/UserInterface/JsonRpcApplication.php(112): React\\EventLoop\\StreamSelectLoop->run()\n#28 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/vendor/symfony/console/Command/Command.php(253): Serenata\\UserInterface\\JsonRpcApplication->runEventLoop(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))\n#29 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/vendor/symfony/console/Application.php(1011): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))\n#30 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/vendor/symfony/console/Application.php(272): Symfony\\Component\\Console\\Application->doRunCommand(Object(Symfony\\Component\\Console\\Command\\Command), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))\n#31 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/vendor/symfony/console/Application.php(148): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))\n#32 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/UserInterface/JsonRpcApplication.php(63): Symfony\\Component\\Console\\Application->run()\n#33 phar://C:/Users/PC-ADMIN/AppData/Roaming/php-ide-serenata/distribution.phar/builds/Serenata/Serenata/src/Main.php(24): Serenata\\UserInterface\\JsonRpcApplication->run()\n#34 C:\\Users\\PC-ADMIN\\AppData\\Roaming\\php-ide-serenata\\distribution.phar(10): include('phar://C:/Users...')\n#35 {main}"
    }
}

The whole console.log content exported https://pastebin.com/h85mUAZk (to long to paste here)

[Gert-dev]

@ThomasLandauer I don't think it's related to the Atom version. This problem existed before as well, but it appears to have gotten worse recently, perhaps this has to do with recent PHP versions changing something (though I already fixed a bunch of errors in e.g. PHP 7.4).

If in your console you see errors related to "Entity of type 'Serenata\Indexing\Structures\File' for IDs id(...) was not found", this is likely related to this upstream issue.

[sjuvonen]

I tried reporting this earlier but my ticket got lost. I've noticed that running PHP apps in Docker containers, when source code is shared through a Docker volume, make indexing repeat and repeat and repeat. Setting noatime in mount options (on Linux) seems to help a little bit.

Another issue is that after closing all PHP file tabs, the Serenata server process is killed. When opening a new file tab after this, the process will be booted again and it will then commence to reindex all files, which will take ages on large projects. This is, sorry, very poor behavior.

There seem to be other problems that trigger massive reindexing as well, but I haven't exactly figured them out yet. My presumption is that the Serenata Atom plugin is listening to some very generic event and will basically trigger reindexing even when just moving the text cursor around in a document.

Serenata server is also ignoring the uris options in config.json file and will instead index each and every PHP file that it can find. The only way to actually limit what will be indexed is via excludedPathExpressions. Ignoring uris leads to indexing all kinds of pointless crap, which will in turn cause the aforementioned pointless reindexing operations to take even longer.

In my case the index size is about 180 MB. Before I learned to exclude files properly via excludedPathExpressions, the index had ballooned to around 280 MB. So it's a huge task to reindex simply the project files let alone adding useless junk on top. In my case the additional files were static caches generated by Doctrine and Symfony as well as implicitly installed PHPUnit environments.

[sjuvonen]

Okay, I've noticed other bad behavior too.

Flushing Symfony caches while Atom editor is open causes the Atom plugin to trigger a reindexing on the Serenata server. This seems to be a bug both in the Atom plugin and Serenata server code.

My Serenata config looks like this:

{
    "uris": [
        "file:///home/.../bundles",
        "file:///home/.../backend/src",
        "file:///home/.../backend/vendor"
    ],
    "indexDatabaseUri": ":memory:",
    "phpVersion": 7.4,
    "excludedPathExpressions": [
        "/backend/phpunit",
        "/backend/var"
    ],
    "fileExtensions": [
        "php"
    ]
}

Folder backend/vendor is Composer vendor dir and backend/src is my project's source code root. In excludedPathExpressions I've explicitly banned the Symfony's var directory from Serenata's indexing. Yet it seems that the Atom plugin dispatches an RPC command to Serenata server pointing to e.g. var/log/dev.log and this triggers some massive, completely pointless reindexing or crawling.

It also seems that if I flush the cache again while a previous Serenata indexing operation is still being executed, the Serenata server will enter a semi-infinite loop and consume massive amounts of CPU time but never finish whatever it is doing. It also seems to not dispatch any updates back to the Atom client. (Normally it would announce progress and what file it finished indexing.)