Stuck importing Netherlands

[andreademasi]

Hi everyone, the container i'm running is stuck since yesterday in the importing phase with this log line:

2025-05-08 17:58:47: Done 5104740 in 4944 @ 1032.582 per second - rank 30 ETA (seconds): 4504.56

Is this a resource problem? In the docker logs i don't see any crash. The host machine has 24GB of memory and about 70GB of disk space available. The container shared memory is also set to 12GB . Any ideas on how i can debug the issue?

[mtmail]

That phase of the import does most work inside the database (stored procedures, https://github.com/osm-search/Nominatim/blob/master/lib-sql/functions/), the output is Python controlling multiple threads.

Do you see anything in the Postgresql server logs? The server might have crashed due to resources (often the message is just 'processed killed'). Or queries might be blocked (pg_top or other tools should show that).

It's restartable, but I'm not sure how to do that inside Docker. I mean which other steps Docker runs automatically and how to run them manually. https://nominatim.org/release-docs/latest/admin/Faq/#can-a-stoppedkilled-import-process-be-resumed

[andreademasi]

These this is the output of cat /var/log/postgresql/postgresql-16-main.log :

...
2025-05-08 18:33:16.080 UTC [45] LOG:  checkpoint starting: time
2025-05-08 18:42:16.052 UTC [45] LOG:  checkpoint complete: wrote 35556 buffers (13.6%); 0 WAL file(s) added, 2 removed, 13 recycled; write=539.965 s, sync=0.001 s, total=539.973 s; sync files=0, longest=0.000 s, average=0.000 s; distance=248250 kB, estimate=404448 kB; lsn=5/CB356AD0, redo lsn=5/CB356458
2025-05-09 07:57:52.540 UTC [3868] root@nominatim FATAL:  role "root" does not exist
2025-05-09 07:57:52.545 UTC [3869] root@nominatim FATAL:  role "root" does not exist

I'm running the container using portainer.. could it be that i messed up something in the container setup?

[mtmail]

The log line look alright. Nominatim first creates the database, sets up tables, imports into the planet_osm_* table, then the indexing (e.g. rank 30) starts. So it already managed to access and write to the database fine. The most common issue is hardware, not enough resources or instability. Does the container run on network mounted storage? If the container is still running check if the postgresql process is still running and if there are any running queries. Next step I'm afraid is to restart the import and see if it crashes at or near the same spot. People regularly import the whole planet without issue.

[andreademasi]

It's not running on network mounted storage, it's an oracle compute instance with its own boot volume. I already restarted it a bunch of time these days and it's always stuck at the same spot:

2025-05-09 20:06:00: Done 4746680 in 4775 @ 994.079 per second - rank 30 ETA (seconds): 5039.37

postgres logs:

2025-05-09 20:33:33.076 UTC [45] LOG:  checkpoint starting: time
2025-05-09 20:42:33.044 UTC [45] LOG:  checkpoint complete: wrote 34221 buffers (13.1%); 0 WAL file(s) added, 2 removed, 15 recycled; write=539.961 s, sync=0.001 s, total=539.969 s; sync files=0, longest=0.000 s, average=0.000 s; distance=280178 kB, estimate=414098 kB; lsn=B/869BBF20, redo lsn=B/7EBA21A8

psql also shows that some queries are still running

At this point i really think it's just a hardware issue, it is weird though that my setup is not enough for importing just a single country

[mtmail]

Could be some weird virtual machine behavior, for example the volume not expanding fast enough. But the logs indeed show no errors of any kind. I've seen errors related to individual OpenStreetMap data object but Nominatim would print an error and many users would report the same fast (especially those running full planet daily/hourly/minutely updates).