Skip to content

Conversation

takb
Copy link
Contributor

@takb takb commented Jun 11, 2025

Allows to store only required node information instead of keeping all ways in memory.

Had to do some optimizations using numpy and cykhash but now init has a significantly smaller memory footprint.

ops-init  | 2025-06-16 14:31:28,670 INFO     Found directory: /deploy/app/osm
ops-init  | 2025-06-16 14:31:28,670 INFO     Starting to import OSM data (1 files in batch)
ops-init  | 2025-06-16 14:31:28,690 INFO     Starting to read /deploy/app/osm/baden-wuerttemberg-latest.osm.pbf
ops-init  | 2025-06-16 14:31:28,691 INFO     Parsing and importing nodes...
ops-init  | 2025-06-16 14:35:50,592 INFO     Found 721367 nodes
ops-init  | 2025-06-16 14:35:50,592 INFO     Parsing relations...
ops-init  | 2025-06-16 14:35:58,811 INFO     Found 2519 ways in relations
ops-init  | 2025-06-16 14:35:58,811 INFO     Parsing ways...
ops-init  | 2025-06-16 14:36:39,755 INFO     Found 326253 ways (including the ones found in relations)
ops-init  | 2025-06-16 14:36:39,755 INFO     process_ways: 326253
ops-init  | 2025-06-16 14:36:54,541 INFO     Importing ways...
ops-init  | Total memory of process_ways: 408.36 MB
ops-init  | 2025-06-16 14:41:02,966 INFO     Found 1047620 POIs
ops-init  | 2025-06-16 14:41:02,966 INFO     Finished import of /deploy/app/osm/baden-wuerttemberg-latest.osm.pbf
ops-init  | 2025-06-16 14:41:02,967 INFO     File import completed in 574281.39 ms
ops-init  | 2025-06-16 14:41:02,967 INFO     Import complete.
ops-init  | 2025-06-16 14:41:02,967 INFO     Batch of 1 files completed in 574297.03 ms
ops-init  | 2025-06-16 14:41:03,162 INFO     Starting to read /deploy/app/osm/baden-wuerttemberg-latest.osm.pbf
ops-init  | 2025-06-16 14:41:03,162 INFO     Parsing and importing nodes...
ops-init  | 2025-06-16 14:46:29,977 INFO     Found 721367 nodes
ops-init  | 2025-06-16 14:46:29,977 INFO     Parsing relations...
ops-init  | 2025-06-16 14:46:38,200 INFO     Found 2519 ways in relations
ops-init  | 2025-06-16 14:46:38,201 INFO     Parsing ways (1st pass)...
ops-init  | 2025-06-16 14:47:16,145 INFO     Found 326432 ways (including the ones found in relations)
ops-init  | 2025-06-16 14:47:16,146 INFO     Parsing coords ...
ops-init  | 2025-06-16 14:48:03,091 INFO     Parsing and importing ways (2nd pass)...
ops-init  | Total memory of Map: 114.09 MB
ops-init  | 2025-06-16 14:51:05,139 INFO     Found 1047799 POIs
ops-init  | 2025-06-16 14:51:05,139 INFO     Finished import of /deploy/app/osm/baden-wuerttemberg-latest.osm.pbf
ops-init  | 2025-06-16 14:51:05,145 INFO     File import completed in 601986.15 ms
ops-init  | 2025-06-16 14:51:05,146 INFO     Import complete.
ops-init  | 2025-06-16 14:51:05,146 INFO     Batch of 1 files completed in 601987.79 ms

…ormation instead of keeping all ways in memory
@takb takb merged commit c99d698 into main Jun 17, 2025
1 check passed
@takb takb deleted the feat/import-multipass branch June 17, 2025 09:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants