Running osrm-extract on us-latest.osm.pbf

[fitzg2]

Has anybody tried to run

docker run -t -v "${PWD}:/data" ghcr.io/project-osrm/osrm-backend osrm-extract -p /opt/car.lua /data/us-latest.osm.pbf

on http://download.geofabrik.de/north-america/us-latest.osm.pbf ?

It's not generating the OSRM files to run osrm-partition and customize

[SiarheiFedartsou]

What do you have as an output of this command? Are you sure you have enough RAM?

[fitzg2]

It seems to be exiting correctly. It doesn't output any error. Tried with a GCP e2-standard-16 instance. So 64gb ram 16 cores.

[SiarheiFedartsou]

It seems to be exiting correctly. It doesn't output any error. Tried with a GCP e2-standard-16 instance. So 64gb ram 16 cores.

Could provide output of command?

[fitzg2]

@SiarheiFedartsou here's the entire output

https://hastebin.com/raw/ajivivisap

[SiarheiFedartsou]

@SiarheiFedartsou here's the entire output

https://hastebin.com/raw/ajivivisap

Succesfull osrm-extract run is usually finished with smth like:

[2022-11-09T16:57:38.626991] [info] RAM: peak bytes used: 596377600

What is not a case here. Can you try smaller .osm.pbf? E.g. berlin-latest.osm.pbf from README?

[williamknn]

I'm with the same problem, it seems apparently exited with success but the osrm file is not generated. See the output here: https://gist.github.com/williamknn/6e4418b83841ff9b1717b9b09df5f569

I've tried with a smaller map (mexico) but the result is the same.

AWS m5.8xlarge (128.0 GiB | 32 vCPUs)
Ubuntu 20.04
libboost1.73
lua5.3

[SiarheiFedartsou]

but the osrm file is not generated

Do you expect file with .osrm extension to be generated? We recently got rid of it #6354, it is was intermediate artefact of osrm-extract which is not actually not needed in other parts of OSRM.

[williamknn]

but the osrm file is not generated

Do you expect file with .osrm extension to be generated? We recently got rid of it #6354, it is was intermediate artefact of osrm-extract which is not actually not needed in other parts of OSRM.

So how do I build the map now? How can I run the contract? Sorry but after this release I don't find anything about this in wiki pages.

PS: Even using --dump-nbg-graph as an argument the file still not being generated.

[williamknn]

I think now I understood how this works... I thought that contract uses the .osrm file as input and then changes it but it doesn't. The .osrm file generated by the extract is only for debug pourposes and contract uses the entire directory to generate the file. Am I correct?

I think this needs to be clear in documentation to avoid this kind of misunderstanding.

[SiarheiFedartsou]

@williamknn exactly. I would be happy to approve and merge any PRs improving our documentation :)

[fitzg2]

@SiarheiFedartsou how much memory would it take to process https://download.geofabrik.de/north-america/us-latest.osm.pbf ?
would it be better to process special sub regions and serve them under different ports?

[fitzg2]

Got it to process us-latest.osm.pbf on a 256gb ram + 32vcpu droplet for anyone interested. Didn't measure the time.