Disk size for whole planet

[dlubom]

Hello, I'm trying to import whole planet planet-latest.osm.pbf but looks like disk size ~1.1T is not enough today. Could someone confirm that ? Or maybe I`m doing something wrong?

$cat .env 
MINATIM_FLATNODE_FILE="/srv/nominatim/nominatim-planet/flatnode.file"


$nominatim import -v -j 10 --osm-file planet-latest.osm.pbf 2>&1 | tee setup5.log


2024-03-04 12:34:49: Using project directory: /srv/nominatim/nominatim-planet
2024-03-04 12:34:53: Creating database
2024-03-04 12:34:57: Setting up country tables
SET
SET
SET
SET
SET
SET
SET
SET
SET
CREATE TABLE
COPY 23217
CREATE INDEX
2024-03-04 12:35:00: Importing OSM data file
2024-03-04 12:35:00  osm2pgsql version 1.9.2
2024-03-04 12:35:00  Database version: 14.11 (Ubuntu 14.11-0ubuntu0.22.04.1)
2024-03-04 12:35:00  PostGIS version: 3.2
2024-03-04 12:35:00  Storing properties to table '"public"."osm2pgsql_properties"'.
2024-03-05 09:48:35  Reading input files done in 76415s (21h 13m 35s).
2024-03-05 09:48:35    Processed 8974341667 nodes in 30769s (8h 32m 49s) - 292k/s
2024-03-05 09:48:35    Processed 1002917655 ways in 31393s (8h 43m 13s) - 32k/s
2024-03-05 09:48:35    Processed 11898194 relations in 14253s (3h 57m 33s) - 835/s
2024-03-05 09:48:35  No marked ways (Skipping stage 2).
2024-03-05 09:48:35  Clustering table 'place' by geometry...
2024-03-05 12:26:56  Done postprocessing on table 'planet_osm_nodes' in 0s
2024-03-05 12:26:56  Building index on table 'planet_osm_ways'
2024-03-05 19:05:32  Done postprocessing on table 'planet_osm_ways' in 23915s (6h 38m 35s)
2024-03-05 19:05:32  Building index on table 'planet_osm_rels'
2024-03-05 19:13:29  Done postprocessing on table 'planet_osm_rels' in 477s (7m 57s)
2024-03-05 19:13:29  ERROR: Error while doing postprocessing on table 'place':
2024-03-05 19:13:30  ERROR: Database error: ERROR:  could not extend file "base/23691418/28303016.100": wrote only 4096 of 8192 bytes at block 13232211
HINT:  Check free disk space.

Traceback (most recent call last):
  File "/usr/local/bin/nominatim", line 12, in <module>
    exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module',
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 225, in nominatim
    return get_set_parser().run(**kwargs)
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 121, in run
    return args.command.run(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/setup.py", line 94, in run
    database_import.import_osm_data(files,
  File "/usr/local/lib/nominatim/lib-python/nominatim/tools/database_import.py", line 113, in import_osm_data
    run_osm2pgsql(options)
  File "/usr/local/lib/nominatim/lib-python/nominatim/tools/exec_utils.py", line 76, in run_osm2pgsql
    subprocess.run(cmd, cwd=options.get('cwd', '.'),
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/local/lib/nominatim/osm2pgsql', '--hstore', '--latlon', '--slim', '--log-progress', 'true', '--number-processes', '1', '--cache', '96757', '--style', '/usr/local/etc/nominatim/import-extratags.lua', '--output', 'flex', '--create', 'planet-latest.osm.pbf']' returned non-zero exit status 1.

$sudo du -hs /var/lib/postgresql
950G    /var/lib/postgresql

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            13G  812K   13G   1% /run
/dev/sda2       1.3T  1.1T  181G  86% /
tmpfs            63G   28K   63G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            13G  4.0K   13G   1% /run/user/1002

I have 128 GB RAM and 10 CPU.

Thanks.

[lonvia]

The 1.3T disk should still be sufficient to import a full planet. Your .env file seems to have a typo. The configuration parameter is called NOMINATIM_FLATNODE_FILE. It would explain the large database and why osm2pgsql is so slow to import.

[dlubom]

Thanks, it working now.