Segmentation fault during import

[Eric64534]

Any ideas?? Here's the log:

tech@4YF0GT3:/mnt/c/Windows/system32$ docker volume create osm-data
osm-data
tech@4YF0GT3:/mnt/c/Windows/system32$ docker run
-v /mnt/e/planet-231002.osm.pbf:/data/region.osm.pbf
-v osm-data:/data/database/
-e "FLAT_NODES=enabled"
-e "OSM2PGSQL_EXTRA_ARGS=-C 8192"
-e THREADS=18
overv/openstreetmap-tile-server
import
Unable to find image 'overv/openstreetmap-tile-server:latest' locally
latest: Pulling from overv/openstreetmap-tile-server
6b851dcae6ca: Pull complete
bb071db28a33: Pull complete
a862358c8622: Pull complete
57ea4a769709: Pull complete
7637307ae531: Pull complete
fa1264c2a6e6: Pull complete
5591dd38bd35: Pull complete
c82b1fb2ea15: Pull complete
47ecf7831eac: Pull complete
18125c639464: Pull complete
2b0335ae4896: Pull complete
4c9807a2e2f3: Pull complete
649b76086468: Pull complete
b52cd0b2e16e: Pull complete
9874a5bac373: Pull complete
acd614dbc6df: Pull complete
8a950b4e43d0: Pull complete
269690f5a06c: Pull complete
19ed3f017280: Pull complete
4b7f8bb34a12: Pull complete
25b1f3b221b7: Pull complete
0056ac6bf411: Pull complete
57110a116d22: Pull complete
208240568089: Pull complete
Digest: sha256:b6a79da39b6d0758368f7c62d22e49dd3ec59e78b194a5ef9dee2723b1f3fa79
Status: Downloaded newer image for overv/openstreetmap-tile-server:latest
++ ls -A /data/style/

• '[' '!' '' ']'

• mv /home/renderer/src/openstreetmap-carto-backup/CARTOGRAPHY.md /home/renderer/src/openstreetmap-carto-backup/CHANGELOG.md /home/renderer/src/openstreetmap-carto-backup/CODE_OF_CONDUCT.md /home/renderer/src/openstreetmap-carto-backup/CONTRIBUTING.md /home/renderer/src/openstreetmap-carto-backup/DOCKER.md /home/renderer/src/openstreetmap-carto-backup/Dockerfile /home/renderer/src/openstreetmap-carto-backup/Dockerfile.db /home/renderer/src/openstreetmap-carto-backup/Dockerfile.import /home/renderer/src/openstreetmap-carto-backup/INSTALL.md /home/renderer/src/openstreetmap-carto-backup/LICENSE.txt /home/renderer/src/openstreetmap-carto-backup/README.md /home/renderer/src/openstreetmap-carto-backup/RELEASES.md /home/renderer/src/openstreetmap-carto-backup/USECASES.md /home/renderer/src/openstreetmap-carto-backup/docker-compose.yml /home/renderer/src/openstreetmap-carto-backup/external-data.yml /home/renderer/src/openstreetmap-carto-backup/indexes.sql /home/renderer/src/openstreetmap-carto-backup/indexes.yml /home/renderer/src/openstreetmap-carto-backup/openstreetmap-carto.lua /home/renderer/src/openstreetmap-carto-backup/openstreetmap-carto.style /home/renderer/src/openstreetmap-carto-backup/patterns /home/renderer/src/openstreetmap-carto-backup/preview.png /home/renderer/src/openstreetmap-carto-backup/project.mml /home/renderer/src/openstreetmap-carto-backup/road-colors.yaml /home/renderer/src/openstreetmap-carto-backup/scripts /home/renderer/src/openstreetmap-carto-backup/style /home/renderer/src/openstreetmap-carto-backup/symbols /data/style/

• '[' '!' -f /data/style/mapnik.xml ']'

• cd /data/style/

• carto project.mml
  Warning: style/landcover.mss:609:4 line-offset is unstable. It may change in the future.
  ... additional warnings removed ...

• '[' import == import ']'

• mkdir -p /data/database/postgres/

• chown renderer: /data/database/

• chown -R postgres: /var/lib/postgresql /data/database/postgres/

• '[' '!' -f /data/database/postgres/PG_VERSION ']'

• createPostgresConfig

• cp /etc/postgresql/15/main/postgresql.custom.conf.tmpl /etc/postgresql/15/main/conf.d/postgresql.custom.conf

• sudo -u postgres echo 'autovacuum = on'

• cat /etc/postgresql/15/main/conf.d/postgresql.custom.conf

Suggested minimal settings from

https://ircama.github.io/osm-carto-tutorials/tile-server-ubuntu/

shared_buffers = 128MB
min_wal_size = 1GB

max_wal_size = 2GB # Overridden below

maintenance_work_mem = 256MB

Suggested settings from

https://github.com/openstreetmap/chef/blob/master/roles/tile.rb#L38-L45

max_connections = 250
temp_buffers = 32MB
work_mem = 128MB
wal_buffers = 1024kB
wal_writer_delay = 500ms
commit_delay = 10000

checkpoint_segments = 60 # unrecognized in psql 10.7.1

max_wal_size = 2880MB
random_page_cost = 1.1
track_activity_query_size = 16384
autovacuum_vacuum_scale_factor = 0.05
autovacuum_analyze_scale_factor = 0.02

listen_addresses = '*'
autovacuum = on

• service postgresql start

• Starting PostgreSQL 15 database server
  ...done.

• sudo -u postgres createuser renderer
• sudo -u postgres createdb -E UTF8 -O renderer gis
• sudo -u postgres psql -d gis -c 'CREATE EXTENSION postgis;'
  CREATE EXTENSION
• sudo -u postgres psql -d gis -c 'CREATE EXTENSION hstore;'
  CREATE EXTENSION
• sudo -u postgres psql -d gis -c 'ALTER TABLE geometry_columns OWNER TO renderer;'
  ALTER TABLE
• sudo -u postgres psql -d gis -c 'ALTER TABLE spatial_ref_sys OWNER TO renderer;'
  ALTER TABLE
• setPostgresPassword
• sudo -u postgres psql -c 'ALTER USER renderer PASSWORD '''renderer''''
  ALTER ROLE
• '[' '!' -f /data/region.osm.pbf ']'
• '[' -n '' ']'
• '[' disabled == enabled ']'
• '[' disabled == 1 ']'
• '[' -f /data/region.poly ']'
• '[' enabled == enabled ']'
• OSM2PGSQL_EXTRA_ARGS='-C 8192 --flat-nodes /data/database/flat_nodes.bin'
• sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /data/style/openstreetmap-carto.lua --number-processes 18 -S /data/style/openstreetmap-carto.style /data/region.osm.pbf -C 8192 --flat-nodes /data/database/flat_nodes.bin
  2023-10-23 01:56:34 osm2pgsql version 1.6.0
  2023-10-23 01:56:34 Database version: 15.3 (Ubuntu 15.3-1.pgdg22.04+1)
  2023-10-23 01:56:34 PostGIS version: 3.3
  2023-10-23 01:56:34 Setting up table 'planet_osm_point'
  2023-10-23 01:56:34 Setting up table 'planet_osm_line'
  2023-10-23 01:56:35 Setting up table 'planet_osm_polygon'
  2023-10-23 01:56:35 Setting up table 'planet_osm_roads'
  2023-10-25 22:14:48 Reading input files done in 245893s (68h 18m 13s).
  2023-10-25 22:14:48 Processed 8658483163 nodes in 3446s (57m 26s) - 2513k/s
  2023-10-25 22:14:48 Processed 971692934 ways in 160275s (44h 31m 15s) - 6k/s
  2023-10-25 22:14:48 Processed 11340554 relations in 82172s (22h 49m 32s) - 138/s
  2023-10-25 22:14:48 Clustering table 'planet_osm_point' by geometry...
  2023-10-25 22:14:48 Clustering table 'planet_osm_polygon' by geometry...
  2023-10-25 22:14:48 Clustering table 'planet_osm_roads' by geometry...
  2023-10-25 22:14:48 Clustering table 'planet_osm_line' by geometry...
  2023-10-25 22:14:51 Done postprocessing on table 'planet_osm_nodes' in 0s
  2023-10-25 22:14:51 Building index on table 'planet_osm_rels'
  2023-10-25 22:14:51 Building index on table 'planet_osm_ways'
  /run.sh: line 130: 76 Segmentation fault sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /data/style/${NAME_LUA:-openstreetmap-carto.lua} --number-processes ${THREADS:-4} -S /data/style/${NAME_STYLE:-openstreetmap-carto.style} /data/region.osm.pbf ${OSM2PGSQL_EXTRA_ARGS:-}
  tech@4YF0GT3:/mnt/c/Windows/system32$

[Istador]

18 THREADS seems too high, that'd result in up to (3 + (1+18) * 14) = 269 connections, whereas max_connections in postgresql.custom.conf.tmpl is configured to 250.

Also resulting from this: does your system really have 269 * 128 MB + 8192 MB = 42 GB of memory?

See #330

[Eric64534]

Thank you so much for the reply! Yes, this system has an Intel i9-10900X (20 threads) and 64 GB of RAM. I'll try again with less threads.

Best,
Eric

[Eric64534]

Well, I got a little further. Now it says I ran out of disk space. The osm-data volume resides on an external USB3-to-SATA adapter with a 1.9TB SSD. With everything idle now (after import aborted), the volume says it is 926 GB and windows says the disk (e:) has 760 GB free. My boot drive (c:) has 500 GB free. Any assistance would be greatly appreciated!

tech@4YF0GT3:/mnt/c/Users/ehughes$ docker run
-v /mnt/e/planet-231002.osm.pbf:/data/region.osm.pbf
-v osm-data:/data/database/
-e "FLAT_NODES=enabled"
-e "OSM2PGSQL_EXTRA_ARGS=-C 8192"
-e THREADS=10
overv/openstreetmap-tile-server
import

• '[' import == import ']'
• mkdir -p /data/database/postgres/
• chown renderer: /data/database/
• chown -R postgres: /var/lib/postgresql /data/database/postgres/
• '[' '!' -f /data/database/postgres/PG_VERSION ']'
• createPostgresConfig
• cp /etc/postgresql/15/main/postgresql.custom.conf.tmpl /etc/postgresql/15/main/conf.d/postgresql.custom.conf
• sudo -u postgres echo 'autovacuum = on'
• cat /etc/postgresql/15/main/conf.d/postgresql.custom.conf

Suggested minimal settings from

https://ircama.github.io/osm-carto-tutorials/tile-server-ubuntu/

shared_buffers = 128MB
min_wal_size = 1GB

max_wal_size = 2GB # Overridden below

maintenance_work_mem = 256MB

Suggested settings from

https://github.com/openstreetmap/chef/blob/master/roles/tile.rb#L38-L45

max_connections = 250
temp_buffers = 32MB
work_mem = 128MB
wal_buffers = 1024kB
wal_writer_delay = 500ms
commit_delay = 10000

checkpoint_segments = 60 # unrecognized in psql 10.7.1

max_wal_size = 2880MB
random_page_cost = 1.1
track_activity_query_size = 16384
autovacuum_vacuum_scale_factor = 0.05
autovacuum_analyze_scale_factor = 0.02

listen_addresses = '*'
autovacuum = on

• service postgresql start

• Starting PostgreSQL 15 database server
  ...done.

• sudo -u postgres createuser renderer
• sudo -u postgres createdb -E UTF8 -O renderer gis
• sudo -u postgres psql -d gis -c 'CREATE EXTENSION postgis;'
  CREATE EXTENSION
• sudo -u postgres psql -d gis -c 'CREATE EXTENSION hstore;'
  CREATE EXTENSION
• sudo -u postgres psql -d gis -c 'ALTER TABLE geometry_columns OWNER TO renderer;'
  ALTER TABLE
• sudo -u postgres psql -d gis -c 'ALTER TABLE spatial_ref_sys OWNER TO renderer;'
  ALTER TABLE
• setPostgresPassword
• sudo -u postgres psql -c 'ALTER USER renderer PASSWORD '''renderer''''
  ALTER ROLE
• '[' '!' -f /data/region.osm.pbf ']'
• '[' -n '' ']'
• '[' disabled == enabled ']'
• '[' disabled == 1 ']'
• '[' -f /data/region.poly ']'
• '[' enabled == enabled ']'
• OSM2PGSQL_EXTRA_ARGS='-C 8192 --flat-nodes /data/database/flat_nodes.bin'
• sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /data/style/openstreetmap-carto.lua --number-processes 10 -S /data/style/openstreetmap-carto.style /data/region.osm.pbf -C 8192 --flat-nodes /data/database/flat_nodes.bin
  2023-10-26 18:54:07 osm2pgsql version 1.6.0
  2023-10-26 18:54:07 Database version: 15.3 (Ubuntu 15.3-1.pgdg22.04+1)
  2023-10-26 18:54:07 PostGIS version: 3.3
  2023-10-26 18:54:08 Setting up table 'planet_osm_point'
  2023-10-26 18:54:08 Setting up table 'planet_osm_line'
  2023-10-26 18:54:08 Setting up table 'planet_osm_polygon'
  2023-10-26 18:54:08 Setting up table 'planet_osm_roads'
  2023-10-29 18:59:03 Reading input files done in 259495s (72h 4m 55s).
  2023-10-29 18:59:03 Processed 8658483163 nodes in 3478s (57m 58s) - 2490k/s
  2023-10-29 18:59:03 Processed 971692934 ways in 175068s (48h 37m 48s) - 6k/s
  2023-10-29 18:59:03 Processed 11340554 relations in 80949s (22h 29m 9s) - 140/s
  2023-10-29 18:59:03 Clustering table 'planet_osm_polygon' by geometry...
  2023-10-29 18:59:03 Clustering table 'planet_osm_roads' by geometry...
  2023-10-29 18:59:03 Clustering table 'planet_osm_line' by geometry...
  2023-10-29 18:59:03 Clustering table 'planet_osm_point' by geometry...
  2023-10-29 18:59:06 Done postprocessing on table 'planet_osm_nodes' in 0s
  2023-10-29 18:59:06 Building index on table 'planet_osm_rels'
  2023-10-29 18:59:06 Building index on table 'planet_osm_ways'
  2023-10-29 22:22:17 ERROR: Database error: ERROR: could not extend file "base/16385/7440649.3": No space left on device
  HINT: Check free disk space.

[Istador]

If I remember correctly, the docker settings on Windows might impose a maximum volume size limit? IDK.

Likely it cleaned up the files from the step it was at after the error happened and that is why there is more free space now than when the error occured. According to the logs the data import took 3 days, and it failed 3 hours later when creating the indicies for the planet_osm_ways table.

According to https://wiki.openstreetmap.org/wiki/Planet.osm the uncompressed data is 1.8 TB, so 1.9 TB for the whole planet might not be enough? But the import doesn't import all the data, so it should be less, though the database also needs space for its own and for indicies and stuff. I think 1.9 TB should be enough if there is nothing else on the disk. What I read from about two years ago the size was about 1.2 TB imported.

[Eric64534]

Just to follow-up, the issue was indeed Windows (and/or docker) imposing a volume size limit of 1TB. I tried all sorts of things to increase it, but nothing worked. I finally gave up and moved to Ubuntu and everything worked. Really appreciate your help!!