Failing when attempting to extract Malaysia

[nicpottier]

Not having any luck getting an extract out of Malaysia. First tried using GeoFabrik's Asia extract, but ran out of memory. Now trying a BBBike extract, but fails with the error below.

PBF file: (grab it while it's hot): http://download.bbbike.org/osm/extract/planet_90.3,-18_130.1,25.8.osm.pbf

Settings:

sources:
    data_dir: '/home/ubuntu/mapzen/data/' # full path to the data file
    output_dir: '/home/ubuntu/mapzen/output/' # full path to output directory
    data_url: 'http://download.bbbike.org/osm/extract/planet_90.3,-18_130.1,25.8.osm.pbf'
    input_dir: '/home/ubuntu/mapzen/input/'
    filename: 'malaysia.pbf'

admin_levels:
    - 2
    - 4
    - 6
    - 8

tolerance: 0.001
country_osm_id: 2108121 # Malaysia

Log:

(mapzen_env)ubuntu@ip-10-63-215-73:~/mapzen-exporter/exporter$ python exporter.py run_all
Trying `CDLL(libgeos_c.so.1)`
DEBUG:shapely.geos:Trying `CDLL(libgeos_c.so.1)`
Library path: 'libgeos_c.so.1'
DEBUG:shapely.geos:Library path: 'libgeos_c.so.1'
DLL: <CDLL 'libgeos_c.so.1', handle 1d27df0 at 7f5c3b0bd250>
DEBUG:shapely.geos:DLL: <CDLL 'libgeos_c.so.1', handle 1d27df0 at 7f5c3b0bd250>
Trying `CDLL(libc.so.6)`
DEBUG:shapely.geos:Trying `CDLL(libc.so.6)`
Library path: 'libc.so.6'
DEBUG:shapely.geos:Library path: 'libc.so.6'
DLL: <CDLL 'libc.so.6', handle 7f5c4c0659b0 at 7f5c3b0bd290>
DEBUG:shapely.geos:DLL: <CDLL 'libc.so.6', handle 7f5c4c0659b0 at 7f5c3b0bd290>
DEBUG:mapzen_settings.log:Reading settings from settings.yaml
INFO:/home/ubuntu/mapzen-exporter/exporter/utility/utils.pyc:Initializing data folder:  /home/ubuntu/mapzen/data/ ...
INFO:/home/ubuntu/mapzen-exporter/exporter/utility/utils.pyc:Completed setting up data folder:  /home/ubuntu/mapzen/data/ ...
INFO:/home/ubuntu/mapzen-exporter/exporter/utility/utils.pyc:Initializing data folder:  /home/ubuntu/mapzen/output/ ...
INFO:/home/ubuntu/mapzen-exporter/exporter/utility/utils.pyc:Completed setting up data folder:  /home/ubuntu/mapzen/output/ ...
INFO:/home/ubuntu/mapzen-exporter/exporter/utility/utils.pyc:Initializing data folder:  /home/ubuntu/mapzen/input/ ...
INFO:/home/ubuntu/mapzen-exporter/exporter/utility/utils.pyc:Completed setting up data folder:  /home/ubuntu/mapzen/input/ ...
DEBUG:mapzen_settings.log:Reading settings from settings.yaml
 100 % [=============================================================================>]  519.3MiB/ 519.3MiB  14.2MiB/s  eta 0:00:00   
DEBUG:mapzen_settings.log:Reading settings from settings.yaml
Extracting admin boundaries from /home/ubuntu/mapzen/data/malaysia.pbf into /home/ubuntu/mapzen/input/

/usr/local/lib/node_modules/fences-builder/src/OSMAreaBuilder.js:41
  this._mp.read_relations(mpReader);
           ^
Error: osmium error: std::bad_alloc
    at new OSMAreaBuilder (/usr/local/lib/node_modules/fences-builder/src/OSMAreaBuilder.js:41:12)
    at extractPolygons (/usr/local/lib/node_modules/fences-builder/src/childProcess.js:46:21)
    at process.stats.matched (/usr/local/lib/node_modules/fences-builder/src/childProcess.js:16:5)
    at process.EventEmitter.emit (events.js:98:17)
    at handleMessage (child_process.js:318:10)
    at Pipe.channel.onread (child_process.js:345:11)
Child process exited with error  8
('Fence Builder extraction of OSM data file failed with error code: ', 2048)

[nicpottier]

Any clues would be mighty useful..

[teehamaral]

I am also the same mistake. But already I have the file locally and just want to run python exporter.py extract_admins.

(mapzen_env)root@ip-172-31-50-249:/home/ubuntu/mapzen-exporter/exporter# python exporter.py extract_admins
Trying CDLL(libgeos_c.so.1)
DEBUG:shapely.geos:Trying CDLL(libgeos_c.so.1)
Library path: libgeos_c.so.1
DEBUG:shapely.geos:Library path: libgeos_c.so.1
DLL: <CDLL libgeos_c.so.1, handle 1e887a0 at 7f5c1b7af3d0>
DEBUG:shapely.geos:DLL: <CDLL libgeos_c.so.1, handle 1e887a0 at 7f5c1b7af3d0>
Trying CDLL(libc.so.6)
DEBUG:shapely.geos:Trying CDLL(libc.so.6)
Library path: libc.so.6
DEBUG:shapely.geos:Library path: libc.so.6
DLL: <CDLL libc.so.6, handle 7f5c2c7639b0 at 7f5c1b7af410>
DEBUG:shapely.geos:DLL: <CDLL libc.so.6, handle 7f5c2c7639b0 at 7f5c1b7af410>
DEBUG:mapzen_settings.log:Reading settings from settings.yaml
Extracting admin boundaries from /home/ubuntu/data/ireland.osm.pbf into /home/ubuntu/data/input/

/usr/local/lib/node_modules/fences-builder/src/OSMAreaBuilder.js:41
  this._mp.read_relations(mpReader);
           ^
Error: osmium error: std::bad_alloc
    at new OSMAreaBuilder (/usr/local/lib/node_modules/fences-builder/src/OSMAreaBuilder.js:41:12)
    at extractPolygons (/usr/local/lib/node_modules/fences-builder/src/childProcess.js:46:21)
    at process.stats.matched (/usr/local/lib/node_modules/fences-builder/src/childProcess.js:16:5)
    at process.EventEmitter.emit (events.js:98:17)
    at handleMessage (child_process.js:318:10)
    at Pipe.channel.onread (child_process.js:345:11)
Child process exited with error  8
('Fence Builder extraction of OSM data file failed with error code: ', 2048)

[teehamaral]

Hello @nicpottier see this osmcode/node-osmium#77.

[nicpottier]

Ya, obviously running out of ram. You'd think 8 gigs of ram would be enough though, so I blame the approach as opposed to the ram. :)

[teehamaral]

Yeah. I think 8 gigs of ram would be enough. Do you already are running with 8 gigs and is still with error?

[nicpottier]

Yep.. I did eventually get this to work in my case by doing another BBBike extract for the area I was doing. But ya, using a full Asia extract always ran out of ram.