Skip to content

Getting started: DATA

Jump to bottom Edit New page
Paul Strickland edited this page May 30, 2018 · 40 revisions

Before you begin

Sources of Data

  • Detailed neighborhood streets, highways, rail, paths, and points-of-interest (POI) data comes from OpenStreetMap.
  • General world and regional scale map data such as countries or states comes from Natural Earth.
  • Detailed place names come from GeoNames, which collate them from a variety of sources. USGS and US Census are also good sources.
  • Digital elevation, landcover and similar data comes from US Geological Survey.
  • US statistical divisions like metro areas or counties come from US Census.

The first three are global in scope, the last two are mostly US-only, alternatives exist on a per-country basis.

Often data is pre-processed to prepare it for rendering. Sometimes this is to make it look a certain way, other times to make it run faster.

Considerations

  • Does your basemap need to reflect the latest edits?
  • Do you cover a rapidly expanding urban area where the roads change all the time and your users will notice?
  • Will you re-render your basemap on a never, yearly, monthly, weekly, daily frequency?
  • World up to a certain zoom then just your local geography?

Open Street Map

Small areas

  • Metro Extracts - Get OSM data extracts for big cities world wide in small chunks (mb instead of gb). Formats include XML, PBF, and osm2pgsql + imposm shapefiles. Includes coastline shapefile option. Updated monthly.

  • BBBike extracts - Allows you to extracts areas from Planet.osm in OSM, PBF, Garmin, Osmand or ESRI shapefile format. The maximum area size is 16,000,000 square km, or 768MB file size. It takes between 10-30 minutes to extract an area. You will be notified by e-mail if your extract is ready for download.

  • WeoGeo OSM Extracts - Free download from their marketplace. Data from the WeoGeo OSM listing can currently be output as FGDB, KML, DWG, FFS, and CSV (no geo info is included). Their developers are working on SHP and TAB support.

Full OSM Planet

  • The big kahuna. You'll need a beefy machine. Updated daily.

  • For making detailed city maps from zooms 12 to 19+.

  • Download OSM Planet »

  • Note: OSM redaction started in July 2012. Once it is complete, OSM will be ready to distribute data under the ODbL and they’ll advise of that with a separate announcement. The final pre-redaction dataset available under CC-BY-SA has now been generated at http://planet.openstreetmap.org/planet-120704.osm.bz2.

Importing OSM

  • Osmosis - Parsing and merging large Planet or other XML and binary OpenStreetMap data files.

  • impOSM - Optimize your OSM files per zoom for faster, for performant rendering.

      imposm -m imposm-mapping.py -U <user_name> -d <db_name> --proj=EPSG:900913 -c 4 --read --write --deploy-production-tables osm-planet-test-areas.pbf

Another:

    imposm -c 3 -U <user_name> -d <db_name> -m <mapping_file.py> --overwrite-cache --read --write --deploy-production-tables osm-planet-test-areas.pbf osm-planet-test-areas.pbf

  • osmfilter - Similar.

  • osm2pgsql - Get raw OSM files into PostGIS databases.

  • High Road - Add views into the data, grouping ways into large, medium, and small roads per zoom.

OSM Data Dictionary

Gotcha's

  • High Road - Important to drop and then read this on each OSM import. Details on the High Road page.

Natural Earth

Help getting Natural Earth into PostGIS

Spherical mercator extent and projection (ref):

EXTENT="-180 -85.05112878 180 85.05112878"
P900913="+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs"

Use http://trac.osgeo.org/gdal/wiki/ConfigOptions#OGR_ENABLE_PARTIAL_REPROJECTION and clip source to include only data within spherical mercator world square. Encoding conversion will only work as of GDAL 1.9.x. Also note -segmentize 1 adds vertex every 1 degree distance on lines for better reprojection fidelity.

ogr2ogr \
    --config OGR_ENABLE_PARTIAL_REPROJECTION TRUE --config SHAPE_ENCODING WINDOWS-1252 \
    -t_srs "+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 \
    +units=m +nadgrids=@null +no_defs" -lco ENCODING=UTF-8 \
    -clipsrc -180 -85.05112878 180 85.05112878 -segmentize 1 -skipfailures \
    outfile.shp infile.shp

Other data

Data conversion between formats

Add a custom footer
Add a custom sidebar
Clone this wiki locally