Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pre-Render? #15

Closed
gareth-johnstone opened this issue Jan 23, 2019 · 10 comments
Closed

Pre-Render? #15

gareth-johnstone opened this issue Jan 23, 2019 · 10 comments

Comments

@gareth-johnstone
Copy link

Is there a way to pre-render the tiles so they are not rendered on the fly?

@Overv
Copy link
Owner

Overv commented Jan 23, 2019

The only way that I'm aware of is to write a script to query all tile images within a region. You can write a script in Python with requests and some loops, for example, to force all tiles to be rendered. This is how I've done it for my own website which requires the entire Netherlands region and it works fairly well.

@Istador
Copy link
Contributor

Istador commented Jan 24, 2019

Or you can use existing programs / scripts for that.

render_list (with --all) is already installed with mod_tile, which you feed by upper and lower x, y and z tile coordinates to render.

And render_list_geo.pl is very useful if you have geo coordinates instead of tile coordinates of your desired area (it's not already installed within the image so you have to insert/install it yourself).

Also interesting is tile expiration to re-render existing tiles (e.g. in an productive environment). You can specify which zoom levels to re-render directly, or later when they are touched by a user.

@gareth-johnstone
Copy link
Author

@Istador total noob question (im new to docker) how do i access render_list within the docker? (like where is its path) :)

@gareth-johnstone
Copy link
Author

EDIT: kinda found it, but getting the following:

root@demo:/var/lib/docker/overlay2/1621bc07c2f86c269a1eb1ae2607549048074f9f57900ec6216ddde91ddd5f55/merged/home/renderer/src/mod_tile# ./render_list
ERROR: init_storage_backend: Failed to stat /var/lib/mod_tile with error: No such file or directory
Failed to initialise storage backend /var/lib/mod_tile

@Overv
Copy link
Owner

Overv commented Jan 24, 2019

You want to go into the container instead of finding its file system on your local hard drive. Try the following while your container is running:

docker ps       <- get the container name
docker exec -it container-name-here bash

You will then be able to execute render_list from within the container itself where /var/lib/mod_tile does exist.

@gareth-johnstone
Copy link
Author

You want to go into the container instead of finding its file system on your local hard drive. Try the following while your container is running:

docker ps       <- get the container name
docker exec -it container-name-here bash

You will then be able to execute render_list from within the container itself where /var/lib/mod_tile does exist.

you sir, are a legend, thank you :)

@nielsole
Copy link
Collaborator

If you use render_list or render_list_geo, make sure to use -m ajt as that is the database name.

@dkhovanskii
Copy link

dkhovanskii commented Jan 4, 2022

Hi everyone,
I am trying to set up a server and run into a problem that I cannot solve for a couple of weeks. I successfully imported planet file, but when I am trying to run it it doesn't render tiles. I assumed that it is just to slow to render. So, I decided pre-render tiles but it doesn't seems to work either. I am using version v1.8.2:

docker run -d -e THREADS=24 -e "OSM2PGSQL_EXTRA_ARGS=-C 24000" --shm-size="512m" \
	--restart unless-stopped \
	-p 8080:80 --name osm -e ALLOW_CORS=enabled -e UPDATES=enabled \
	-v /home/data/osm/nodes:/nodes \
	-v /home/data/osm/data:/var/lib/postgresql/12/main \
	-v /home/data/osm/tiles:/var/lib/mod_tile \
	overv/openstreetmap-tile-server:v1.8.2 run

it starts fine but nothing rendered.

docker exec -it osm /bin/bash
then
/usr/local/bin/render_list -m ajt -a -z 0 -Z 4 -n 24
The output looks like this:

root@358e72bb0bfc:/# /usr/local/bin/render_list -m ajt -a -z 0 -Z 4 -n 24
debug: init_storage_backend: initialising file storage backend at: /var/lib/mod_tile
Rendering client
Starting 24 rendering threads
Rendering all tiles from zoom 0 to zoom 4
Rendering all tiles for zoom 0 from (0, 0) to (0, 0)
Rendering all tiles for zoom 1 from (0, 0) to (1, 1)
Rendering all tiles for zoom 2 from (0, 0) to (3, 3)
Rendering all tiles for zoom 3 from (0, 0) to (7, 7)
Rendering all tiles for zoom 4 from (0, 0) to (15, 15)
Waiting for rendering threads to finish
rendering failed with command 4, pausing.
rendering failed with command 4, pausing.
rendering failed with command 4, pausing.
rendering failed with command 4, pausing.
rendering failed with command 4, pausing.
rendering failed with command 4, pausing.
rendering failed with command 4, pausing.
rendering failed with command 4, pausing.

*****************************************************
*****************************************************
Total for all tiles rendered
Meta tiles rendered: Rendered 8 tiles in 10.00 seconds (0.80 tiles/s)
Total tiles rendered: Rendered 512 tiles in 10.00 seconds (51.18 tiles/s)
Total tiles handled: Rendered 8 tiles in 10.00 seconds (0.80 tiles/s)
*****************************************************
*****************************************************
*****************************************************

Docker container logs contains a few errors, but I am not sure if it is a sign of a main issue, because on an instance I set up 2 years ago I had similar issues, but it works fine. Docker logs:

.....
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
  encountered during parsing of layer 'icesheet-poly' in Layer at line 5911 of '/home/renderer/src/openstreetmap-carto/mapnik.xml'
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
  encountered during parsing of layer 'icesheet-poly' in Layer at line 5911 of '/home/renderer/src/openstreetmap-carto/mapnik.xml'
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
  encountered during parsing of layer 'icesheet-poly' in Layer at line 5911 of '/home/renderer/src/openstreetmap-carto/mapnik.xml'
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
  encountered during parsing of layer 'icesheet-poly' in Layer at line 5911 of '/home/renderer/src/openstreetmap-carto/mapnik.xml'
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
  encountered during parsing of layer 'icesheet-poly' in Layer at line 5911 of '/home/renderer/src/openstreetmap-carto/mapnik.xml'
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
  encountered during parsing of layer 'icesheet-poly' in Layer at line 5911 of '/home/renderer/src/openstreetmap-carto/mapnik.xml'
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
  encountered during parsing of layer 'icesheet-poly' in Layer at line 5911 of '/home/renderer/src/openstreetmap-carto/mapnik.xml'
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
  encountered during parsing of layer 'icesheet-poly' in Layer at line 5911 of '/home/renderer/src/openstreetmap-carto/mapnik.xml'
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
  encountered during parsing of layer 'icesheet-poly' in Layer at line 5911 of '/home/renderer/src/openstreetmap-carto/mapnik.xml'
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
  encountered during parsing of layer 'icesheet-poly' in Layer at line 5911 of '/home/renderer/src/openstreetmap-carto/mapnik.xml'
renderd[148]: Updating max_connection parameter for mapnik layers to reflect thread count
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
renderd[148]: Updating max_connection parameter for mapnik layers to reflect thread count
renderd[148]: Updating max_connection parameter for mapnik layers to reflect thread count
renderd[148]: Updating max_connection parameter for mapnik layers to reflect thread count
renderd[148]: Updating max_connection parameter for mapnik layers to reflect thread count
renderd[148]: Updating max_connection parameter for mapnik layers to reflect thread count
renderd[148]: Updating max_connection parameter for mapnik layers to reflect thread count
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
renderd[148]: Updating max_connection parameter for mapnik layers to reflect thread count
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
renderd[148]: Updating max_connection parameter for mapnik layers to reflect thread count
renderd[148]: Updating max_connection parameter for mapnik layers to reflect thread count
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
renderd[148]: Updating max_connection parameter for mapnik layers to reflect thread count
renderd[148]: An error occurred while loading the map layer 'ajt': Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
renderd[148]: DEBUG: Got incoming connection, fd 5, number 1
renderd[148]: DEBUG: Got incoming connection, fd 6, number 2
renderd[148]: DEBUG: Got incoming connection, fd 7, number 3
renderd[148]: DEBUG: Got incoming connection, fd 8, number 4
renderd[148]: DEBUG: Got incoming connection, fd 9, number 5
renderd[148]: DEBUG: Got incoming connection, fd 10, number 6
renderd[148]: DEBUG: Got incoming connection, fd 11, number 7
renderd[148]: DEBUG: Got incoming connection, fd 12, number 8
renderd[148]: DEBUG: Got incoming connection, fd 13, number 9
renderd[148]: DEBUG: Got incoming connection, fd 14, number 10
renderd[148]: DEBUG: Got incoming connection, fd 15, number 11
renderd[148]: DEBUG: Got incoming connection, fd 16, number 12
renderd[148]: DEBUG: Got incoming connection, fd 17, number 13
renderd[148]: DEBUG: Got incoming connection, fd 18, number 14
renderd[148]: DEBUG: Got incoming connection, fd 21, number 15
renderd[148]: DEBUG: Got incoming connection, fd 22, number 16
renderd[148]: DEBUG: Got incoming connection, fd 23, number 17
renderd[148]: DEBUG: Got incoming connection, fd 24, number 18
renderd[148]: DEBUG: Got incoming connection, fd 25, number 19
renderd[148]: DEBUG: Got incoming connection, fd 26, number 20
renderd[148]: DEBUG: Got incoming connection, fd 27, number 21
renderd[148]: DEBUG: Got incoming connection, fd 28, number 22
renderd[148]: DEBUG: Got incoming connection, fd 29, number 23
renderd[148]: DEBUG: Got incoming connection, fd 30, number 24
renderd[148]: DEBUG: Got incoming request with protocol version 2
renderd[148]: DEBUG: Got command RenderBulk fd(29) xml(ajt), z(0), x(0), y(0), mime(image/png), options()
renderd[148]: DEBUG: Got incoming request with protocol version 2
renderd[148]: DEBUG: Got command RenderBulk fd(6) xml(ajt), z(1), x(0), y(0), mime(image/png), options()
renderd[148]: DEBUG: Got incoming request with protocol version 2
renderd[148]: Received request for map layer 'ajt' which failed to load
renderd[148]: DEBUG: Got command RenderBulk fd(8) xml(ajt), z(2), x(0), y(0), mime(image/png), options()
renderd[148]: Received request for map layer 'ajt' which failed to load
renderd[148]: DEBUG: Sending render cmd(4 ajt 0/0/0) with protocol version 2 to fd 29
renderd[148]: DEBUG: Sending render cmd(4 ajt 1/0/0) with protocol version 2 to fd 6
renderd[148]: DEBUG: Connection 0, fd 5 closed, now 23 left
renderd[148]: Received request for map layer 'ajt' which failed to load
renderd[148]: DEBUG: Connection 1, fd 7 closed, now 22 left
renderd[148]: DEBUG: Got incoming request with protocol version 2
renderd[148]: DEBUG: Sending render cmd(4 ajt 2/0/0) with protocol version 2 to fd 8
renderd[148]: DEBUG: Got command RenderBulk fd(9) xml(ajt), z(3), x(0), y(0), mime(image/png), options()
renderd[148]: DEBUG: Connection 3, fd 10 closed, now 21 left
renderd[148]: DEBUG: Connection 4, fd 12 closed, now 20 left
renderd[148]: DEBUG: Got incoming request with protocol version 2
renderd[148]: DEBUG: Got command RenderBulk fd(14) xml(ajt), z(4), x(0), y(8), mime(image/png), options()
renderd[148]: Received request for map layer 'ajt' which failed to load
renderd[148]: DEBUG: Got incoming request with protocol version 2
renderd[148]: Received request for map layer 'ajt' which failed to load
renderd[148]: DEBUG: Sending render cmd(4 ajt 3/0/0) with protocol version 2 to fd 9
renderd[148]: DEBUG: Sending render cmd(4 ajt 4/0/8) with protocol version 2 to fd 14
renderd[148]: DEBUG: Got command RenderBulk fd(15) xml(ajt), z(4), x(8), y(8), mime(image/png), options()
renderd[148]: DEBUG: Connection 7, fd 16 closed, now 19 left
renderd[148]: DEBUG: Connection 8, fd 18 closed, now 18 left
renderd[148]: Received request for map layer 'ajt' which failed to load
renderd[148]: DEBUG: Connection 9, fd 22 closed, now 17 left
renderd[148]: DEBUG: Connection 10, fd 24 closed, now 16 left
renderd[148]: DEBUG: Sending render cmd(4 ajt 4/8/8) with protocol version 2 to fd 15
renderd[148]: DEBUG: Connection 11, fd 26 closed, now 15 left
renderd[148]: DEBUG: Got incoming request with protocol version 2
renderd[148]: DEBUG: Got command RenderBulk fd(28) xml(ajt), z(4), x(8), y(0), mime(image/png), options()
renderd[148]: DEBUG: Connection 14, fd 30 closed, now 14 left
renderd[148]: DEBUG: Got incoming request with protocol version 2
renderd[148]: Received request for map layer 'ajt' which failed to load
renderd[148]: DEBUG: Got command RenderBulk fd(11) xml(ajt), z(4), x(0), y(0), mime(image/png), options()
renderd[148]: DEBUG: Sending render cmd(4 ajt 4/8/0) with protocol version 2 to fd 28
renderd[148]: DEBUG: Connection 4, fd 13 closed, now 13 left
renderd[148]: Received request for map layer 'ajt' which failed to load
renderd[148]: DEBUG: Sending render cmd(4 ajt 4/0/0) with protocol version 2 to fd 11
renderd[148]: DEBUG: Connection 6, fd 17 closed, now 12 left
....

What am I doing wrong here?

P.S. Hardware: HP BL460c G8, 2 x E5-2690; RAM 128Gb; SSD 1920Gb

@nielsole
Copy link
Collaborator

nielsole commented Jan 5, 2022

As you are likely unable to generate any tiles even on the fly this is unrelated to the issue. Nonetheless this is likely the step that failed:

sudo -E -u renderer python3 /home/renderer/src/openstreetmap-carto/scripts/get-external-data.py -c /home/renderer/src/openstreetmap-carto/external-data.yml -D /home/renderer/src/openstreetmap-carto/data

@dkhovanskii
Copy link

dkhovanskii commented Jan 6, 2022

Thank you very much for this hint @nielsole ! It was the issue. When I run this command then recreated a container it started to work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants