Mechanism for adding arbitrary pages like /about

[simonw]

For www.niche-museums.com I solved this by creating an empty about.db database file - see https://simonwillison.net/2019/Nov/25/niche-museums/

I want a neater mechanism for this.

[simonw]

Idea: do this with a simple template naming convention.

If you hit /about and there is no matching database, check for a template file called about-page.html. If it exists, render it. Otherwise return a 404 database not found.

[simonw]

Alternative idea: a new concept of "pages" which live inside templates/pages/ and where the file name minus the .html extension defines the URL.

templates/about/me.html would be served at /about/me - but only if no matching database and table were found.

This only takes effect on 404 errors from core Datasette.

[simonw]

Stretch goal: it would be neat if these pages could return custom HTTP headers (eg content-type) and maybe even status codes (eg for redirects) somehow.

[simonw]

The implementation in c5e8cd8 acts as a proof of concept. It has a big flaw though: it doesn't reuse the regular render() mechanism, which means it doesn't register custom template tags from plugins.

This is bad because it means that pages rendered in this way cannot take advantage of things like datasette-template-sql.

This means this issue is likely dependent on #577 - a documented mechanism to allow plugins to render templates.

[simonw]

Now that I've closed #577 this should be pretty easy to implement.

[simonw]

Stretch goal: it would be neat if these pages could return custom HTTP headers (eg content-type) and maybe even status codes (eg for redirects) somehow.

I think I could do that with a custom template function - if that function is called during the render then we follow those instructions instead of returning the rendered HTML.

[simonw]

The trickiest part here is the 404 logic. It's spread out through a couple of places right now. It's in datasette.utils.asgi.AsgiRouter:

datasette/datasette/utils/asgi.py

Lines 98 to 134 in 227bb3e

	class AsgiRouter:
	def __init__(self, routes=None):
	routes = routes or []
	self.routes = [
	# Compile any strings to regular expressions
	((re.compile(pattern) if isinstance(pattern, str) else pattern), view)
	for pattern, view in routes
	]
	async def __call__(self, scope, receive, send):
	# Because we care about "foo/bar" v.s. "foo%2Fbar" we decode raw_path ourselves
	path = scope["path"]
	raw_path = scope.get("raw_path")
	if raw_path:
	path = raw_path.decode("ascii")
	return await self.route_path(scope, receive, send, path)
	async def route_path(self, scope, receive, send, path):
	for regex, view in self.routes:
	match = regex.match(path)
	if match is not None:
	new_scope = dict(scope, url_route={"kwargs": match.groupdict()})
	try:
	return await view(new_scope, receive, send)
	except Exception as exception:
	return await self.handle_500(scope, receive, send, exception)
	return await self.handle_404(scope, receive, send)
	async def handle_404(self, scope, receive, send):
	await send(
	{
	"type": "http.response.start",
	"status": 404,
	"headers": [[b"content-type", b"text/html"]],
	}
	)
	await send({"type": "http.response.body", "body": b"<h1>404</h1>"})

Then also in the `DatasetteRouter subclass of that:

datasette/datasette/app.py

Lines 736 to 757 in 227bb3e

	class DatasetteRouter(AsgiRouter):
	def __init__(self, datasette, routes):
	self.ds = datasette
	super().__init__(routes)
	async def route_path(self, scope, receive, send, path):
	# Strip off base_url if present before routing
	base_url = self.ds.config("base_url")
	if base_url != "/" and path.startswith(base_url):
	path = "/" + path[len(base_url) :]
	return await super().route_path(scope, receive, send, path)
	async def handle_404(self, scope, receive, send):
	# If URL has a trailing slash, redirect to URL without it
	path = scope.get("raw_path", scope["path"].encode("utf8"))
	if path.endswith(b"/"):
	path = path.rstrip(b"/")
	if scope["query_string"]:
	path += b"?" + scope["query_string"]
	await asgi_send_redirect(send, path.decode("latin1"))
	else:
	await super().handle_404(scope, receive, send)

[simonw]

Idea: handle_404 can hand certain not found errors off to handle_500 so it can render them correctly;

datasette/datasette/app.py

Lines 748 to 765 in 227bb3e

	async def handle_404(self, scope, receive, send):
	# If URL has a trailing slash, redirect to URL without it
	path = scope.get("raw_path", scope["path"].encode("utf8"))
	if path.endswith(b"/"):
	path = path.rstrip(b"/")
	if scope["query_string"]:
	path += b"?" + scope["query_string"]
	await asgi_send_redirect(send, path.decode("latin1"))
	else:
	await super().handle_404(scope, receive, send)
	async def handle_500(self, scope, receive, send, exception):
	title = None
	if isinstance(exception, NotFound):
	status = 404
	info = {}
	message = exception.args[0]
	elif isinstance(exception, DatasetteError):

[simonw]

Stretch goal: it would be neat if these pages could return custom HTTP headers (eg content-type) and maybe even status codes (eg for redirects) somehow.

I think I could do that with a custom template function - if that function is called during the render then we follow those instructions instead of returning the rendered HTML.

[simonw]

Documentation: https://datasette.readthedocs.io/en/latest/custom_templates.html#custom-pages

[simonw]

https://www.niche-museums.com/about now uses this mechanism. It used to be an empty about.db database but you can see from https://www.niche-museums.com/-/databases that it's now using the new mechanism.

[chekos]

Hi! I'm using datasette on this repository: https://github.com/chekos/RIPA-2018-datasette

and on my local machine i can see an /about page i created but when i deploy to heroku i get a 404 (http://ripa-2018-db.herokuapp.com)

I bumped datasette in my requirements file to 0.41 so I'm 100% what the issue is 🤔

Do you have any idea what could be the problem? 👀

EDIT: for context, I have a templates directory with a pages/about.html file in https://github.com/chekos/RIPA-2018-datasette/tree/master/datasette/templates

[chekos]

It seems that heroku wasn't updating to 0.41 on deployment.

Had to add --branch 0.41 and that solved it! Heroku caches dependencies

and (i think) because the requirements.txt doesn't specify the datasette version, it didn't update from 0.40 to 0.41 on heroku even though it was specified on my local requirements file as datasette >= 0.41

These are the lines that gave me an idea on how to solve it:

datasette/datasette/publish/heroku.py

Lines 164 to 186 in 182e5c8

	try:
	os.chdir(tmp.name)
	if metadata_content:
	open("metadata.json", "w").write(json.dumps(metadata_content, indent=2))
	open("runtime.txt", "w").write("python-3.8.0")
	if branch:
	install = [
	"https://github.com/simonw/datasette/archive/{branch}.zip".format(
	branch=branch
	)
	] + list(install)
	else:
	install = ["datasette"] + list(install)
	open("requirements.txt", "w").write("\n".join(install))
	os.mkdir("bin")
	open("bin/post_compile", "w").write(
	"datasette inspect --inspect-file inspect-data.json"
	)