Proxito: query the DB instead of storage when serving robots.txt

[stsewd]

What's the problem this feature will solve?

Similar to #10512, we could hit the DB instead of storage to check if this file exists.

readthedocs.org/readthedocs/proxito/views/serve.py

Lines 957 to 963 in f90a387

	response = self._serve_docs(
	request=request,
	project=project,
	version=version,
	filename="robots.txt",
	check_if_exists=True,
	)

Currently, this isn't possible, since we only track .html files in our DB.

Describe the solution you'd like

We need to start tracking this file for new projects, and we need to iterate over all previous projects to check if this file exists in storage or not, and create the corresponding entry in the DB

Alternative solutions

I don't think this view is expensive, it's just one call to storage, and it's cached, not like 404, that they result from any URL and can't be cached beforehand. So, maybe doing nothing should be fine too. We also need yet to see if this idea of querying the DB instead of storage doesn't destroy our DB.