500 error if sorted by a column not in the ?_col= list

[simonw]

For example: https://latest.datasette.io/fixtures/sortable?_sort_desc=sortable&_col=sortable_with_nulls

That's ?_sort_desc=sortable&_col=sortable_with_nulls

[simonw]

Traceback:

Traceback (most recent call last):
  File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 1264, in route_path
    response = await view(request, send)
  File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 134, in view
    return await self.dispatch_request(request)
  File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 91, in dispatch_request
    return await handler(request)
  File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 361, in get
    response_or_template_contexts = await self.data(request, **data_kwargs)
  File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 157, in data
    return await self._data_traced(request, default_labels, _next, _size)
  File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 633, in _data_traced
    prefix = rows[-2][sort or sort_desc]
IndexError: No item with that key

That's this code here:

datasette/datasette/views/table.py

Lines 631 to 645 in 7af67b5

	# If there's a sort or sort_desc, add that value as a prefix
	if (sort or sort_desc) and not is_view:
	prefix = rows[-2][sort or sort_desc]
	if isinstance(prefix, dict) and "value" in prefix:
	prefix = prefix["value"]
	if prefix is None:
	prefix = "$null"
	else:
	prefix = tilde_encode(str(prefix))
	next_value = f"{prefix},{next_value}"
	added_args = {"_next": next_value}
	if sort:
	added_args["_sort"] = sort
	else:
	added_args["_sort_desc"] = sort_desc

[simonw]

So the problem here is that in generating the ?_next= next page link we need the value from the specified sort column - but we're not selecting it any more.

Possible fixes:

• Always include the sort column in the list of columns that are selected, then filter that out before they are displayed
• Use a second query to figure out the _sort or _sort_desc value for that last row, since we know its primary key (we always select primary keys)

Not sure which solution is more elegant. I think it might be the second one.

[simonw]

So code would look something like this:

try:
    prefix = rows[-2][sort or sort_desc]
except KeyError:
    # Didn't select sort/sort_desc column - look up value by primary key instead
    primary_key = rows[-2]["pk"] # But more complex than this
    prefix = (await db.execute("select * from {table} where pk = ?", [primary_key])).first_value()

[simonw]

https://latest.datasette.io/fixtures/sortable?_sort_desc=sortable&_col=sortable_with_nulls works now.