Pagination header Content-Range / Range-Unit is not working

[rv2931]

Environment

• PostgreSQL version: Dokcer Postgres:10
• PostgREST version: postgrest/postgrest:v12.2.3
• Operating system: Windows/Podman

Description of issue

According to the documentation, it is possible to manage pagination with header following parameters:

• Range-Unit: items
• Content-Range: 0-14/*

Unfortunetly I'm trying to paginate with these parameters but the response contains all items and not only the 15th first

I've triued using query parameters limit & offset and these query parameters are working well

• How you make the request to PostgREST (curl command preferred).
• Here you have the curl command I use to test, I'm using postman with the same result
  curl -H 'Content-Range: 0-14/*' -H 'Range-Unit: items' http://localhost:3000/my_table_items
  this returns a non paginated answer with Content-Range: 0-394/395
• The PostgREST response.
  IS 200 OK but contains all items present in the my_table_items table

[wolfgangwalther]

Hm. The section you are referring to is this, I guess: https://docs.postgrest.org/en/v12/references/api/pagination_count.html#limits-and-pagination

This mentions explicitly that this the response format. You can expect the Content-Range header as a return. (This is the same with all Content-XX headers btw, you never send them in a request)

Further down, you can find the first mention of how to request that: https://docs.postgrest.org/en/v12/references/api/pagination_count.html#range-header

The header you need to send is Range.

[laurenceisla]

Can you cite that RFC exactly and where you think this is wrong?

Ah, it's mentioned in RFC 7233, we are tracking the issue in #1089. It's confirmed by Roy Fielding there: #1089 (comment)

[rv2931]

Yeah sure. Sorry I thought I joined the link
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range

Range: <unit>=<range-start>-
Range: <unit>=<range-start>-<range-end>
Range: <unit>=<range-start>-<range-end>, <range-start>-<range-end>, …
Range: <unit>=-<suffix-length>

[rv2931]

I found the same in officiel RFC 9110
https://www.rfc-editor.org/rfc/rfc9110.html#field.range
Range = ranges-specifier
https://www.rfc-editor.org/rfc/rfc9110.html#rule.ranges-specifier

  ranges-specifier = range-unit "=" range-set
  range-set        = 1#range-spec
  range-spec       = int-range
                   / suffix-range
                   / other-range

[wolfgangwalther]

Interesting. I read the "range-unit" part when I looked for the header and was too quick to note the difference between:

range-unit = ... ... and Range-Unit = .... The former exists, the latter doesn't.

I.e. the way this is written, it's just a definition of what range-unit means and not a new header definition. Those are capitalized.

[rv2931]

It seems to be an old discussion about rfc7233 compliance and breaking change (since 2022 and before)
Did you know if something is planned on this ? Even if doing something means accept both formats (Range-Unit: items+Range: x-y and Range: items=x-y) for a defined time/versions and warning user in some way this will become obsolete?

[jmealo]

@wolfgangwalther: TIL: It looks like the latest RFC / HTTP Semantics spec has some valid use cases for Content-Range in PUT request headers (provided there is a request body). It's apparently valid for a partial PUT to specify the ranges being provided in the body. This doesn't apply here, but it's interesting none-the-less.
https://www.rfc-editor.org/rfc/rfc9110.html#section-14.5

[wolfgangwalther]

Right, I wasn't precise enough: Content-XXX headers are always about the body that you are sending. So when you are making a request, you can say Content-Type to describe the mime type of the body you are sending. The server will do the same. The same applies to Content-Range.

[foxydevloper]

I ran into this issue too but I also tried using the Range header and it does not work.
curl https://example.com/postgrest -H "range-unit: items" -H "range: 0-1" just ignores it and gives me all results in the table

[foxydevloper]

Is the range header meant to work? I've never had a situation where I tried it and it worked, does it work for anyone else?

[rv2931]

My bad, I meant to have /table, I was just showcasing an example because I didn't want to be specific about my tables This is what I ran.

curl https://mysite.com/api/pgrst/mytable -H "Authorization: Bearer ${JWT}" -I -H "Range: 0-10" -H "Range-Unit: items"
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
content-location: /mytable
content-range: 0-99/*
server-timing: jwt;dur=0.0, parse;dur=0.1, plan;dur=0.1, transaction;dur=1.1, response;dur=0.0

How many items have you in table "mytable" ?
Values are confusing you're right but 200 OK should means that all results are returned in response so the content-range becomes useless I think, even if we could expect for content-range: 0-10/*

I confused for the /* in addition. I get it for many range request without having the" total count" result in place of the "*" so it seems not to be compliant with the postgrest documentation and what every one could expect

[foxydevloper]

My bad, I meant to have /table, I was just showcasing an example because I didn't want to be specific about my tables This is what I ran.

curl https://mysite.com/api/pgrst/mytable -H "Authorization: Bearer ${JWT}" -I -H "Range: 0-10" -H "Range-Unit: items"
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
content-location: /mytable
content-range: 0-99/*
server-timing: jwt;dur=0.0, parse;dur=0.1, plan;dur=0.1, transaction;dur=1.1, response;dur=0.0

How many items have you in table "mytable" ? Values are confusing you're right but 200 OK should means that all results are returned in response so the content-range becomes useless I think, even if we could expect for content-range: 0-10/*

I confused for the /* in addition. I get it for many range request without having the" total count" result in place of the "*" so it seems not to be compliant with the postgrest documentation and what every one could expect

Around 3 million rows. I tested Range on another postgrest instance I run for a database with 650 million rows, and it worked on that instance. That postgrest instance runs on v11.2.2, so maybe it's a regression.
Update: nope, not a regression. I ran a new postgrest instance pointing to the same database with v12.2.3 and it works fine. So it has to be something wrong with my configuration or database that is causing Range not to work.

The database that range works on is PostgreSQL 14, while the database range doesn't work on is PostgreSQL 16

[wolfgangwalther]

Is the range header meant to work? I've never had a situation where I tried it and it worked, does it work for anyone else?

I have never had a situation where it didn't work. So yes, it should work.

Do you have reverse proxy in front of PostgREST which might be doing bad things to those headers?

[foxydevloper]

Is the range header meant to work? I've never had a situation where I tried it and it worked, does it work for anyone else?

I have never had a situation where it didn't work. So yes, it should work.

Do you have reverse proxy in front of PostgREST which might be doing bad things to those headers?

Yes, I'm using cloudflare, that's likely the cause. It probably messes with the Range header. I'll test calls to postgrest directly without cloudflare to confirm
update: definitely cloudflare, testing it directly solved the problem. I tested cloudflare using a header echo webserver and cloudflare just erases the Range header sent to the backend. I'll likely have to set up transform rules to bypass cloudflare messing with the range header, or just only use query parameters