Headers included in a fetch()

[annevk]

(This does not strictly affect SW, but does affect the fetch() API.)

At the moment http://fetch.spec.whatwg.org/#http-network-or-cache-fetch is unclear with respect to which headers are and can be included by user agents.

E.g. for XMLHttpRequest user agents will include cache related headers at that point and also, for responses, will transform 304 into a 200. Ideally we reduce the vagueness and perhaps provide options.

[domenic]

I want to describe Node's HTTP client APIs, which are very low-level (although not completely, as we'll see). I don't think they made the best choices in all cases, but it's a good data point. I put together a gist to show how I obtained these results.

Basically, it only adds the host and connection headers. (The connection header will be either 'close' or 'keep-alive' depending.) For requests with a body, it adds transfer-encoding: chunked, unless you have set content-length.

It never does anything cache-related, leaving that to user-space.

---

IMO the caching things sucks in Node. There are many use cases for a transparent HTTP cache that Just Works in the manner XMLHttpRequest does, and not having one in Node has really held us back and created lots of hacky ad-hoc half-solutions (e.g. npm/npm#4159). So there needs to be a very easy way of turning a good transparent cache on, and I think it should be on by default.

Beyond that, I'm not sure what else we'd contemplate including. What does XHR include? User-Agent, I know.

/cc @mikeal who maintains the very popular request library, which is basically Node's version of the fetch API, 3 years later after accreting every possible feature known to man ;). I think he could give some value into how this would work on the web platform.

[annevk]

• Accept
• Accept-Encoding
• Accept-Language
• Connection
• Cookie
• DNT
• Host
• Referer
• User-Agent

Anything from http://fetch.spec.whatwg.org/#forbidden-header-name minus obsolete could be there too, but cache headers are sometimes added too. Varies a bit per browser. Would have to carefully study the networking libraries to know more. @bzbarsky @sicking @wanderview may know more about that.

[annevk]

Accept/Accept-Encoding/Accept-Language could probably be opt-in. Cookie is opt-in through a setting on Request.

Origin would also be included for cross-origin fetches (seems it isn't always for same-origin yet).

[domenic]

It's a shame you can't control several of those, like User-Agent or Referer. Lots of servers sniff on those to discriminate against web clients, so you have to go through a proxy.

[annevk]

Omitting Referer we should allow I think (see rel=noreferrer). Not sure about User-Agent control. Note that a proxy is often needed anyway if you want to read the response.

[hsivonen]

Accept/Accept-Encoding/Accept-Language could probably be opt-in.

Making Accept and Accept-Language opt-in makes sense to me, since they are about negotiating things based on the user's use case (Accept PDF vs. Word file) or the user's capabilities (Accept-Language) and, therefore, IMO shouldn't be part of this layer in the protocol stack anyway.

Accept-Encoding, however, is about negotiating compression, which is a lossless transform to optimize bandwidth. In that sense, it seems more like something that should belong to the browser's networking stack like Host and Connection.

[sicking]

I agree with Henri on Accept*.

Accept-Encoding/Connection/DNT/Host

should probably be considered part of the network layer. I wouldn't be surprised if SPDY/HTTP2 changes this list. Over time I would expect this list to change. I think it'd be a bad idea to hardcode this list too hard into the fetch spec, this seems like part of http.

Cookies and auth headers probably needs to be sent by default for same-origin in order to keep things consistent with existing APIs. But it should be possible to opt out of them.

I'm not sure how I feel about User-Agent header. It seems like it should be sent by default for the same reasons of cookies/auth. But I don't see any obvious harm in allowing it to be overridden. But it would need careful consideration I think. Probably deserves a separate thread of its own.

Regarding caching, like @domenic I think it should be on by default. But it should be easy to disable without having to learn about the various HTTP headers that are involved. I.e. basically something like a "bypass-cache: true" flag which would act as if no UA cache existed.

[sicking]

Actually, with regards to caching, there's probably several interesting modes:

• Use cache as normally
• Don't use any locally cached data, but if possible, store the response in cache
• Completely bypass cache

[annevk]

I spun of two items into new tickets. I guess the only remaining thing here is fixing the prose about what user agents can set in the network & cache layer.

[annevk]

Review of that change would be appreciated. I'd like to say things a bit more definitive, but not sure how.

[annevk]

This is done from the perspective of the specification, as far as I can tell.

[KenjiBaheux]

Tracked at crbug.com/423253