We "need" a way to create an Http2Session from a socket

[grantila]

While building fetch-h2 and when discussing with node-fetch, and while participating in #14671 I realize that we need this in Node.js; a protocol-agnostic connect() to a TLS-enabled web server and let the ALPN handshake allow for both "h2" and "http/1.1". Without this, we won't be able to connect to an https-server without knowing in beforehand if that server wants to (or even can!) speak HTTP/1.1 or HTTP/2.

I'm thinking of something like tls.handoverConnect() which doesn't take application-level (http) arguments (path, headers, etc), it just connects and returns (through a callback, likely) the protocol ("http/1.1" or "h2") and a corresponding session.

A question is what the session data is. In case of HTTP/2, it should be an Http2Session, but for HTTP/1.1 it must be something where the user can perform the request, and it should probably be integrated with Agents. There's unfortunately an inheritance jungle in the design of these old functions, like https.request() where ip/tcp/application-level options are merged into one blob, but the "https/1.1 session" here (let's call it Http1SessionProxy) should be something that looks like the http module, in how it has a request() function, with the tcp/ip-level options removed.

Let's say this feature belongs in tls (which I think it does):

const connectOpts = { host: "my-server.com", port: 443, /* ... */ };

tls.handoverConnect( connectOpts, ( err, protocol ) =>
{
    if ( err ) ... ;

    // session is an Http2Session or Http1SessionProxy
    const { name, session } = protocol;

    if ( name === 'http/1.1' )
    {
        // session.request acts just like http.request
        const req = session.request(
            { path: '/foo', headers, ... },
            ( res ) => { ... }
        );
    }
    else if ( name === 'h2' )
    {
        // well, the usual http2-stuff
        const stream = session.request( headers, options );
    }
} );

In this example, perhaps an Agent could be passed in the connectOpts, so that if the session turns out to be HTTP/1.1, the connection will be added to this agent (and by default the global Agent is used, just like traditional http{s}.request()).

Any thoughts on this? I guess it's a good time to look into this now, as the http2 module is being tested in the wild, but before it's "stable".

[addaleax]

It’s not 100 % clear what you are asking for, but this feels like something one should already be able to do using existing APIs…

Just to clarify, the TLS module’s connect() method is protocol-agnostic, and you can pass it a list of supported ALPN protocols, and once connected tlsSocket.alpnProtocol will contain the negotiated protocol.

[grantila]

Right. After that callback you have two things: A socket and a protocol name. How do you then go about to turn this socket into an Http2Session (or something on which you can perform HTTP/1.1 requests)? That's what I'm after. Those things seem very internal to Node.js and not exposed by the current API's (disclaimer: I might have overlooked them).

[addaleax]

Right – I think it should be possible to create HTTP or HTTP2 sessions from arbitrary streams, or at least I think the internals structures of HTTP were made with such a use case in mind…

Edit: I’ll definitely be looking into this later today and make sure

[grantila]

Sounds great, I didn't think about it, but perhaps this already works for http1 by setting the createConnection property to a function returning an already existing socket. It sure sounds like a hack, and perhaps the option to http.request should take a socket property instead.

For http2, I'd suggest allowing http2.connect to take a socket as first "authority" argument too (not just string or URL).

It does sound like this could be a pretty simple fix with a huge impact, and given the bug #14671 I think this issue is more important to look at, since by allowing the user to first perform TLS handshake and then handover to http2, #14671 is less of a problem (and perhaps even automatically fixed as well).

[apapirovski]

The connect implementation in http2 is currently very bare-bones, basically enough to do some fetching and run our tests. Any PRs to improve it are very welcome. I don't think we ultimately expect user-code to use tls.connect if they want https & http2.

But if you were trying to do that right now, you would need tls.connect(), wait for the negotiation to happen and then create the Http2Session within that listener (3rd argument is socket).

(No one has really shown that #14671 is a real bug. As far as I can tell people are just not passing in the right options to connect to a secure server so their client is rejecting the connection. http2 literally just defers to tls so I don't really see why there would be any bugs there.)

[grantila]

@apapirovski How do I create an Http2Session from a socket?

Did you read my comment in #14671? If you still mean that I haven't shown that it's a real bug, please explain there what I do wrong.

[apapirovski]

So... we don't actually export Http2Session at the moment which means this isn't really possible but if we did, it would look something like this:

const https = require('https');
const {
  constants,
  Http2Session
} = require('http2');
const tls = require('tls');

const tlsConnection = tls.connect({
  host: 'nghttp2.org',
  port: 443,
  ALPNProtocols: ['h2', 'http/1.1']
});
tlsConnection.once('secureConnect', () => {
  if (tlsConnection.alpnProtocol === 'h2')
    withHttp2(tlsConnection);
  else if (tlsConnection.alpnProtocol === 'http/1.1')
    withHttp1(tlsConnection);
  else
    throw new Error('no valid protocol negotiated');
});

function withHttp1(socket) {
  const req = https.request({
    host: 'nghttp2.org',
    path: '/httpbin/ip',
    port: 443,
    createConnection: () => socket
  }, (res) => console.log(res.headers));
  req.end();
}

function withHttp2(socket) {
  const client = new Http2Session(constants.NGHTTP2_SESSION_CLIENT,
                                  {},
                                  socket);
  const req = client.request({ ':path': '/httpbin/ip' });
  req.on('response', (headers) => console.log(headers));
  req.end();
}

If you feel like it, you could probably extract parts of this to change connect to support the allowHTTP1 setting and open a PR. :)

[addaleax]

@apapirovski @grantila Fwiw, I’ve opened #16267 and #16269 with the goal of making sure HTTP and HTTP/2 can use arbitrary duplex streams. For net.Sockets and maybe TLS socket as well, most things here should already work out of the box (the exception being the HTTP/2 client side due to lack of a socket injection API).

I wouldn’t consider these to be a final API for solving the particular problem you are bringing up, though.

[jasnell]

HTTP/2 is already set up to use arbitrary StreamBase implementations, we just do not currently expose the Http2Session class from core to make this easy. This is largely because it is still experimental and we're still working on getting the base functionality correct.

There are a couple of gotchas, however... notice that I said StreamBase implementation above. This is because all of the data flow to and from the socket occurs within the C/C++ layer. This means that it will not be possible to pull data from an arbitrary JS-only Duplex because once the Http2Stream takes over, the js interface is ignored and data is written and read directly using the StreamBase APIs.

It is implemented that way for performance reasons and there would be no justification at all for changing that.

That said, the Http2Session class is already implemented to allow it to wrap and consume a socket. What is being asked for in this issue would theoretically be possible to do.

Take a look here https://github.com/nodejs/node/blob/master/lib/internal/http2/core.js#L700 and here https://github.com/nodejs/node/blob/master/lib/internal/http2/core.js#L700

The logic within connectionListener essentially glues the socket and the Http2Session together, setting appropriate properties and event listeners on the socket, and creating the Http2Session instance which completely takes over ownership of the socket. We could provide a low level API that encapsulates these steps such that a user's own connection listener can hand off the socket to an Http2Session. That would be relatively trivial to do (it would essentially just be exporting the connectionListener itself as an API... allowing you to do something along the lines of:

const { connectionListener } = require('http2').connectionListener;
const server = net.createServer((socket) => {
  connectionListener(socket);
});

As long as socket was a StreamBase object whose JS api implements Duplex, the connectionListener should just do the right thing.

Now, if you're talking about the client side, that works just about the same way. See the code here: https://github.com/nodejs/node/blob/master/lib/internal/http2/core.js#L2412-L2460

This is a small variation on the same theme with a few different requirements. Again, however, the Http2Session takes over complete control over the socket and the socket must be a StreamBase object. We could extract the setup logic into a utility function that would allow other sockets to be wrapped.

[addaleax]

@jasnell did you see #16269? That should be “would theoretically be possible” in practice :)

[jasnell]

looking...

[addaleax]

Not sure, but I think what this issue essentially boils down to is that https.connect should, possibly optionally and/or transparently, support HTTP/2. Does that sound about right?

[apapirovski]

Yes but it probably makes more sense for http2.connect to support downgrading to h1 given that some of the semantics around how it works are different from http & https.

[addaleax]

@apapirovski I’m not sure on this either, but I think what users would expect is that https.connect should be able to handle HTTP/2, if only because that’s how browsers deal with it (i.e. it’s sufficiently transparent whether HTTP/2 is used or not)

[apapirovski]

It's tricky. Personally, I would have a big concern around the user-friendliness of that because there are major differences. A user that enables something like allowHTTP2 might've not familiarized themselves with those and might just assume that it works the same (or similar enough). Whereas anyone that's using http2.connect likely has read at least some of that documentation.

Otherwise we might need to implement a whole other compatibility layer for the client side (like we did for server).

Also, there probably aren't very many cases of servers that only support h2 (so someone using https will never — at least in the medium to long term — have their code stop working) whereas there are ones that only support h1.

But that's just my current feelings on this... could be way off base.

[jasnell]

would expect is that https.connect should be able to handle HTTP/2

In theory, yes, and it's likely something we should work towards. We didn't do this yet for a few reasons: (a) we needed the core http/2 functionality done first, (b) we wanted to implement it in a way that intentionally did not touch the existing http/1 implementation and (c) there's a bit of work necessary to figure out how to enable this transparently.

There are a couple of issues with making this all work transparently. The Agent model currently used by the http/1 clients (with or without the s on the end) is not appropriate for the long lived http/2 connection model. One would need to come up with a reasonable compatibility API implementation as @apapirovski suggests in order to make it work. Second, the benefits are somewhat questionable. That is, realistically, how likely is it that someone is going to create a client that needs to speak either HTTP/1 or HTTP/2 and is it possible to do that check in userland instead? Perhaps all we need to do here is expose the ClientHttp2Session constructor so that if someone wanted to transition between the two they could, without us providing the actual implementation to do so.

If someone wanted to spend some time coming up with a reasonable proposal, I certainly wouldn't be opposed to it tho.

[grantila]

would expect is that https.connect should be able to handle HTTP/2

I would say no, not necessarily. Why not let other packages handle this (like the fetch-h2 I'm writing, the existing node-fetch or the relatively popular request)? I don't think it makes sense for these compatibility layers, they really add no value. Sure, node could itself have a generic httpAny.request(), but HTTP/1 and HTTP/2 are radically different, and ignoring this would make people use it and not understand what really happens; what real objects they get back and how sessions/agents automagically work.

Can someone tell me what's wrong with my initial proposal? It separates connecting to a TLS server from the handling of HTTP/n protocols, and instead delegates this logic to the user instead. The whole issue with StreamBase and sockets become transparent to the user (if this is problematic), as well as the problem with exposing Http2Session constructors since you get pre-constructed sessions back. No one would need to construct these objects themselves, or to use the createConnection callback hack (which bypasses Agents). If this idea has flaws, let's discuss them; I'd just like to really rule out this option with arguments before this issue turns into something else.

Disclaimer; a) The name handoverConnect is maybe bad. b) This title should maybe be changed back to something like its original, or: "Proposal: A session-constructing, protocol-agnostic TLS-connect()"

[apapirovski]

This can now be accomplished like so:

const https = require('https');
const http2 = require('http2');
const tls = require('tls');

const tlsConnection = tls.connect({
  host: 'nghttp2.org',
  port: 443,
  ALPNProtocols: ['h2', 'http/1.1']
});
tlsConnection.once('secureConnect', () => {
  if (tlsConnection.alpnProtocol === 'h2')
    withHttp2(tlsConnection);
  else if (tlsConnection.alpnProtocol === 'http/1.1')
    withHttp1(tlsConnection);
  else
    throw new Error('no valid protocol negotiated');
});

function withHttp1(socket) {
  const req = https.request({
    host: 'nghttp2.org',
    path: '/httpbin/ip',
    port: 443,
    createConnection: () => socket
  }, (res) => console.log(res.headers));
  req.end();
}

function withHttp2(socket) {
  const client = http2.connect(
    { host: 'nghttp2.org', port: 443 },
    { createConnection: () => socket }
  );
  const req = client.request({ ':path': '/httpbin/ip' });
  req.on('response', (headers) => console.log(headers));
  req.end();
}

That said, http2 still ideally needs an Agent implementation.

[grantila]

I love the fast work you guys do, @apapirovski, @addaleax and @jasnell. Haven't yet looked into using this (btw, we'll see this in 8.9, or?)

Just one thing; I don't think we need an Agent implementation for http2. You might want a "fake" agent-to-session wrapper to the compatibility layer, but those not using the compat should deal with sessions instead, right? After all, Agents are only there because http1 can't multiplex requests over the same socket.

[jasnell]

Going to close this issue as I believe this is handled. An agent/pool implementation should be coming by end of week (hopefully)

[paambaati]

@jasnell Just curious, did the agent/pool implementation land?