Example failing on newer firefox

[mikedilger]

"Connection failed. WebTransportError: WebTransport connection rejected"

Works in Firefox 132.0.2 (64-bit). Tested with a fresh clean profile.
Fails in Firefox Developer Edition 133.0b8 (64-bit). Tested with a fresh clean profile.

I see a related earlier issue that was resolved: #166

[BiagioFesta]

Thank you for the report.

I was indeed able to reproduce with Firefox Developer Edition 133.0b8 (64-bit).

From a first glance, it does not seems to be related to #166.
From what I see, inspecting the packets, the HTTP3 layer does not even start. Firefox closes the connection right after the QUIC/TLS handshake (maybe certificate validation fails, but without firefox sending the TLS alert?)

I am going to investigate more on this, but it does not seem to be related to webtransport protocol

[BiagioFesta]

@mikedilger have you tried by using proper certificates? That is, a webtransport server that uses a certificated signed by a CA trusted by the system root store

[mikedilger]

No, I just tried running the example and wanted to report the situation. I'm not developing using wtransport yet. And when I do I intend to use rustls RFC 7250 RawPublicKey "certificates" (and maybe not browsers at all)

[bugnano]

I confirm that on Firefox version 133.0.3 it does not work indeed.

I'm using certificates generated by mkcert, and that the CA store of Firefox recognizes the mkcert authority.

In the Rust debug log I can see:

2024-12-14T07:27:24.401607Z DEBUG wtransport::endpoint: New incoming QUIC connection
2024-12-14T07:27:24.401637Z DEBUG wtransport_native: Connection accepted (127.0.0.1:4433)
2024-12-14T07:27:24.401761Z  INFO Connection{id=0}: wtransport_native: Waiting for session request...
2024-12-14T07:27:24.402560Z DEBUG Connection{id=0}: quinn_proto::connection: failed to authenticate packet
2024-12-14T07:27:24.403578Z DEBUG Connection{id=0}:drive{id=0}: quinn_proto::connection: ECN not acknowledged by peer
2024-12-14T07:27:24.418908Z ERROR Connection{id=0}: wtransport_native: ConnectionClosed(ConnectionClose(ConnectionClose { error_code: APPLICATION_ERROR, frame_type: None, reason: b"" }))

if that could help.

[bugnano]

And I confirm that firefox-esr 128 works correctly

[BiagioFesta]

Firefox closes the connection but sometimes it returns a crypto error instead of application error

CONNECTION_CLOSE (Transport) Error code: CRYPTO_ERROR (Unsupported Certificate)
    Frame Type: CONNECTION_CLOSE (Transport) (0x000000000000001c)
    Error code: CRYPTO_ERROR (299)
    TLS Alert Description: Unsupported Certificate (43)
    Frame Type: 0
    Reason phrase Length: 0
    Reason phrase: 

[robertpogue]

Amazing work on this project - It's been tremendously valuable to me. I've reproduced the issue on Firefox with a fresh copy of the repository using cargo run --example full. Unfortunately it's also an issue in my own project which uses wtransport (and certgen for local TLS certificates). I believe it started occurring in firefox after an update. Here are some additional details if it's helpful.

• Tested on commit 26872fd
• Firefox version: 133.0 (64-bit)
• Operation System: Arch linux

Steps to reproduce

• Clone new repository
• cargo run --example full

Results (with RUST_LOG=trace):

2024-12-15T20:14:29.276902Z  INFO full::webtransport: Server running on port 49653
2024-12-15T20:14:29.276911Z  INFO full::http: Server running on port 8080
2024-12-15T20:14:33.596086Z DEBUG wtransport::endpoint: New incoming QUIC connection
2024-12-15T20:14:33.596221Z  INFO Connection{id=0}: full::webtransport: Waiting for session request...
2024-12-15T20:14:33.596306Z DEBUG wtransport::endpoint: New incoming QUIC connection
2024-12-15T20:14:33.596371Z  INFO Connection{id=1}: full::webtransport: Waiting for session request...
2024-12-15T20:14:33.596644Z TRACE Connection{id=0}:first recv:frame{ty=CRYPTO}: quinn_proto::connection: consumed 615 CRYPTO bytes
2024-12-15T20:14:33.597266Z TRACE Connection{id=0}:first recv: quinn_proto::connection: Handshake keys ready
2024-12-15T20:14:33.597285Z TRACE Connection{id=0}:first recv: quinn_proto::connection: wrote 90 Initial CRYPTO bytes
2024-12-15T20:14:33.597328Z TRACE Connection{id=0}:first recv: quinn_proto::connection: Data keys ready
2024-12-15T20:14:33.597359Z TRACE Connection{id=0}:first recv: quinn_proto::connection: wrote 636 Handshake CRYPTO bytes
2024-12-15T20:14:33.597392Z TRACE Connection{id=0}:first recv: quinn_proto::connection: negotiated max idle timeout Some(30s)
2024-12-15T20:14:33.597412Z TRACE Connection{id=0}:first recv: quinn_proto::connection: got Data packet (595 bytes) from [::ffff:127.0.0.1]:34117 using id 0000000000000000
2024-12-15T20:14:33.597439Z TRACE Connection{id=0}:first recv: quinn_proto::connection::packet_crypto: decryption failed with packet number 5011686
2024-12-15T20:14:33.597451Z DEBUG Connection{id=0}:first recv: quinn_proto::connection: failed to authenticate packet
2024-12-15T20:14:33.597465Z TRACE Connection{id=0}: quinn_proto::endpoint: new connection id=0 icid=00d6694e19deb7cc
2024-12-15T20:14:33.597541Z TRACE Connection{id=0}:drive{id=0}:send{space=Initial pn=0}: quinn_proto::connection: ACK ArrayRangeSet([0..1]), Delay = 1212us
2024-12-15T20:14:33.597562Z TRACE Connection{id=0}:drive{id=0}:send{space=Initial pn=0}: quinn_proto::connection: CRYPTO: off 0 len 90
2024-12-15T20:14:33.597614Z TRACE Connection{id=0}:drive{id=0}:send{space=Handshake pn=0}: quinn_proto::connection: CRYPTO: off 0 len 636
2024-12-15T20:14:33.597634Z TRACE Connection{id=0}:drive{id=0}:send{space=Handshake pn=0}: quinn_proto::connection::packet_builder: PADDING * 385
2024-12-15T20:14:33.597667Z TRACE Connection{id=0}:drive{id=0}: quinn_proto::connection: sending 1200 bytes in 1 datagrams
2024-12-15T20:14:33.597717Z TRACE Connection{id=1}:first recv:frame{ty=CRYPTO}: quinn_proto::connection: consumed 615 CRYPTO bytes
2024-12-15T20:14:33.598169Z TRACE Connection{id=1}:first recv: quinn_proto::connection: Handshake keys ready
2024-12-15T20:14:33.598183Z TRACE Connection{id=1}:first recv: quinn_proto::connection: wrote 90 Initial CRYPTO bytes
2024-12-15T20:14:33.598210Z TRACE Connection{id=1}:first recv: quinn_proto::connection: Data keys ready
2024-12-15T20:14:33.598230Z TRACE Connection{id=1}:first recv: quinn_proto::connection: wrote 636 Handshake CRYPTO bytes
2024-12-15T20:14:33.598252Z TRACE Connection{id=1}:first recv: quinn_proto::connection: negotiated max idle timeout Some(30s)
2024-12-15T20:14:33.598263Z TRACE Connection{id=1}:first recv: quinn_proto::connection: got Data packet (595 bytes) from [::ffff:127.0.0.1]:37198 using id 0000000000000000
2024-12-15T20:14:33.598279Z TRACE Connection{id=1}:first recv: quinn_proto::connection::packet_crypto: decryption failed with packet number 209
2024-12-15T20:14:33.598287Z DEBUG Connection{id=1}:first recv: quinn_proto::connection: failed to authenticate packet
2024-12-15T20:14:33.598296Z TRACE Connection{id=1}: quinn_proto::endpoint: new connection id=1 icid=3da0ab65a789d21d
2024-12-15T20:14:33.598342Z TRACE Connection{id=1}:drive{id=1}:send{space=Initial pn=0}: quinn_proto::connection: ACK ArrayRangeSet([0..1]), Delay = 812us
2024-12-15T20:14:33.598354Z TRACE Connection{id=1}:drive{id=1}:send{space=Initial pn=0}: quinn_proto::connection: CRYPTO: off 0 len 90
2024-12-15T20:14:33.598380Z TRACE Connection{id=1}:drive{id=1}:send{space=Handshake pn=0}: quinn_proto::connection: CRYPTO: off 0 len 636
2024-12-15T20:14:33.598390Z TRACE Connection{id=1}:drive{id=1}:send{space=Handshake pn=0}: quinn_proto::connection::packet_builder: PADDING * 385
2024-12-15T20:14:33.598390Z TRACE Connection{id=0}:drive{id=0}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=1 id=d23591d4e23c068d
2024-12-15T20:14:33.598408Z TRACE Connection{id=1}:drive{id=1}: quinn_proto::connection: sending 1200 bytes in 1 datagrams
2024-12-15T20:14:33.598414Z TRACE Connection{id=0}:drive{id=0}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=2 id=f275ba5c0a349902
2024-12-15T20:14:33.598430Z TRACE Connection{id=0}:drive{id=0}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=3 id=b65101bdf86173d5
2024-12-15T20:14:33.598445Z TRACE Connection{id=0}:drive{id=0}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=4 id=87e9265d77a64f52
2024-12-15T20:14:33.598457Z TRACE Connection{id=0}:drive{id=0}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=5 id=ef11e62223832a01
2024-12-15T20:14:33.598469Z TRACE Connection{id=0}:drive{id=0}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=6 id=0601602efe2ef9d2
2024-12-15T20:14:33.598483Z TRACE Connection{id=0}:drive{id=0}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=7 id=a2688350018fbc98
2024-12-15T20:14:33.598513Z TRACE Connection{id=0}:drive{id=0}: quinn_proto::connection: sending 217 bytes in 1 datagrams
2024-12-15T20:14:33.598533Z TRACE Connection{id=1}:drive{id=1}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=1 id=71b11657a8c6126a
2024-12-15T20:14:33.598554Z TRACE Connection{id=1}:drive{id=1}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=2 id=67bda07c25d40434
2024-12-15T20:14:33.598565Z TRACE Connection{id=1}:drive{id=1}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=3 id=efb2ff74ccc13c5e
2024-12-15T20:14:33.598561Z TRACE Connection{id=0}:drive{id=0}: quinn_proto::connection: got Handshake packet (47 bytes) from [::ffff:127.0.0.1]:34117 using id 923e7138de5f50e3
2024-12-15T20:14:33.598574Z TRACE Connection{id=1}:drive{id=1}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=4 id=5587c9791248eac0
2024-12-15T20:14:33.598584Z TRACE Connection{id=1}:drive{id=1}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=5 id=632a7f6cb2e004bd
2024-12-15T20:14:33.598591Z TRACE Connection{id=0}:drive{id=0}:recv{space=Handshake pn=0}: quinn_proto::connection: discarding Initial keys
2024-12-15T20:14:33.598594Z TRACE Connection{id=1}:drive{id=1}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=6 id=207a05e0a3f2ef25
2024-12-15T20:14:33.598604Z TRACE Connection{id=1}:drive{id=1}:send{space=Data pn=0}: quinn_proto::connection: NEW_CONNECTION_ID sequence=7 id=ed6c3c2d9b4e6812
2024-12-15T20:14:33.598628Z TRACE Connection{id=1}:drive{id=1}: quinn_proto::connection: sending 217 bytes in 1 datagrams
2024-12-15T20:14:33.598628Z DEBUG Connection{id=0}:drive{id=0}:recv{space=Handshake pn=0}:frame{ty=ACK}: quinn_proto::connection: ECN not acknowledged by peer
2024-12-15T20:14:33.598655Z TRACE Connection{id=0}:drive{id=0}: quinn_proto::connection: connection closed
2024-12-15T20:14:33.598703Z  INFO Connection{id=0}: full::webtransport: Result: Err(connection aborted by peer: the application or application protocol caused the connection to be closed during the handshake)
2024-12-15T20:14:33.598979Z TRACE Connection{id=1}:drive{id=1}: quinn_proto::connection: got Handshake packet (46 bytes) from [::ffff:127.0.0.1]:37198 using id 14ffb35e7a8f38ee
2024-12-15T20:14:33.598988Z TRACE Connection{id=0}:drive{id=0}: quinn_proto::connection: got Handshake packet (47 bytes) from [::ffff:127.0.0.1]:34117 using id 923e7138de5f50e3
2024-12-15T20:14:33.599002Z TRACE Connection{id=1}:drive{id=1}:recv{space=Handshake pn=0}: quinn_proto::connection: discarding Initial keys
2024-12-15T20:14:33.599032Z DEBUG Connection{id=1}:drive{id=1}:recv{space=Handshake pn=0}:frame{ty=ACK}: quinn_proto::connection: ECN not acknowledged by peer
2024-12-15T20:14:33.599054Z TRACE Connection{id=1}:drive{id=1}: quinn_proto::connection: connection closed
2024-12-15T20:14:33.599079Z  INFO Connection{id=1}: full::webtransport: Result: Err(connection aborted by peer: the application or application protocol caused the connection to be closed during the handshake)
2024-12-15T20:14:33.666179Z TRACE Connection{id=1}:drive{id=1}: quinn_proto::connection: timeout timer=Close
2024-12-15T20:14:33.667397Z TRACE Connection{id=0}:drive{id=0}: quinn_proto::connection: timeout timer=Close

I'm willing to help investigate - any suggestions where to start? Test with older versions of Firefox to determine when it broke? Test with real TLS certificates instead of self-signed?

[BiagioFesta]

Thank you for the support and the help.

When reproducing, Firefox seemed to return Unsupported Certificate (43) sometimes (instead of application error). At this point of the investigation, the error happens during TLS handshake. It should not be related to webtransport protocol, rather to TLS/QUIC layer.

I've tried both with self-signed certificate and mkcert.

We should understand what changed in new Firefox release for certificate validation (maybe a look at firefox source code changes might worth it)

[robertpogue]

The last working firefox release was 132.0.2 and the first broken release was 133.0.
I built firefox from source and bisected further. The first broken commit is 845419:299911741634. It references bug Bug 1929368 "Disable HTTP/3 only when a third party root certificate is used in connection" which sounds relevant.

[robertpogue]

It looks like both chrome and firefox recently chose to require that "all certificates used for WebTransport over HTTP/3 are issued by a known certificate root". My application stopped working in both browsers when tested locally using self-signed certificates by mkcert.
Here is a workaround:

• Firefox: enter about:config in the firefox address bar, accept the scary warnings and change network.http.http3.disable_when_third_party_roots_found to false
• Chrome: enter chrome://flags and set #webtransport-developer-mode to Enabled

wtransport's example provides a 'serverCertificateHashes` to the wtransport config, which apparently chromium considers acceptable and firefox does not - I think this is a quirk of the browser implementation. Perhaps it could even be considered a bug in Firefox if their intention was to disable HTTP/3 when third party certificate roots are used.

Regardless, it is a browser change that makes wtransport a little more difficult to test locally, and requires browser configuration. Any thought on what we should do about it? Documentation at a minimum. I can prepare a pull request.

[UkoeHB]

The diff proposed to fix this Mozilla bug appears to re-enable serverCertificateHashes. Relevant discussion: w3c/webtransport#623