Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(quic): rewrite quic using quinn #3454

Merged
merged 83 commits into from
Jul 28, 2023
Merged
Show file tree
Hide file tree
Changes from 64 commits
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
8bd903d
Init (it compiles)
kpp Feb 10, 2023
8bce874
Implement fn dial
kpp Feb 11, 2023
2fa3408
Handle quic version
kpp Feb 11, 2023
22b7944
Implement tokio/async-std switch
kpp Feb 11, 2023
3c77392
Fix one error handling
kpp Feb 11, 2023
729ebf9
Revisit todo
kpp Feb 11, 2023
ff2dc4b
Fix one error handling
kpp Feb 11, 2023
a288c97
Reduce the number of warnings
kpp Feb 11, 2023
a5493a3
Remove a lot of commented code
kpp Feb 11, 2023
b781c87
Merge branch 'master' into quic_quinn
kpp Feb 11, 2023
ce2403d
Remove quinn-proto dep
kpp Feb 13, 2023
b1560dd
Remove redundant Arcs
kpp Feb 14, 2023
8171862
Add more comments
kpp Feb 14, 2023
dd394db
Merge branch 'master' into quic_quinn
kpp Feb 14, 2023
a1789af
More comments
kpp Feb 14, 2023
b46feb7
More comments
kpp Feb 14, 2023
b947b12
Remove unused commented code
kpp Feb 14, 2023
0bce473
Fix old tests to match new structs
kpp Feb 14, 2023
26ac662
Merge branch 'master' into quic_quinn
kpp Feb 14, 2023
7e788b6
Rename endpoint.rs -> config.rs
kpp Feb 14, 2023
953ce83
Fix doc links
kpp Feb 15, 2023
df0f789
cargo fmt
kpp Feb 15, 2023
e8cbd5a
Merge branch 'master' into quic_quinn
kpp Feb 15, 2023
ee7c2cd
Merge branch 'master' into quic_quinn
kpp Feb 16, 2023
ab1c047
Review fixes
kpp Mar 7, 2023
54606bf
Review fixes
kpp Mar 7, 2023
bb5b0f9
Merge branch 'master' into quic_quinn
kpp Mar 7, 2023
32f0c08
Merge branch 'master' of https://github.com/libp2p/rust-libp2p into q…
mxinden Mar 26, 2023
808a6b4
Make incoming and outgoing futures in connection an Option<_>
kpp Mar 31, 2023
ce30bcb
Update transports/quic/src/connection.rs
kpp Mar 31, 2023
6e9be7b
Merge branch 'master' into quic_quinn
kpp Mar 31, 2023
11bf960
Refactor poll_close for Substream
kpp Mar 31, 2023
45f7222
Merge branch 'master' into quic_quinn
kpp Apr 11, 2023
2f6e719
cargo fmt
kpp Apr 11, 2023
2a93d21
Remove quinn structs from public API
kpp Apr 19, 2023
2826b7c
Merge branch 'master' into quic_quinn
kpp Apr 19, 2023
ffc7767
Merge branch 'master' into quic_quinn
thomaseizinger May 2, 2023
c357f73
Remove quinn structs from public Runtime API
kpp May 3, 2023
2a1948c
Remove thiserror::from impl for error types
kpp May 3, 2023
9158f48
Merge branch 'master' into quic_quinn
kpp May 3, 2023
9cd937b
Merge branch 'master' of https://github.com/libp2p/rust-libp2p into q…
mxinden May 11, 2023
34d15cf
Merge branch 'master' into quic_quinn
kpp May 22, 2023
ae0b427
Fix merge
kpp May 22, 2023
e507e44
Implement hole punching with SO_REUSEPORT
kpp May 24, 2023
c48b8e6
fix clippy
kpp May 24, 2023
af684ce
Use try_clone UdpSocket instead of PORT_REUSE
kpp Jun 7, 2023
608e94d
Update transports/quic/src/config.rs
kpp Jun 7, 2023
78f3e3c
Explain why Listener::socket_addr cannot fail
kpp Jun 7, 2023
be97ffe
Simplify calls to .poll/.poll_unpin
kpp Jun 7, 2023
d4de2d5
Rename Substream -> Stream
kpp Jun 7, 2023
5ee2e1f
Merge branch 'master' into quic_quinn
kpp Jun 7, 2023
686ec3f
Fix merge
kpp Jun 7, 2023
0b49010
Remove duplicate code
kpp Jun 7, 2023
18e7661
Merge branch 'master' into quic_quinn
kpp Jun 13, 2023
dcd1eda
Fix clippy
kpp Jun 13, 2023
49a9e27
Fix fmt
kpp Jun 13, 2023
e90556d
Revert changes in eligible_listener to match master
kpp Jun 13, 2023
769c6c1
Remove param need_if_watcher from Listener::new
kpp Jun 13, 2023
d4b8a58
Remove unused variable need_if_watcher
kpp Jun 13, 2023
4a0d9bc
Fix poll_read/poll_close on a closed stream
kpp Jun 14, 2023
038d00c
Fix dcutr example to bind to 127.0.0.1 instead of 0.0.0.0
kpp Jun 14, 2023
08f58f9
Fix test read_after_peer_dropped_stream
kpp Jun 14, 2023
33869b3
Merge branch 'master' into quic_quinn
kpp Jun 14, 2023
fd0187b
cargo fmt
kpp Jun 14, 2023
459dedd
fix(quic): use Provider::send_to for UDP datagram
mxinden Jun 28, 2023
bae4983
Clone ErrorKind in Stream::poll_close
kpp Jun 28, 2023
dd619ca
cargo clippy
kpp Jun 28, 2023
f31da2e
Simplify and hide dependency types
mxinden Jul 2, 2023
29c575e
Merge pull request #28 from mxinden/quic-quinn-send-to
kpp Jul 4, 2023
f019c7a
Merge branch 'master' into quic_quinn
kpp Jul 4, 2023
bde3252
fix(quic/stream): return error on read
mxinden Jul 7, 2023
7505c65
refactor(quic/provider): remove Provider::spawn
mxinden Jul 7, 2023
d977cb5
fix(Cargo.lock): udpate
mxinden Jul 7, 2023
16aa52b
Merge branch 'master' into quic_quinn
kpp Jul 18, 2023
19ff4d9
Merge pull request #29 from mxinden/quic_quinn
kpp Jul 24, 2023
8b65105
fix(quic/connection): await connection.closed
mxinden Jul 24, 2023
d275e85
Merge pull request #30 from mxinden/quinn-closed
kpp Jul 26, 2023
0b08770
Revert back to 0.0.0.0 listen_on in dcutr example
mxinden Jul 27, 2023
b6bd51f
Bump version
mxinden Jul 27, 2023
0a842b8
Merge branch 'master' of https://github.com/libp2p/rust-libp2p into q…
mxinden Jul 27, 2023
fccd979
Try hiding Provider trait
mxinden Jul 27, 2023
cfd147e
Fix doc comment
mxinden Jul 27, 2023
a92954a
Revert hiding Provider trait
mxinden Jul 28, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 34 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion examples/dcutr/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ fn main() -> Result<(), Box<dyn Error>> {
.build();

swarm
.listen_on("/ip4/0.0.0.0/udp/0/quic-v1".parse().unwrap())
.listen_on("/ip4/127.0.0.1/udp/0/quic-v1".parse().unwrap())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You said this is only because of how we implemented eligible_listener right?

But if you only listen on localhost, then this example is sort of use-less. Hole-punching on localhost doesn't make much sense and if we don't listen on all interfaces, you can't actually use this example to test hole-punching.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But it does not work on 0.0.0.0 either because of how eligible_listener is implemented.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In such case the eligible_listener implementation needs to be changed, no?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Frankly speaking I don't know what purpose serves that listen_addr.ip().is_loopback() == socket_addr.ip().is_loopback(). It was introduced in 1a4b8ac#diff-b1dcda9367774dd8a742457ff79bc528d030a8cc38d6791efefc78b30f3f2e03R141

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My guess is that it forces that the local addr of an "outgoing" packet with destination localhost uses a localhost listening socket, if present.

I believe @elenaf9 originally implemented this, maybe she can shed some light on this :)

Copy link
Contributor Author

@kpp kpp Jul 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mxinden That's weird we can hole punch !is_loopback() remote addrs with 0.0.0.0 but we can't punch 127.0.0.1 addrs. Maybe we should remove this listen_addr.ip().is_loopback() == socket_addr.ip().is_loopback() check?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mxinden That's weird we can hole punch !is_loopback() remote addrs with 0.0.0.0 but we can't punch 127.0.0.1 addrs.

Why would that be weird? 127.0.0.1 is a localhost only address. If the node is listening on localhost only, how should a remote node establish a connection to it, let alone establish a connection to it through a NAT or firewall?

Maybe we should remove this listen_addr.ip().is_loopback() == socket_addr.ip().is_loopback() check?

I don't understand the issue at hand. Nor do I understand how removing this line would fix something. Can you expand @kpp? If we want to dial a node on loopback, i.e. on localhost (socket_addr.ip().is_loopback() is true), then we want to choose a listener that is listening on loopback, i.e. localhost (listen_addr.ip().is_loopback() is true).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue is simple, we have have the following setup:

cargo r -p relay-server-example -- --secret-key-seed 1 --port 15353

cargo r -p dcutr -- --mode listen --secret-key-seed 2 --relay-address /ip4/127.0.0.1/udp/15353/quic-v1/p2p/12D3KooWPjceQrSwdWXPyLLeABRXmuqt69Rg3sBYbU1Nft9HyQ6X

cargo r -p dcutr -- --mode dial --secret-key-seed 3 --relay-address /ip4/127.0.0.1/udp/15353/quic-v1/p2p/12D3KooWPjceQrSwdWXPyLLeABRXmuqt69Rg3sBYbU1Nft9HyQ6X --remote-peer-id 12D3KooWH3uVF6wv47WnArKHk5p6cvgCJEb74UTmxztmQDc298L3

And it does not work without my patch, it fails with "no listeners" error. First of all, this code does not work on master and I am pretty surprised. Do you thing the setup above is wrong? If yes, lets fix my setup. If it is a correct setup then how shall we make the code work?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for being persistent here @kpp. This indeed seems to be a bug.

In TCP we track the specific listen addresses, even if the user calls listen_on with an unspecified address (e.g. 0.0.0.0). If the user calls listen_on with a specific IP address i.e. not a wildcard like 0.0.0.0 but e.g. 127.0.0.1 we track (here register) the specific address right away.

self.port_reuse.register(local_addr.ip(), local_addr.port());

In case where the user calls listen_on with an unspecified address (e.g. 0.0.0.0) we depend on if-watch, more specifically poll_if_watch to provide us with the concrete addresses:

/// Poll for a next If Event.
fn poll_if_addr(&mut self, cx: &mut Context<'_>) -> Poll<<Self as Stream>::Item> {
let if_watcher = match self.if_watcher.as_mut() {
Some(if_watcher) => if_watcher,
None => return Poll::Pending,
};
let my_listen_addr_port = self.listen_addr.port();
while let Poll::Ready(Some(event)) = if_watcher.poll_next_unpin(cx) {
match event {
Ok(IfEvent::Up(inet)) => {
let ip = inet.addr();
if self.listen_addr.is_ipv4() == ip.is_ipv4() {
let ma = ip_to_multiaddr(ip, my_listen_addr_port);
log::debug!("New listen address: {}", ma);
self.port_reuse.register(ip, my_listen_addr_port);

Either way, we end up tracking specific listen addresses with port-reuse enabled and can thus choose the right listener in local_dial_addr:

/// Selects a listening socket address suitable for use
/// as the local socket address when dialing.
///
/// If multiple listening sockets are registered for port
/// reuse, one is chosen whose IP protocol version and
/// loopback status is the same as that of `remote_ip`.
///
/// Returns `None` if port reuse is disabled or no suitable
/// listening socket address is found.
fn local_dial_addr(&self, remote_ip: &IpAddr) -> Option<SocketAddr> {
if let PortReuse::Enabled { listen_addrs } = self {
for (ip, port) in listen_addrs
.read()
.expect("`local_dial_addr` never panic while holding the lock")
.iter()
{
if ip.is_ipv4() == remote_ip.is_ipv4()
&& ip.is_loopback() == remote_ip.is_loopback()
{
if remote_ip.is_ipv4() {
return Some(SocketAddr::new(IpAddr::V4(Ipv4Addr::UNSPECIFIED), *port));
} else {
return Some(SocketAddr::new(IpAddr::V6(Ipv6Addr::UNSPECIFIED), *port));
}
}
}
}
None
}

In libp2p-quic we only track the initial address that the user provided on the listen_on call.

/// Channel used to send commands to the [`Driver`].
#[derive(Debug, Clone)]
pub(crate) struct Channel {
/// Channel to the background of the endpoint.
to_endpoint: mpsc::Sender<ToEndpoint>,
/// Address that the socket is bound to.
/// Note: this may be a wildcard ip address.
socket_addr: SocketAddr,
}

This might be a wildcard address. Later on, when dialing a 127.0.0.1 address, the check listen_addr.ip().is_loopback() == socket_addr.ip().is_loopback() discards the listener with the unspecified listen address, thus a new dial-only socket is used. Using a new socket for dialing makes hole punching from the dcutr dialer to the dcutr listener impossible.

How to move forward?

I am not sure simply removing the listen_addr.ip().is_loopback() == socket_addr.ip().is_loopback() check is the way to go. This check is still helpful. E.g. say we have two listeners, one on localhost, one not on localhost. Ideally we would use the former when dialing.

We could mirror what libp2p-tcp does. That is, track the specified listen addresses reported by if-watch. Later, when dialing, choose the listener with the right address.

Ideally I would like to have the operating system choose. It has the most information about all available routes to each interface.

For now, I suggest we go with the libp2p-tcp approach.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking about it some more, this is not a bug introduced in this pull request, but instead an already existing bug in our libp2p-quic implementation. With that in mind, I don't think this pull request needs to block on a fix for it.

I documented it in #4259 instead. Let's revert the dcutr/main.rs change in this pull request and tackle the root issue in a follow-up pull request.

.unwrap();
swarm
.listen_on("/ip4/0.0.0.0/tcp/0".parse().unwrap())
Expand Down
6 changes: 3 additions & 3 deletions transports/quic/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,15 +19,15 @@ libp2p-tls = { workspace = true }
libp2p-identity = { workspace = true }
log = "0.4"
parking_lot = "0.12.0"
quinn-proto = { version = "0.10.1", default-features = false, features = ["tls-rustls"] }
quinn = { version = "0.10.1", default-features = false, features = ["tls-rustls", "futures-io"] }
rand = "0.8.5"
rustls = { version = "0.21.1", default-features = false }
thiserror = "1.0.40"
tokio = { version = "1.28.2", default-features = false, features = ["net", "rt", "time"], optional = true }

[features]
tokio = ["dep:tokio", "if-watch/tokio"]
async-std = ["dep:async-std", "if-watch/smol"]
tokio = ["dep:tokio", "if-watch/tokio", "quinn/runtime-tokio"]
async-std = ["dep:async-std", "if-watch/smol", "quinn/runtime-async-std"]

# Passing arguments to the docsrs builder in order to properly document cfg's.
# More information: https://docs.rs/about/builds#cross-compiling
Expand Down
142 changes: 142 additions & 0 deletions transports/quic/src/config.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
// Copyright 2017-2020 Parity Technologies (UK) Ltd.
//
// Permission is hereby granted, free of charge, to any person obtaining a
// copy of this software and associated documentation files (the "Software"),
// to deal in the Software without restriction, including without limitation
// the rights to use, copy, modify, merge, publish, distribute, sublicense,
// and/or sell copies of the Software, and to permit persons to whom the
// Software is furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
// OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
// DEALINGS IN THE SOFTWARE.

use quinn::VarInt;
use std::{sync::Arc, time::Duration};

/// Config for the transport.
#[derive(Clone)]
pub struct Config {
/// Timeout for the initial handshake when establishing a connection.
/// The actual timeout is the minimum of this and the [`Config::max_idle_timeout`].
pub handshake_timeout: Duration,
/// Maximum duration of inactivity in ms to accept before timing out the connection.
pub max_idle_timeout: u32,
/// Period of inactivity before sending a keep-alive packet.
/// Must be set lower than the idle_timeout of both
/// peers to be effective.
///
/// See [`quinn::TransportConfig::keep_alive_interval`] for more
/// info.
pub keep_alive_interval: Duration,
/// Maximum number of incoming bidirectional streams that may be open
/// concurrently by the remote peer.
pub max_concurrent_stream_limit: u32,

/// Max unacknowledged data in bytes that may be send on a single stream.
pub max_stream_data: u32,

/// Max unacknowledged data in bytes that may be send in total on all streams
/// of a connection.
pub max_connection_data: u32,

/// Support QUIC version draft-29 for dialing and listening.
///
/// Per default only QUIC Version 1 / [`libp2p_core::multiaddr::Protocol::QuicV1`]
/// is supported.
///
/// If support for draft-29 is enabled servers support draft-29 and version 1 on all
/// QUIC listening addresses.
/// As client the version is chosen based on the remote's address.
pub support_draft_29: bool,

/// TLS client config for the inner [`quinn::ClientConfig`].
client_tls_config: Arc<rustls::ClientConfig>,
/// TLS server config for the inner [`quinn::ServerConfig`].
server_tls_config: Arc<rustls::ServerConfig>,
}

impl Config {
/// Creates a new configuration object with default values.
pub fn new(keypair: &libp2p_identity::Keypair) -> Self {
let client_tls_config = Arc::new(libp2p_tls::make_client_config(keypair, None).unwrap());
let server_tls_config = Arc::new(libp2p_tls::make_server_config(keypair).unwrap());
Self {
client_tls_config,
server_tls_config,
support_draft_29: false,
handshake_timeout: Duration::from_secs(5),
max_idle_timeout: 30 * 1000,
max_concurrent_stream_limit: 256,
keep_alive_interval: Duration::from_secs(15),
max_connection_data: 15_000_000,

// Ensure that one stream is not consuming the whole connection.
max_stream_data: 10_000_000,
}
}
}

/// Represents the inner configuration for [`quinn`].
#[derive(Debug, Clone)]
pub(crate) struct QuinnConfig {
pub(crate) client_config: quinn::ClientConfig,
pub(crate) server_config: quinn::ServerConfig,
pub(crate) endpoint_config: quinn::EndpointConfig,
}

impl From<Config> for QuinnConfig {
fn from(config: Config) -> QuinnConfig {
let Config {
client_tls_config,
server_tls_config,
max_idle_timeout,
max_concurrent_stream_limit,
keep_alive_interval,
max_connection_data,
max_stream_data,
support_draft_29,
handshake_timeout: _,
} = config;
let mut transport = quinn::TransportConfig::default();
// Disable uni-directional streams.
transport.max_concurrent_uni_streams(0u32.into());
transport.max_concurrent_bidi_streams(max_concurrent_stream_limit.into());
// Disable datagrams.
transport.datagram_receive_buffer_size(None);
transport.keep_alive_interval(Some(keep_alive_interval));
transport.max_idle_timeout(Some(VarInt::from_u32(max_idle_timeout).into()));
transport.allow_spin(false);
transport.stream_receive_window(max_stream_data.into());
transport.receive_window(max_connection_data.into());
let transport = Arc::new(transport);

let mut server_config = quinn::ServerConfig::with_crypto(server_tls_config);
server_config.transport = Arc::clone(&transport);
// Disables connection migration.
// Long-term this should be enabled, however we then need to handle address change
// on connections in the `Connection`.
server_config.migration(false);

let mut client_config = quinn::ClientConfig::new(client_tls_config);
client_config.transport_config(transport);

let mut endpoint_config = quinn::EndpointConfig::default();
if !support_draft_29 {
endpoint_config.supported_versions(vec![1]);
}

QuinnConfig {
client_config,
server_config,
endpoint_config,
}
}
}
Loading