refactor(*): Transport redesign #4568

umgefahren · 2023-09-27T21:20:50Z

Description

Resolves: #4226.
Resolves: #3953.
Resolves: #3889.

Notes & open questions

What do you think of the implementation?

Change checklist

I have performed a self-review of my own code
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
A changelog entry has been made in the appropriate crates

core/src/transport.rs

thomaseizinger

Great start! I left some high-level comments :)

core/src/transport.rs

swarm/src/dial_opts.rs

transports/tcp/src/lib.rs

core/src/transport.rs

swarm/src/lib.rs

umgefahren · 2023-09-28T09:29:57Z

I have implemented some of the suggestions. However I didn't touch the question on whether DialOpts should be struct or the field should just be passed as arguments.

I also moved some methods into a macro.

umgefahren · 2023-09-28T09:40:57Z

Since we are touching PortUse anyway and we don't want to introduce breaking changes in the future: Is there a use for choosing not only if we should use an existing or new one, but also a specific one?

swarm/src/lib.rs

…n if we already have a listener for outgoing connections.

…rt-redesign

umgefahren · 2023-10-04T09:34:08Z

At the moment I'm pretty pleased with the implementation. But I don't really like having the list of things that we wan't to strip. It feels like something that might cause a bug later on, but I honestly don't have a better idea.

thomaseizinger · 2023-10-05T04:05:47Z

But I don't really like having the list of things that we wan't to strip. It feels like something that might cause a bug later on, but I honestly don't have a better idea.

As long as we cover it with unit tests, that is fine. We can always ship improvements as patch releases later if we find that we are missing something :)

swarm/src/lib.rs

thomaseizinger

I think this is a fine solution. It is known that one of the disadvantages of enabling port reuse is that you can only have a single connection to another peer. Thus, our best way of fulfilling the request to establish another connection is to ignore port reuse and simply allocate a new port. I think that is better than the connection failing.

I have two suggestions for slightly revising the log and bumping its severity.

transports/tcp/src/lib.rs

Co-authored-by: Thomas Eizinger <[email protected]>

umgefahren · 2024-08-03T10:03:10Z

I think this is a fine solution. It is known that one of the disadvantages of enabling port reuse is that you can only have a single connection to another peer. Thus, our best way of fulfilling the request to establish another connection is to ignore port reuse and simply allocate a new port. I think that is better than the connection failing.

I have two suggestions for slightly revising the log and bumping its severity.

I'm glad you are fine with the solution and I implemented your suggestions. I'm ready to merge.

thomaseizinger

LGTM

thomaseizinger · 2024-08-03T12:22:43Z

Great to see this being so close to merging. Thanks for pushing this over the finish line @umgefahren.

umgefahren · 2024-08-03T12:26:00Z

Is there something left for me to do for merging?

thomaseizinger · 2024-08-03T12:32:32Z

Is there something left for me to do for merging?

Not at this stage. I'll leave it to @jxs to coordinate when it is good to go in!

jxs

LGTM Thanks Hannes for your hard work, and Thomas for the review! ❤️
I will try to release 0.54 on Monday

umgefahren · 2024-08-03T14:22:54Z

Thanks to everyone involved, I'm sorry it took so much longer than anticipated.

thomaseizinger · 2024-08-03T22:16:43Z

I will try to release 0.54 on Monday

Do you not want to release AutoNatV2 with this together? There is another PR coming for that I'd assume.

umgefahren · 2024-08-04T15:03:12Z

I will try to release 0.54 on Monday

Do you not want to release AutoNatV2 with this together? There is another PR coming for that I'd assume.

In case you do: I have opened #5526.

Closes: #4524 This is the implementation of the evolved AutoNAT protocol, named AutonatV2 as defined in the [spec](https://github.com/libp2p/specs/blob/03718ef0f2dea4a756a85ba716ee33f97e4a6d6c/autonat/autonat-v2.md). The stabilization PR for the spec can be found under libp2p/specs#538. The work on the Rust implementation can be found in the PR to my fork: umgefahren#1. The implementation has been smoke-tested with the Go implementation (PR: libp2p/go-libp2p#2469). The new protocol addresses shortcomings of the original AutoNAT protocol: - Since the server now always dials back over a newly allocated port, this made #4568 necessary; the client can be sure of the reachability state for other peers, even if the connection to the server was made through a hole punch. - The server can now test addresses different from the observed address (i.e., the connection to the server was made through a `p2p-circuit`). To mitigate against DDoS attacks, the client has to send more data to the server than the dial-back costs. Pull-Request: #5526.

Resolves: libp2p#4226. Resolves: libp2p#3953. Resolves: libp2p#3889. Pull-Request: libp2p#4568.

Closes: libp2p#4524 This is the implementation of the evolved AutoNAT protocol, named AutonatV2 as defined in the [spec](https://github.com/libp2p/specs/blob/03718ef0f2dea4a756a85ba716ee33f97e4a6d6c/autonat/autonat-v2.md). The stabilization PR for the spec can be found under libp2p/specs#538. The work on the Rust implementation can be found in the PR to my fork: umgefahren#1. The implementation has been smoke-tested with the Go implementation (PR: libp2p/go-libp2p#2469). The new protocol addresses shortcomings of the original AutoNAT protocol: - Since the server now always dials back over a newly allocated port, this made libp2p#4568 necessary; the client can be sure of the reachability state for other peers, even if the connection to the server was made through a hole punch. - The server can now test addresses different from the observed address (i.e., the connection to the server was made through a `p2p-circuit`). To mitigate against DDoS attacks, the client has to send more data to the server than the dial-back costs. Pull-Request: libp2p#5526.

# Description Fixes #5996 https://github.com/libp2p/rust-libp2p/releases/tag/libp2p-v0.53.0 https://github.com/libp2p/rust-libp2p/blob/master/CHANGELOG.md ## Integration Nothing special is needed, just note that `yamux_window_size` is no longer applicable to libp2p (litep2p seems to still have it though). ## Review Notes There are a few simplifications and improvements done in libp2p 0.53 regarding swarm interface, I'll list a few key/applicable here. libp2p/rust-libp2p#4788 removed `write_length_prefixed` function, so I inlined its code instead. libp2p/rust-libp2p#4120 introduced new `libp2p::SwarmBuilder` instead of now deprecated `libp2p::swarm::SwarmBuilder`, the transition is straightforward and quite ergonomic (can be seen in tests). libp2p/rust-libp2p#4581 is the most annoying change I have seen that basically makes many enums `#[non_exhaustive]`. I mapped some, but those that couldn't be mapped I dealt with by printing log messages once they are hit (the best solution I could come up with, at least with stable Rust). libp2p/rust-libp2p#4306 makes connection close as soon as there are no handler using it, so I had to replace `KeepAlive::Until` with an explicit future that flips internal boolean after timeout, achieving the old behavior, though it should ideally be removed completely at some point. `yamux_window_size` is no longer used by libp2p thanks to libp2p/rust-libp2p#4970 and generally Yamux should have a higher performance now. I have resolved and cleaned up all deprecations related to libp2p except `BandwidthSinks`. Libp2p deprecated it (though it is still present in 0.54.1, which is why I didn't handle it just yet). Ideally Substrate would finally [switch to the official Prometheus client](paritytech/substrate#12699), in which case we'd get metrics for free. Otherwise a bit of code will need to be copy-pasted to maintain current behavior with `BandwidthSinks` gone, which I left a TODO about. The biggest change in 0.54.0 is libp2p/rust-libp2p#4568 that changed transport APIs and enabled unconditional potential port reuse, which can lead to very confusing errors if running two Substrate nodes on the same machine without changing listening port explicitly. Overall nothing scary here, but testing is always appreciated. # Checklist * [x] My PR includes a detailed description as outlined in the "Description" and its two subsections above. * [x] My PR follows the [labeling requirements]( https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process ) of this project (at minimum one label for `T` required) * External contributors: ask maintainers to put the right label on your PR. --- Polkadot Address: 1vSxzbyz2cJREAuVWjhXUT1ds8vBzoxn2w4asNpusQKwjJd --------- Co-authored-by: Dmitry Markin <[email protected]>

umgefahren added 3 commits September 27, 2023 14:56

Proposed API design

326a2b9

Implement proposed changes to the Transport trait.

25e6ccf

Merge branch 'master' into transport-redesign

85ea10a

umgefahren changed the title ~~Transport redesign~~ refactor(core): Transport redesign Sep 27, 2023

umgefahren mentioned this pull request Sep 27, 2023

core: introduce a unified Transport::dial interface #4226

Closed

dariusc93 reviewed Sep 27, 2023

View reviewed changes

core/src/transport.rs Show resolved Hide resolved

thomaseizinger reviewed Sep 27, 2023

View reviewed changes

umgefahren added 2 commits September 28, 2023 10:30

Merge branch 'master' into transport-redesign

b90eac7

Reworked proposed API changes

c9e33e1

Merge branch 'master' into transport-redesign

dc47b08

thomaseizinger reviewed Sep 28, 2023

View reviewed changes

swarm/src/lib.rs Outdated Show resolved Hide resolved

umgefahren added 13 commits September 29, 2023 16:30

Introduced listener presence structure. Allows for efficient detectio…

461494b

…n if we already have a listener for outgoing connections.

Merge remote-tracking branch 'origin/transport-redesign' into transpo…

309fad6

…rt-redesign

Continous updates and BTree

d6ea5bc

Merge branch 'master' into transport-redesign

9f6ffa5

Make for better testability

9cc3ff5

Nevermind let's use HashMap

5570b60

Resolved conflict

cfd8786

Little error

31ad325

Write more comments and switch to u16.

dbd3277

extremly reduced complexity

6c11a8b

extremly reduced complexity and removed the now unnecessary files

9aa1df2

Merge branch 'master' into transport-redesign

092c3ba

extremly reduced complexity and removed the now unnecessary files

0fbe7e7

umgefahren changed the title ~~refactor(core): Transport redesign~~ refactor(core, swarm): Transport redesign Oct 4, 2023

thomaseizinger reviewed Oct 5, 2023

View reviewed changes

swarm/src/lib.rs Outdated Show resolved Hide resolved

swarm/src/lib.rs Outdated Show resolved Hide resolved

umgefahren changed the title ~~refactor(*): Transport redesign, allowing for specifying port (re-)use policy when dealing~~ refactor(*): Transport redesign, allowing for specifying port (re-)use policy when dialing Aug 2, 2024

umgefahren changed the title ~~refactor(*): Transport redesign, allowing for specifying port (re-)use policy when dialing~~ refactor(*): Transport redesign Aug 2, 2024

umgefahren added 2 commits August 2, 2024 16:05

Merge branch 'master' into transport-redesign

362c3a2

Remove unnecessary sleep

aa60a7e

thomaseizinger approved these changes Aug 3, 2024

View reviewed changes

transports/tcp/src/lib.rs Outdated Show resolved Hide resolved

transports/tcp/src/lib.rs Outdated Show resolved Hide resolved

umgefahren and others added 2 commits August 3, 2024 11:52

Update transports/tcp/src/lib.rs

95338bd

Co-authored-by: Thomas Eizinger <[email protected]>

Implement thomas suggestions

0d45773

thomaseizinger approved these changes Aug 3, 2024

View reviewed changes

jxs approved these changes Aug 3, 2024

View reviewed changes

jxs added the send-it label Aug 3, 2024

mergify bot merged commit 079b2d6 into libp2p:master Aug 3, 2024
72 checks passed

umgefahren deleted the transport-redesign branch August 3, 2024 15:02

umgefahren mentioned this pull request Aug 4, 2024

feat(autonat): Implement AutoNATv2 #5526

Merged

5 tasks

b-zee mentioned this pull request Aug 5, 2024

AutoNAT on QUIC falsely reports public NAT status #3900

Closed

hrxi mentioned this pull request Aug 13, 2024

build(deps): bump the minor-and-patch-cargo-updates group across 1 directory with 11 updates nimiq/core-rs-albatross#2828

Closed

TimTinkers pushed a commit to unattended-backpack/rust-libp2p that referenced this pull request Sep 14, 2024

refactor(*): Transport redesign

c298325

Resolves: libp2p#4226. Resolves: libp2p#3953. Resolves: libp2p#3889. Pull-Request: libp2p#4568.

nazar-pc mentioned this pull request Oct 27, 2024

Upgrade libp2p from 0.52.4 to 0.54.1 paritytech/polkadot-sdk#6248

Open

2 tasks

shamil-gadelshin mentioned this pull request Dec 10, 2024

Research the hole-punching solution for DSN. autonomys/subspace#2111

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(*): Transport redesign #4568

refactor(*): Transport redesign #4568

umgefahren commented Sep 27, 2023 •

edited

Loading

thomaseizinger left a comment

umgefahren commented Sep 28, 2023 •

edited

Loading

umgefahren commented Sep 28, 2023

umgefahren commented Oct 4, 2023

thomaseizinger commented Oct 5, 2023

thomaseizinger left a comment

umgefahren commented Aug 3, 2024

thomaseizinger left a comment

thomaseizinger commented Aug 3, 2024

umgefahren commented Aug 3, 2024

thomaseizinger commented Aug 3, 2024

jxs left a comment

umgefahren commented Aug 3, 2024

thomaseizinger commented Aug 3, 2024 •

edited

Loading

umgefahren commented Aug 4, 2024

refactor(*): Transport redesign #4568

refactor(*): Transport redesign #4568

Conversation

umgefahren commented Sep 27, 2023 • edited Loading

Description

Notes & open questions

Change checklist

thomaseizinger left a comment

Choose a reason for hiding this comment

umgefahren commented Sep 28, 2023 • edited Loading

umgefahren commented Sep 28, 2023

umgefahren commented Oct 4, 2023

thomaseizinger commented Oct 5, 2023

thomaseizinger left a comment

Choose a reason for hiding this comment

umgefahren commented Aug 3, 2024

thomaseizinger left a comment

Choose a reason for hiding this comment

thomaseizinger commented Aug 3, 2024

umgefahren commented Aug 3, 2024

thomaseizinger commented Aug 3, 2024

jxs left a comment

Choose a reason for hiding this comment

umgefahren commented Aug 3, 2024

thomaseizinger commented Aug 3, 2024 • edited Loading

umgefahren commented Aug 4, 2024

umgefahren commented Sep 27, 2023 •

edited

Loading

umgefahren commented Sep 28, 2023 •

edited

Loading

thomaseizinger commented Aug 3, 2024 •

edited

Loading