Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stall under high load #61

Closed
Stebalien opened this issue Jul 26, 2021 · 4 comments
Closed

Stall under high load #61

Stebalien opened this issue Jul 26, 2021 · 4 comments
Assignees
@marten-seemann
Labels
status/blocked Unable to be worked further until needs are met

Comments

@Stebalien
Copy link
Member

It appears that transfer can stall under high load. Reproducer is https://github.com/raulk/whitenoise/tree/repro/lotus-6602.

This appears to have been introduced in the recent send window changes.
@aschmahmann
Copy link

Update: 08-06-21: The above does not reproduce on @marten-seemann's machine. This is stalled until we get more information or the ability to reproduce. @raulk any assistance would be appreciated.

@BigLep BigLep added the status/blocked Unable to be worked further until needs are met label Aug 13, 2021
@BigLep
Copy link

BigLep commented Sep 3, 2021

Context: this was biting Lotus. Lotus worked around this by downgrading the version of yamux used.

@Stebalien
Copy link
Member Author

Update: I have a theory that this is caused by synchronous writes. The changes in question make the stream window ramp up over multiple round-trips instead of starting out at a full 16MiB (which is what go-libp2p-yamux configured the window to be in the past). Now that it ramps up slowly, attempting to write a megabyte may now block waiting for the other side to read.

Stebalien added a commit to filecoin-project/lotus that referenced this issue Oct 18, 2021
@Stebalien
unpin the yamux dependency
3b2e8c4 
This has been pinned for a while under suspicion that it was causing
graphsync stalls. However, the libp2p team has been unable to reliably
reproduce the issue and believes that it's likely a graphsync issue.

Libp2p issue: libp2p/go-yamux#61
@Stebalien Stebalien mentioned this issue Oct 18, 2021
Merged
@BigLep
Copy link

BigLep commented Nov 12, 2021

Closing since haven't been able to reproduce at the yamux layer and Lotus is now unpinned per filecoin-project/lotus#7532

@BigLep BigLep closed this as completed Nov 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@marten-seemann marten-seemann

Labels
status/blocked Unable to be worked further until needs are met
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@BigLep @Stebalien @marten-seemann @aschmahmann