Using the async-timer crate with Tokio triggers weird crashes #5975
Comments
rubenfiszel
added
A-tokio
|
Anything I can do to help root cause this? |
|
If you're able to get a backtrace, then that would definitely be helpful. A minimal reproducer would be even better. Either way, I will look over the relevant code at some point to see if I can figure out what could cause this. |
|
It would also be helpful to know whether and in which way you are using the |
|
@Darksonn I cannot reproduce this in a deterministic way but I can reproduce it under 20mins by doing sufficient benchmarks on windmill.dev: https://github.com/windmill-labs/windmill/tree/main/benchmarks It is sufficient to run in the backend folder: And then in another CLI: For a sufficiently long time. I will provide a backtrace ASAP. Windmill doesn't directly use block_in_place, but since we use many complex dependencies, maybe one of ours does. |
|
@Darksonn Here is a full stack trace: |
|
Okay, that backtrace definitely contains some clues. I'll have to look more closely later, but there's a good chance that it is enough to diagnose the issue. |
|
A bit off-topic but we (Windmill Labs, Inc.) rely on tokio for the stability of windmill (and we are very happy) and are very grateful for all the hard work done on tokio. What is the best way to support the project? Github sponsorhip? |
|
Yeah, we have GitHub sponsors set up for the Tokio org. |
|
Sponsored, any update on this ? :) |
|
I am very puzzled by this stacktrace. Frame 40 is here: Then in 39, we go inside the Then, frame 37 is I could understand if we were running the destructor of the old core when replacing it (it should be I'm not sure what to make of this. |
|
@Darksonn what is the best way to help on this ? I can add more stack trace to see if they're all similar or try to see where we use async-timer |
|
Hey, thanks for the report and the backtrace. very helpful. After looking at the stack trace, I am fairly confident that the issue is with the You can see this is the issue from the stack trace based on the fact that it goes from inside Tokio's worker and jumps to Hope this helps, I will close this issue, but if you feel that I am wrong, feel free to open a new one w/ more details & link this one. |
Darksonn
changed the title
already borrowed: BorrowMutError from multi_thread/worker in 1.32.0
mikonieminen
added a commit
to mikonieminen/apalis
that referenced
this issue
Apr 4, 2024
Use of `async-timer` can cause panic with `already borrowed: BorrowMutError`. This issue is explained in a ticket in `tokio` project, see: tokio-rs/tokio#5975 (comment) Only the `sleep` function in `apalis-core` uses `async-timer` and since `apalis` only supports `tokio` and `async-std` as runtimes, we can use `sleep` functions from these two runtimes directly and drop `async-timer` as a dependency.
mikonieminen
added a commit
to mikonieminen/apalis
that referenced
this issue
Apr 4, 2024
Use of `async-timer` can cause panic with `already borrowed: BorrowMutError`. This issue is explained in a ticket in `tokio` project, see: tokio-rs/tokio#5975 (comment) Only the `sleep` function in `apalis-core` uses `async-timer` and since `apalis` only supports `tokio` and `async-std` as runtimes, we can use `sleep` functions from these two runtimes directly and drop `async-timer` as a dependency.
mikonieminen
added a commit
to mikonieminen/apalis
that referenced
this issue
Apr 10, 2024
Use of `async-timer` can cause panic with `already borrowed: BorrowMutError`. This issue is explained in a ticket in `tokio` project, see: tokio-rs/tokio#5975 (comment) Only the `sleep` function in `apalis-core` uses `async-timer` and since `apalis` only supports `tokio` and `async-std` as runtimes, we can use `sleep` functions from these two runtimes directly and drop `async-timer` as a dependency.
This was referenced Apr 24, 2024
wsxiaoys
pushed a commit
to wsxiaoys/apalis
that referenced
this issue
May 3, 2024
Use of `async-timer` can cause panic with `already borrowed: BorrowMutError`. This issue is explained in a ticket in `tokio` project, see: tokio-rs/tokio#5975 (comment) Only the `sleep` function in `apalis-core` uses `async-timer` and since `apalis` only supports `tokio` and `async-std` as runtimes, we can use `sleep` functions from these two runtimes directly and drop `async-timer` as a dependency.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Version
Platform
Linux rubx1 6.4.11-arch2-1 #1 SMP PREEMPT_DYNAMIC Sat, 19 Aug 2023 15:38:34 +0000 x86_64 GNU/Linux
Description
I run a high-throughtput, background workers with each worker being on spawn_blocking. From time to time (every 30mins) I get the following error:
The text was updated successfully, but these errors were encountered: