Turbo randomly hangs for 60-90 seconds in between task commands during a Docker image build

[ondrej]

Verify canary release

• I verified that the issue exists in the latest Turborepo canary release.

Link to code that reproduces this issue

https://github.com/ondrej/letsgo/tree/turbo-bug

What package manager are you using / does the bug impact?

Yarn v1

What operating system are you using?

Mac

Which canary version will you have in your reproduction?

1.12.4-canary.1

Describe the Bug

When executing a task (e.g. turbo run build --filter=api...), Turborepo 1.11.0 and higher will frequently, randomly hang for 60-90 seconds before proceeding to the next task. Debugging confirms the previous task completed (e.g. tsc && echo 'Done') and control seems to return back to Turbo, which then waits before successfully moving on to the next task. Running Turbo with -vvv doesn't show Turbo generating any traces between when the previous command completes and when Turbo moves on to the next one. I was never able to reproduce the bug outside of a Docker image build.

Expected Behavior

Turbo will immediately move on to the next command, as it does in versions up to 1.10.16.

To Reproduce

1. git clone [email protected]:ondrej/letsgo.git
2. git switch turbo-bug
3. yarn install
4. yarn build
5. cd apps/api
6. yarn buildx

You may need to run the last command 2-3 times to see the bug reproduce. If you do, change a source file (e.g. edit the log in src/index.ts) to avoid a cached layer.

Additional context

I tested my reproduction with versions 1.10.16, 1.11.0, 1.12.4, and 1.12.4-canary.1. It works on version 1.10.16, fails on the others.

The repro repo uses yarn but the bug reproduces with npm as well.

The bug reproduces frequently, maybe 4/5 times. When it does reproduce, it can reproduce multiple times for multiple commands in a single task, extending total run time by 60-90 seconds for every time it hits.

I experimented with different --concurrency= values but it made no difference.

Docker Desktop for Mac is on version 4.27.1.

[arlyon]

Hello, I tried to repro this on linux + podman and with macos + docker desktop and can't seem to get the hang. I will mention that to make buildx play nicely with turbo you should probably ensure it is logging in 'plain' mode (rather than 'tty' mode). The latter will break formatting which (maybe..?) causes turbo to look like it's not doing anything.

To give more detailed advice given I can't reproduce I would request you produce a profile (using the --profile flag)

[unional]

I have CI job hang indefinitely until the CI timed out (6h)
https://github.com/repobuddy/repobuddy/actions/runs/8490047264/job/23260750272

Also observe the same issue on Macbook randomly, from multiple people.

related: #7057

[arlyon]

Hey @unional I let my machine run overnight on repobuddy with the command that failed on CI to try to collect a profile and can't repro it. If you are able to run turbo with the --profile flag for a while and upload the result I can try to investigate.

[unional]

Sure, it's pretty random. I experienced that locally mostly on a private repo related to work. Not running build or doing development of repobuddy on work laptop. :)

Somehow it is very consistent in the GitHub CI.