Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel compilation disabled to avoid triggering deadlock in concurrent invocations of CMake #2145

Merged
merged 1 commit into from
Jan 5, 2024

Conversation

edwardalee
Copy link
Collaborator

In CI, macOS tests have been nondeterministically failing because the federated tests lock up and get cancelled after a long time (originally 6 hours, recently modified to 2 hours). It turns out that these failures were due to the compiler locking up, not the generated federated programs. This PR disables parallel compilation of the federated programs, which apparently can cause such lock ups.

Copy link
Collaborator

@petervdonovan petervdonovan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, this is not what I was expecting and I have no idea how this could cause deadlocks. Congratulations on finally fixing this -- until I have the time to find out exactly how the deadlocks happened, I am totally happy to keep parallel compilation disabled.

@edwardalee edwardalee added this pull request to the merge queue Jan 5, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jan 5, 2024
@lhstrh
Copy link
Member

lhstrh commented Jan 5, 2024

Glad we finally found the source of the problem! Disabling for now is the right thing to do, and we can revisit and optimize later.

@lhstrh lhstrh added this pull request to the merge queue Jan 5, 2024
@lhstrh lhstrh changed the title Disable parallel compilation Parallel compilation disabled to avoid triggering deadlock in concurrent invocations of CMake Jan 5, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jan 5, 2024
@lhstrh lhstrh merged commit eafbd5d into master Jan 5, 2024
43 checks passed
@lhstrh lhstrh deleted the disable-parallel-compilation branch January 5, 2024 23:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants