Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wait for up to a second for existing database lock to clear #617

Closed
wants to merge 1 commit into from

Conversation

Mark-Simulacrum
Copy link
Member

This lets users queue new builds and otherwise schedule work while an ongoing
run is proceeding (with lots of record-progress hits). Arguably we want to
prioritize user-initiated commands to take effect and cancel ongoing
record-progress operations (and other automated work), but that will need more
care.

Should fix cases like rust-lang/rust#94775 (comment),
or at least make them much more rare (individual database operations should very rarely take >1s,
though our logs do suggest it is not entirely infrequent today. (That's something that's worth looking into,
just haven't had time yet).

[2022-03-10T16:08:38Z DEBUG crater::db] sql query "INSERT INTO results (experiment, crate, toolchain, result, log, encoding) VALUES (?1, ?2, ?3, ?4, ?5, ?6);" executed in 5.009915144s
[2022-03-10T16:08:43Z DEBUG crater::db] sql query "INSERT INTO results (experiment, crate, toolchain, result, log, encoding) VALUES (?1, ?2, ?3, ?4, ?5, ?6);" executed in 5.009725792s
[2022-03-10T16:08:46Z DEBUG crater::db] sql query "INSERT INTO results (experiment, crate, toolchain, result, log, encoding) VALUES (?1, ?2, ?3, ?4, ?5, ?6);" executed in 3.235525973s

This lets users queue new builds and otherwise schedule work while an ongoing
run is proceeding (with lots of record-progress hits). Arguably we want to
prioritize user-initiated commands to take effect and cancel ongoing
record-progress operations (and other automated work), but that will need more
care.
@Mark-Simulacrum
Copy link
Member Author

@bors r+

@bors
Copy link
Contributor

bors commented Mar 10, 2022

📌 Commit 3802622 has been approved by Mark-Simulacrum

bors added a commit that referenced this pull request Mar 10, 2022
Wait for up to a second for existing database lock to clear

This lets users queue new builds and otherwise schedule work while an ongoing
run is proceeding (with lots of record-progress hits). Arguably we want to
prioritize user-initiated commands to take effect and cancel ongoing
record-progress operations (and other automated work), but that will need more
care.

Should fix cases like rust-lang/rust#94775 (comment),
or at least make them much more rare (individual database operations should very rarely take >1s,
though our logs do suggest it is not entirely infrequent today. (That's something that's worth looking into,
just haven't had time yet).

```
[2022-03-10T16:08:38Z DEBUG crater::db] sql query "INSERT INTO results (experiment, crate, toolchain, result, log, encoding) VALUES (?1, ?2, ?3, ?4, ?5, ?6);" executed in 5.009915144s
[2022-03-10T16:08:43Z DEBUG crater::db] sql query "INSERT INTO results (experiment, crate, toolchain, result, log, encoding) VALUES (?1, ?2, ?3, ?4, ?5, ?6);" executed in 5.009725792s
[2022-03-10T16:08:46Z DEBUG crater::db] sql query "INSERT INTO results (experiment, crate, toolchain, result, log, encoding) VALUES (?1, ?2, ?3, ?4, ?5, ?6);" executed in 3.235525973s
```
@bors
Copy link
Contributor

bors commented Mar 10, 2022

⌛ Testing commit 3802622 with merge 1691543...

@bors
Copy link
Contributor

bors commented Mar 10, 2022

💔 Test failed - checks-actions

@Mark-Simulacrum
Copy link
Member Author

It looks like the default busy timeout is claimed to be 5 seconds, so this would not actually help. I'm trying to figure out why we're seeing the database is locked error in the first place then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants