kv/kvserver: TestStoreRangeMergeDuringShutdown failed #118348
Assignees
Labels
A-testing
Testing tools and infrastructure
branch-release-23.1
Used to mark GA and release blockers, technical advisories, and bugs for 23.1
C-bug
Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
C-test-failure
Broken test (automatically or manually discovered).
O-robot
Originated from a bot.
P-2
Issues/test failures with a fix SLA of 3 months
T-kv
KV Team
Milestone
Comments
cockroach-teamcity
added
branch-release-23.1
|
Passes 1k/1k runs with: Appears like an uncommon test flake, similar to #81884. |
|
Changing cockroach/pkg/kv/kvserver/client_merge_test.go Line 4137 in 7ce170b I'll open a PR to deflake the test by doubling the sleep. |
kvoli
added
C-bug
exalate-issue-sync
bot
added
P-2
craig bot
pushed a commit
that referenced
this issue
Jan 30, 2024
118312: roachtest: remove interesting whitespace characters in random SQL test logs r=mgartner a=mgartner In #102038 we started writing statements to the query log in SQL comments. The change removes newline characters in the statement to ensure that the statement will be on one line all within the comment. However, we still see test logs where the comments are broken onto multiple lines, as in #118273, which makes the file invalid syntactically. This makes reproducing the test failure more difficult because the comments have to be manually fixed so that the log file can be successfully parsed. I was able to determine that the line breaks are coming from other "interesting" whitespace characters. The `xxd` output from the log file in #118273 shows the carriage return character, `0d` being used in the middle of a column name: 00000090: 2041 5320 2263 6f0d 6c33 3830 2246 524f AS "co.l380"FRO This commit strips all interesting whitespace characters from the statement to prevent the comment from being broken on multiple lines. Epic: None Release note: None 118368: roachprod: fix default backup schedule creation on start r=herkolategan a=renatolabs For a while now, `roachprod` has created a default backup schedule on cluster creation when `--schedule-backups` is passed. This is also the default in clusters created to run roachtests. When we fixed an issue with starting external-process tenants in 3715eb5, however, we unadvertently changed the order of two operations performed by roachprod on cluster start: setting the default cluster settings, and creating the default backup schedule. As a consequence, the command used to create the backup schedule fails because, at that point, we haven't configured a license key yet. To make matters worse, there was a bug in the error handling of `createFixedBackupSchedule` that prevented errors from being reported to the user; these errors were being swallowed and went unnoticed for a few months. In this commit, we fix the error checking in that function, and also officially remove code that partially supported creating backups or admin users in tenants. Currently, roachprod will run that part of the setup for the system tenant. In the future, we might revisit this and also create a backup schedule and admin users for application tenants. Epic: none Release note: None 118476: changefeedccl: use correct channel size in parallelio r=jayshrivastava a=jayshrivastava Previously, the channels used for sending requests and receiving results were too small. This meant that a caller could block on sending a request even after acquiring quota. This change ensures that the size of the channels is large enough so that this blocking does not occur. Closes: #118463 Closes: #118462 Closes: #118461 Closes: #118460 Closes: #118459 Closes: #118458 Epic: none 118482: kvserver: increase shutdown propagation time in range merge test r=andrewbaptist a=kvoli `TestStoreRangeMergeDuringShutDown` could occasionally flake when the shutdown hadn't propagated before applying the lease. Increase the post-shutdown sleep from 10ms to 20ms. Fixes: #118348 Release note: None 118483: roachpb: increase test make priority trials r=andrewbaptist a=kvoli `TestMakePriority` could (very) rarely flake due to slight differences in the sampled vs underlying distribution. Increase the trial runs by 33% from 750k to 1000k to reduce the likelihood of this occurring. Fixes: #118399 Release note: None 118498: gcjob_test: deflake TestSchemaChangeGCJob r=rafiss a=rafiss This updates a test assertion so that if the GC TTL already began, the test does not fail. fixes #117485 fixes #118467 Release note: None Co-authored-by: Marcus Gartner <[email protected]> Co-authored-by: Renato Costa <[email protected]> Co-authored-by: Jayant Shrivastava <[email protected]> Co-authored-by: Austen McClernon <[email protected]> Co-authored-by: Rafi Shamim <[email protected]>
blathers-crl bot
pushed a commit
that referenced
this issue
Jan 30, 2024
`TestStoreRangeMergeDuringShutDown` could occasionally flake when the shutdown hadn't propagated before applying the lease. Increase the post-shutdown sleep from 10ms to 20ms. Fixes: #118348 Release note: None
blathers-crl bot
pushed a commit
that referenced
this issue
Jan 30, 2024
`TestStoreRangeMergeDuringShutDown` could occasionally flake when the shutdown hadn't propagated before applying the lease. Increase the post-shutdown sleep from 10ms to 20ms. Fixes: #118348 Release note: None
wenyihu6
pushed a commit
to wenyihu6/cockroach
that referenced
this issue
Feb 21, 2024
`TestStoreRangeMergeDuringShutDown` could occasionally flake when the shutdown hadn't propagated before applying the lease. Increase the post-shutdown sleep from 10ms to 20ms. Fixes: cockroachdb#118348 Release note: None
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
kv/kvserver.TestStoreRangeMergeDuringShutdown failed with artifacts on release-23.1 @ 19ba39bf5c28cd15b89796332c0bbf6d4a7b0f50:
Help
See also: How To Investigate a Go Test Failure (internal)
This test on roachdash | Improve this report!
Jira issue: CRDB-35688
The text was updated successfully, but these errors were encountered: