Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql: TestTxnContentionEventsTable failed #118236

Closed
cockroach-teamcity opened this issue Jan 23, 2024 · 0 comments · Fixed by #118372
Closed

sql: TestTxnContentionEventsTable failed #118236

cockroach-teamcity opened this issue Jan 23, 2024 · 0 comments · Fixed by #118372
Assignees
@michae2
Labels
branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-queries SQL Queries Team
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jan 23, 2024 •
edited by cockroach-jira-scripts
Loading

sql.TestTxnContentionEventsTable failed with artifacts on release-23.2 @ f6d211271f2ba51b57d083ffc5c2cbadc38b5b23:

=== RUN   TestTxnContentionEventsTable
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/68358425a2d75ead146501979d1133be/logTestTxnContentionEventsTable4120879167
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:146: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    crdb_internal_test.go:1032: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/sql_test/pkg/sql/crdb_internal_test.go:1032
        	            				github.com/cockroachdb/cockroach/pkg/sql_test/pkg/sql/crdb_internal_test.go:1051
        	            				github.com/cockroachdb/cockroach/pkg/sql_test/pkg/sql/crdb_internal_test.go:918
        	Error:      	"491.644574ms" is not greater than or equal to "499ms"
        	Test:       	TestTxnContentionEventsTable
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/68358425a2d75ead146501979d1133be/logTestTxnContentionEventsTable4120879167
--- FAIL: TestTxnContentionEventsTable (2.51s)
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-35633
@cockroach-teamcity cockroach-teamcity added branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-queries SQL Queries Team labels Jan 23, 2024
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Jan 23, 2024
@github-project-automation github-project-automation bot moved this to Triage in SQL Queries Jan 23, 2024
@yuzefovich yuzefovich removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Jan 23, 2024
michae2 added a commit to michae2/cockroach that referenced this issue Jan 26, 2024
@michae2
sql: fix flake in TestTxnContentionEventsTable
434094c 
Instead of 499 ms, allow pg_sleep(.5) to sometimes take 490 ms.

Fixes: cockroachdb#118236

Release note: None
@michae2 michae2 mentioned this issue Jan 26, 2024
Merged
michae2 added a commit to michae2/cockroach that referenced this issue Feb 1, 2024
@michae2
sql: fix flake in TestTxnContentionEventsTable
373310d 
In causeContention we deliberately hold a transaction open using
pg_sleep to block an update statement. The timing we're trying to
achieve is:

1. transaction insert
2. update starts and blocks
3. transaction held open using pg_sleep

We were using a WaitGroup to order (2) after (1), but there was no
synchronization to ensure (3) came after (2).

This commit adds a retry loop that checks
`crdb_internal.cluster_queries` to ensure (3) comes after (2).

Fixes: cockroachdb#118236

Release note: None
@yuzefovich yuzefovich moved this from Triage to Active in SQL Queries Feb 5, 2024
craig bot pushed a commit that referenced this issue Feb 6, 2024
@shralex @wenyihu6
@michae2 @xinhaoz @rail @rickystewart
Merge #117117 #117693 #118372 #118760 #118767 #118789 #118792 #118797
804d37e 
117117: kv: log slow requests on replica level in addition to range level r=shralex a=shralex

Previously, slow requests were only logged at the range level, but the logs did not indicate which replica is slow. Moreover, the SlowRPC metric attempted to represent the number of requests currently being retried, however it was done on the range level and therefore missed a second level of replica-level retries being done underneath.

This PR adds logging on the replica level, removes a confusing log line, and changes the metric to count the number of slow requests in a simpler manner.

Epic: https://cockroachlabs.atlassian.net/browse/CRDB-33510
Fixes: #114431

117693: changefeedccl: add observability metrics into sarama code r=rharding6373 a=wenyihu6

Now that this patch (#117544) has been merged, sarama now acknowledges and
reacts to kafka server's throttling messages by slowing down. To provide better
observability into sarama code, this patch adds a metrics registry interceptor
and a new metrics `changefeed.kafka_throttling_hist_nanos` which tracks time (in
nanos) spent in sarama's throttling when cockroachdb exceed the kafka quota.

Fixes: #117618

Release note: changefeed.kafka_throttling_hist_nanos has now been added to
metrics to monitor sarama throttling behavior resulting from exceeding kafka
quota.

118372: sql: fix flake in TestTxnContentionEventsTable r=yuzefovich a=michae2

In causeContention we deliberately hold a transaction open using pg_sleep to block an update statement. The timing we're trying to achieve is:

1. transaction insert
2. update starts and blocks
3. transaction held open using pg_sleep

We were using a WaitGroup to order (2) after (1), but there was no synchronization to ensure (3) came after (2).

This commit adds a retry loop that checks `crdb_internal.cluster_queries` to ensure (3) comes after (2).

Fixes: #118236

Release note: None

118760: builtins: allow VIEWACTIVITY priv to use crdb_internal.request_statem… r=xinhaoz a=xinhaoz

…ent_bundle

Previously only those with the VIEWACTIVITY role could use the crdb_internal.request_statement_bundle builtin. We should allow the VIEWACTIVITY privilege as well since role options are now deprecated. This allow also allow stmt bundle requests to be made from db-console for users with this granted privilege.

Epic: none
Fixes: #118759

Release note (bug fix): Those with VIEWACTIVITY privilege can now request statement bundles using crdb_internal.requets_statement_bundle or via db-console's sql activity page.

118767: release: confirm yum install r=celiala a=rail

This adds `-y` flag to install `yum` without user prompt.

Epic: none
Release note: None

118789: jobs,application_api: replace calls to `skip.Stress` with `skip.Duress` r=celiala a=rickystewart

`skip.Duress()` seems like it should have been used in this case as it gives more time under both `race` and `deadlock`. This will give these tests some extra time if they run in a heavyweight configuration but not "under stress".

Epic: CRDB-8308
Release note: None

118792: kvfollowerreadsccl: skip test under `race` not `stressrace` r=celiala a=rickystewart

Epic: CRDB-8308
Release note: None

118797: bincheck: do not run geos tests on Windows r=celiala a=rail

In #106642 we stopped shipping libgeos on Windows, but didn't update the bincheck test to reflect the change.

Epic: none
Release note: None

Co-authored-by: shralex <[email protected]>
Co-authored-by: Wenyi Hu <[email protected]>
Co-authored-by: Michael Erickson <[email protected]>
Co-authored-by: Xin Hao Zhang <[email protected]>
Co-authored-by: Rail Aliiev <[email protected]>
Co-authored-by: Ricky Stewart <[email protected]>
@craig craig bot closed this as completed in 199a586 Feb 6, 2024
@github-project-automation github-project-automation bot moved this from Active to Done in SQL Queries Feb 6, 2024
blathers-crl bot pushed a commit that referenced this issue Feb 6, 2024
@michae2
sql: fix flake in TestTxnContentionEventsTable
a0cb448 
In causeContention we deliberately hold a transaction open using
pg_sleep to block an update statement. The timing we're trying to
achieve is:

1. transaction insert
2. update starts and blocks
3. transaction held open using pg_sleep

We were using a WaitGroup to order (2) after (1), but there was no
synchronization to ensure (3) came after (2).

This commit adds a retry loop that checks
`crdb_internal.cluster_queries` to ensure (3) comes after (2).

Fixes: #118236

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Feb 6, 2024
Merged
wenyihu6 pushed a commit to wenyihu6/cockroach that referenced this issue Feb 21, 2024
@michae2 @wenyihu6
sql: fix flake in TestTxnContentionEventsTable
ea9468a 
In causeContention we deliberately hold a transaction open using
pg_sleep to block an update statement. The timing we're trying to
achieve is:

1. transaction insert
2. update starts and blocks
3. transaction held open using pg_sleep

We were using a WaitGroup to order (2) after (1), but there was no
synchronization to ensure (3) came after (2).

This commit adds a retry loop that checks
`crdb_internal.cluster_queries` to ensure (3) comes after (2).

Fixes: cockroachdb#118236

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@michae2 michae2

Labels
branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-queries SQL Queries Team
Projects
SQL Queries
Archived in project
Milestone
23.2
Development

Successfully merging a pull request may close this issue.

sql: fix flake in TestTxnContentionEventsTable michae2/cockroach
3 participants
@yuzefovich @cockroach-teamcity @michae2