Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ccl/changefeedccl: TestAlterChangefeedAddTargetsDuringBackfill failed #118463

Closed
cockroach-teamcity opened this issue Jan 30, 2024 · 0 comments · Fixed by #118476
Closed

ccl/changefeedccl: TestAlterChangefeedAddTargetsDuringBackfill failed #118463

cockroach-teamcity opened this issue Jan 30, 2024 · 0 comments · Fixed by #118476
Labels
A-cdc Change Data Capture branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc
Milestone

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jan 30, 2024

ccl/changefeedccl.TestAlterChangefeedAddTargetsDuringBackfill failed on master @ 1d2534b831775dd84edcf7372d38e35ed4903278:

    helpers_test.go:867: making server as system tenant
    helpers_test.go:942: making webhook feed factory
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:682: waiting for checkpoint
    helpers_test.go:689: could not read checkpoint
    alter_changefeed_test.go:1392: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/alter_changefeed_test.go:1392
        	            				GOROOT/src/runtime/panic.go:523
        	            				GOROOT/src/testing/testing.go:999
        	            				GOROOT/src/testing/testing.go:1076
        	            				github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:689
        	            				github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/alter_changefeed_test.go:1399
        	            				github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1133
        	Error:      	Received unexpected error:
        	            	operation "webhook.Next-938878945888141313" timed out after 30s (given timeout 30s): context deadline exceeded
        	            	(1) operation "webhook.Next-938878945888141313" timed out after 30s (given timeout 30s)
        	            	Wraps: (2) context deadline exceeded
        	            	Error types: (1) *timeutil.TimeoutError (2) context.deadlineExceededError
        	Test:       	TestAlterChangefeedAddTargetsDuringBackfill/webhook
    --- FAIL: TestAlterChangefeedAddTargetsDuringBackfill/webhook (94.30s)

Parameters:

  • attempt=1
  • run=15
  • shard=8
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-35746

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc labels Jan 30, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Jan 30, 2024
@blathers-crl blathers-crl bot added the A-cdc Change Data Capture label Jan 30, 2024
craig bot pushed a commit that referenced this issue Jan 30, 2024
118312: roachtest: remove interesting whitespace characters in random SQL test logs r=mgartner a=mgartner

In #102038 we started writing statements to the query log in SQL
comments. The change removes newline characters in the statement to
ensure that the statement will be on one line all within the comment.
However, we still see test logs where the comments are broken onto
multiple lines, as in #118273, which makes the file invalid
syntactically. This makes reproducing the test failure more difficult
because the comments have to be manually fixed so that the log file can
be successfully parsed.

I was able to determine that the line breaks are coming from other
"interesting" whitespace characters. The `xxd` output from the log file
in #118273 shows the carriage return character, `0d` being used in the
middle of a column name:

    00000090: 2041 5320 2263 6f0d 6c33 3830 2246 524f   AS "co.l380"FRO

This commit strips all interesting whitespace characters from the
statement to prevent the comment from being broken on multiple lines.

Epic: None

Release note: None


118368: roachprod: fix default backup schedule creation on start r=herkolategan a=renatolabs

For a while now, `roachprod` has created a default backup schedule on cluster creation when `--schedule-backups` is passed. This is also the default in clusters created to run roachtests.

When we fixed an issue with starting external-process tenants in 3715eb5, however, we unadvertently changed the order of two operations performed by roachprod on cluster start: setting the default cluster settings, and creating the default backup schedule.

As a consequence, the command used to create the backup schedule fails because, at that point, we haven't configured a license key yet.

To make matters worse, there was a bug in the error handling of `createFixedBackupSchedule` that prevented errors from being reported to the user; these errors were being swallowed and went unnoticed for a few months.

In this commit, we fix the error checking in that function, and also officially remove code that partially supported creating backups or admin users in tenants. Currently, roachprod will run that part of the setup for the system tenant. In the future, we might revisit this and also create a backup schedule and admin users for application tenants.

Epic: none

Release note: None

118476: changefeedccl: use correct channel size in parallelio r=jayshrivastava a=jayshrivastava

Previously, the channels used for sending requests and receiving results were too small. This meant that a caller could block on sending a request even after acquiring quota. This change ensures that the size of the channels is large enough so that this blocking does not occur.

Closes: #118463
Closes: #118462
Closes: #118461
Closes: #118460
Closes: #118459
Closes: #118458
Epic: none

118482: kvserver: increase shutdown propagation time in range merge test r=andrewbaptist a=kvoli

`TestStoreRangeMergeDuringShutDown` could occasionally flake when the shutdown hadn't propagated before applying the lease.

Increase the post-shutdown sleep from 10ms to 20ms.

Fixes: #118348
Release note: None

118483: roachpb: increase test make priority trials r=andrewbaptist a=kvoli

`TestMakePriority` could (very) rarely flake due to slight differences in the sampled vs underlying distribution. Increase the trial runs by 33% from 750k to 1000k to reduce the likelihood of this occurring.

Fixes: #118399
Release note: None

118498: gcjob_test: deflake TestSchemaChangeGCJob r=rafiss a=rafiss

This updates a test assertion so that if the GC TTL already began, the test does not fail.

fixes #117485
fixes #118467
Release note: None

Co-authored-by: Marcus Gartner <[email protected]>
Co-authored-by: Renato Costa <[email protected]>
Co-authored-by: Jayant Shrivastava <[email protected]>
Co-authored-by: Austen McClernon <[email protected]>
Co-authored-by: Rafi Shamim <[email protected]>
@craig craig bot closed this as completed in #118476 Jan 30, 2024
@craig craig bot closed this as completed in b5799ba Jan 30, 2024
wenyihu6 pushed a commit to wenyihu6/cockroach that referenced this issue Feb 21, 2024
Previously, the channels used for sending requests and receiving results were
too small. This meant that a caller could block on sending a request even
after acquiring quota. This change ensures that the size of the channels
is large enough so that this blocking does not occur.

Closes: cockroachdb#118463
Closes: cockroachdb#118462
Closes: cockroachdb#118461
Closes: cockroachdb#118460
Closes: cockroachdb#118459
Closes: cockroachdb#118458
Epic: none
@rharding6373 rharding6373 added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Mar 6, 2024
wenyihu6 pushed a commit to wenyihu6/cockroach that referenced this issue Oct 22, 2024
Previously, the channels used for sending requests and receiving results were
too small. This meant that a caller could block on sending a request even
after acquiring quota. This change ensures that the size of the channels
is large enough so that this blocking does not occur.

Closes: cockroachdb#118463
Closes: cockroachdb#118462
Closes: cockroachdb#118461
Closes: cockroachdb#118460
Closes: cockroachdb#118459
Closes: cockroachdb#118458
Epic: none
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-cdc Change Data Capture branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants