fix: Skip bad events in CDC. #8076
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
danielmai
commented
Oct 14, 2021
There was a problem hiding this comment.
The error message can now look similar to the one below instead of panicking:
E1013 16:34:48.129933 19 cdc_ee.go:208] error while marshalling batch for event [&{Operation:set Uid:2097730 Attr:myEdge Value:53565-01-01 00:00:00 +0000 UTC ValueType:datetime}]: json: error calling MarshalJSON for type time.Time: Time.MarshalJSON: year outside of range [0,9999]
E1013 16:34:48.130017 19 cdc_ee.go:208] error while marshalling batch for event [&{Operation:set Uid:2097730 Attr:myEdge Value:53584-01-01 00:00:00 +0000 UTC ValueType:datetime}]: json: error calling MarshalJSON for type time.Time: Time.MarshalJSON: year outside of range [0,9999]
E1013 16:34:48.131383 19 cdc_ee.go:220] error while sending cdc event to sink kafka: Failed to deliver 2 messages.
E1013 16:34:48.131426 19 cdc_ee.go:355] unable to send events kafka: Failed to deliver 2 messages.
unable to send messages to sink
github.com/dgraph-io/dgraph/worker.(*CDC).processCDCEvents.func2
/home/dmai/go/src/github.com/dgraph-io/dgraph/worker/cdc_ee.go:299
github.com/dgraph-io/dgraph/worker.(*CDC).processCDCEvents.func3
/home/dmai/go/src/github.com/dgraph-io/dgraph/worker/cdc_ee.go:334
github.com/dgraph-io/dgraph/worker.(*CDC).processCDCEvents
/home/dmai/go/src/github.com/dgraph-io/dgraph/worker/cdc_ee.go:354
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1581
CDC: unable to process raft entry
github.com/dgraph-io/dgraph/worker.(*CDC).processCDCEvents.func3
/home/dmai/go/src/github.com/dgraph-io/dgraph/worker/cdc_ee.go:335
github.com/dgraph-io/dgraph/worker.(*CDC).processCDCEvents
/home/dmai/go/src/github.com/dgraph-io/dgraph/worker/cdc_ee.go:354
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1581
Reviewable status: 0 of 1 files reviewed, all discussions resolved (waiting on @manishrjain)
danielmai
changed the title
|
|
mangalaman93
force-pushed
the
danielmai/cdc-error-log
branch
from
7861a82 to
075d6c8
Compare
mangalaman93
requested review from
akon-dey,
MichelDiz,
matthewmcneely,
darkn3rd,
meghalims and
billprovince
as code owners
This can have an error like the following:
2021/10/12 21:58:11 json: error calling MarshalJSON for type
time.Time: Time.MarshalJSON: year outside of range [0,9999]
github.com/dgraph-io/dgraph/x.Check
/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:42
github.com/dgraph-io/dgraph/worker.(*CDC).processCDCEvents.func1
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/cdc_ee.go:207
github.com/dgraph-io/dgraph/worker.(*CDC).processCDCEvents.func2
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/cdc_ee.go:295
github.com/dgraph-io/dgraph/worker.(*CDC).processCDCEvents.func3
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/cdc_ee.go:331
github.com/dgraph-io/dgraph/worker.(*CDC).processCDCEvents
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/cdc_ee.go:351
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371
mangalaman93
requested review from
mangalaman93,
sanjayk-github-dev,
harshil-goel,
all-seeing-code and
joshua-goldstein
as code owners
mangalaman93
force-pushed
the
danielmai/cdc-error-log
branch
from
075d6c8 to
991cc9b
Compare
nswamy
approved these changes
Aug 11, 2023
jbhamra1
pushed a commit
that referenced
this pull request
Aug 17, 2023
A CDC event can return an error during JSON marshalling like the
following. This change skips these events from being sent to CDC.
2021/10/12 21:58:11 json: error calling MarshalJSON for type time.Time:
Time.MarshalJSON: year outside of range [0,9999]
github.com/dgraph-io/dgraph/x.Check
/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:42
github.com/dgraph-io/dgraph/worker.(*CDC).processCDCEvents.func1
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/cdc_ee.go:207
github.com/dgraph-io/dgraph/worker.(*CDC).processCDCEvents.func2
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/cdc_ee.go:295
github.com/dgraph-io/dgraph/worker.(*CDC).processCDCEvents.func3
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/cdc_ee.go:331
github.com/dgraph-io/dgraph/worker.(*CDC).processCDCEvents
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/cdc_ee.go:351
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371
Co-authored-by: Aman Mangal <[email protected]>
jbhamra1
pushed a commit
that referenced
this pull request
Aug 17, 2023
A CDC event can return an error during JSON marshalling like the
following. This change skips these events from being sent to CDC.
2021/10/12 21:58:11 json: error calling MarshalJSON for type time.Time:
Time.MarshalJSON: year outside of range [0,9999]
github.com/dgraph-io/dgraph/x.Check
/ext-go/1/src/github.com/dgraph-io/dgraph/x/error.go:42
github.com/dgraph-io/dgraph/worker.(*CDC).processCDCEvents.func1
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/cdc_ee.go:207
github.com/dgraph-io/dgraph/worker.(*CDC).processCDCEvents.func2
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/cdc_ee.go:295
github.com/dgraph-io/dgraph/worker.(*CDC).processCDCEvents.func3
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/cdc_ee.go:331
github.com/dgraph-io/dgraph/worker.(*CDC).processCDCEvents
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/cdc_ee.go:351
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371
Co-authored-by: Aman Mangal <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A CDC event can return an error during JSON marshalling like the following:
This change skips these events from being sent to CDC.