Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add metrics to monitor task validation. #5466

Merged
merged 3 commits into from
Dec 6, 2023
Merged

Add metrics to monitor task validation. #5466

merged 3 commits into from
Dec 6, 2023

Conversation

agautam478
Copy link
Contributor

What changed?

  • Introduced the metric monitoring to the taskvalidator service.
  • We are counting the number of workflows that enter the service for validation. This will give us a perspective of how many workflows are suspected to be corrupted when going through it's respective processing.
  • Currently we are only monitoring the workflows from history update where the maximum attempts exceeded to update history.
  • The plan is to roll it out for other task processing flows as well.

Why?
Made these changes to get a sense of how many workflows are assumed to be corrupted.

How did you test it?
tested locally.
Wrote a unit test.

Potential risks
NA

Release notes
NA

Documentation Changes
NA

@ketsiambaku ketsiambaku self-requested a review December 6, 2023 10:54
ketsiambaku
ketsiambaku approved these changes Dec 6, 2023
View reviewed changes
Copy link
Member

@ketsiambaku ketsiambaku left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

taylanisikdemir
taylanisikdemir reviewed Dec 6, 2023
View reviewed changes
common/taskvalidator/validateworkflow.go
tag.WorkflowRunID(runID),
tag.WorkflowDomainID(domainID))
// Emit the number of workflows that have come in for the validation.
w.metricsClient.Scope(metrics.TaskValidatorScope).IncCounter(metrics.ValidatedWorkflowCount)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which tags will this metric have? can you share example from local run?

@agautam478 agautam478 merged commit 186d253 into cadence-workflow:master Dec 6, 2023
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@taylanisikdemir taylanisikdemir taylanisikdemir left review comments

@ketsiambaku ketsiambaku ketsiambaku approved these changes

@Shaddoll Shaddoll Awaiting requested review from Shaddoll Shaddoll is a code owner

@neil-xie neil-xie Awaiting requested review from neil-xie neil-xie is a code owner

@davidporter-id-au davidporter-id-au Awaiting requested review from davidporter-id-au davidporter-id-au is a code owner

@Groxx Groxx Awaiting requested review from Groxx Groxx is a code owner

@shijiesheng shijiesheng Awaiting requested review from shijiesheng shijiesheng is a code owner

@jakobht jakobht Awaiting requested review from jakobht jakobht is a code owner

@3vilhamster 3vilhamster Awaiting requested review from 3vilhamster 3vilhamster is a code owner

@sankari165 sankari165 Awaiting requested review from sankari165 sankari165 is a code owner

@dkrotx dkrotx Awaiting requested review from dkrotx dkrotx is a code owner

@demirkayaender demirkayaender Awaiting requested review from demirkayaender demirkayaender is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@agautam478 @taylanisikdemir @ketsiambaku