Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix reset reapply #3252

Merged
merged 6 commits into from
May 12, 2020
Merged

Fix reset reapply #3252

merged 6 commits into from
May 12, 2020

Conversation

yux0
Copy link
Contributor

@yux0 yux0 commented May 11, 2020

What changed?

  1. Fix check slice length bug
  2. Check max signal count on re-application

Why?
This is to fix the feedback loop on reset and re-apply

How did you test it?
Local test

Potential risks
This feedback loop issue is not fixed and causes backlog in replication stack

@yux0 yux0 requested a review from yycptt May 11, 2020 22:06
executionInfo := mutableState.GetExecutionInfo()
maxAllowedSignals := e.config.MaximumSignalsPerExecution(domainEntry.GetInfo().Name)
if maxAllowedSignals > 0 && (int(executionInfo.SignalCount)+len(toReapplyEvents)) >= maxAllowedSignals {
e.logger.Info("Execution limit reached for maximum signals during signal re-application",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a quite critical error, as it's non-retryable and the passive side doesn't even know the reapply has failed. Shall we use Error level log and probably emit some metrics?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

signal reapplication should ignore this limit.
or at least return error

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this check in this PR. If it returns an error, the caller side has. to handle this error or it will retry forever. I will include it in a different PR.

service/history/historyEngine_test.go Outdated Show resolved Hide resolved
@yux0 yux0 merged commit 5321680 into master May 12, 2020
@yux0 yux0 deleted the fix_reset_reapply branch May 12, 2020 18:31
yux0 added a commit that referenced this pull request May 12, 2020
yux0 added a commit to yux0/cadence that referenced this pull request May 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants