Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BugFix] follower sync dictionary refresh state #47802

Merged
merged 3 commits into from
Jul 12, 2024

Conversation

srlch
Copy link
Contributor

@srlch srlch commented Jul 3, 2024

Why I'm doing:

In the current implementations, follower will not be synced any information about the dictionary refreshing which is a problem.

What I'm doing:

Reuse dictionaryMgrInfo and DICTIONARY_MODIFY log to save the dictionary state information.

Fixes #issue
https://github.com/StarRocks/StarRocksTest/issues/7961

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 3.3
    • 3.2
    • 3.1
    • 3.0
    • 2.5

@srlch srlch requested a review from a team as a code owner July 3, 2024 09:22
@wanpengfei-git wanpengfei-git requested a review from a team July 3, 2024 09:22
@mergify mergify bot assigned srlch Jul 3, 2024
@@ -1890,5 +1901,7 @@ service FrontendService {

TListSessionsResponse listSessions(1: TListSessionsRequest request)
TGetTemporaryTablesInfoResponse getTemporaryTablesInfo(1: TGetTemporaryTablesInfoRequest request)

TUpdateFollowerDictionaryStateResult updateFollowerDictionaryState(1: TUpdateFollowerDictionaryStateParams request)
}

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The most risky bug in this code is:
Missing semicolon after last_success_finished_time.

You can modify the code like this:

struct TUpdateFollowerDictionaryStateParams {
    1: optional i64 dictionary_id;
    2: optional i64 success_txn_id;
    3: optional bool error;
    4: optional string error_msg;
    5: optional i64 last_success_finished_time;
}

} catch (Exception e) {
LOG.warn("sync dictionary refresh state failed, fe: {}", fe.getHost(), e);
}
}
}

@Override
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The most risky bug in this code is:
Failure to synchronize dictionary states can lead to inconsistencies among followers.

You can modify the code like this:

public TUpdateFollowerDictionaryStateResult updateFollowerDictionaryState(TUpdateFollowerDictionaryStateParams request) {
    TUpdateFollowerDictionaryStateResult result = new TUpdateFollowerDictionaryStateResult();
    long leaderDictionaryId = request.getDictionary_id();  // Use getter methods for encapsulation
    long leaderSuccessTxnId = request.getSuccess_txn_id(); // Same here
    boolean error = request.isError(); // get request parameters properly
    String errMsg = request.getError_msg(); // get request parameters properly
    long lastSuccessFinishedTime = request.getLast_success_finished_time(); // use getter method

    Dictionary dictionary = dictionariesMapById.get(leaderDictionaryId);
    if (dictionary == null) {
        LOG.warn("Dictionary does not exist when sync the refresh state to follower, dictionary id: {}", leaderDictionaryId);
        return result;
    }
    if (!error) {
        GlobalStateMgr.getCurrentState().getDictionaryMgr().updateLastSuccessTxnId(leaderDictionaryId, leaderSuccessTxnId);
        dictionary.setFinished(lastSuccessFinishedTime); // update with correct finished time
        dictionary.setErrorMsg(""); // reset error msg
    } else if (dictionary.getIgnoreFailedRefresh()) {
        dictionary.resetStateBeforeRefresh();
        dictionary.setErrorMsg("Cancelled and rollback to previous state, errMsg: " + errMsg);
    } else {
        dictionary.setCancelled();
        dictionary.setErrorMsg(errMsg);
    }

    return result;
}

Additional explanation:

  1. Getter Methods: Ensure you're using getter methods (request.getDictionary_id(), etc.) instead of accessing fields directly. This follows Java best practices for encapsulation.
  2. Proper Error Handling: Consistently handle any potential errors by accurately updating the Dictionary object.
  3. Consistency Check: Ensure consistency checks between leader and followers are adequately addressed, which includes proper synchronization mechanisms if needed.

@srlch srlch force-pushed the bugfix_dictionary_follower branch from a70d0a6 to 5392c2f Compare July 5, 2024 02:12
@wanpengfei-git wanpengfei-git requested a review from a team July 5, 2024 06:01
@wanpengfei-git wanpengfei-git removed the request for review from a team July 5, 2024 06:43
@srlch srlch force-pushed the bugfix_dictionary_follower branch from 462ec7c to 173b5e1 Compare July 5, 2024 06:48
@srlch srlch requested a review from a team as a code owner July 5, 2024 06:55
@srlch srlch force-pushed the bugfix_dictionary_follower branch 4 times, most recently from 443058a to 29453f1 Compare July 9, 2024 02:17
@srlch srlch changed the title [WIP]follower sync dictionary refresh state [BugFix] follower sync dictionary refresh state Jul 9, 2024
@srlch srlch force-pushed the bugfix_dictionary_follower branch from 62134de to 3d9d702 Compare July 11, 2024 06:16
Why I'm doing:
In the current implementations, follower will not be synced any
information about the dictionary refreshing which is a problem.

What I'm doing:
Reuse dictionaryMgrInfo and DICTIONARY_MODIFY log to save the
dictionary state information.

Signed-off-by: srlch <[email protected]>
Signed-off-by: srlch <[email protected]>
@srlch srlch force-pushed the bugfix_dictionary_follower branch from 3d9d702 to 33738ca Compare July 11, 2024 06:26
Signed-off-by: srlch <[email protected]>
Copy link

sonarcloud bot commented Jul 11, 2024

Copy link

[FE Incremental Coverage Report]

pass : 64 / 74 (86.49%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 com/starrocks/catalog/DictionaryMgr.java 40 50 80.00% [124, 494, 495, 496, 497, 499, 501, 502, 504, 536]
🔵 com/starrocks/persist/DictionaryMgrInfo.java 9 9 100.00% []
🔵 com/starrocks/catalog/Dictionary.java 15 15 100.00% []

Copy link

[BE Incremental Coverage Report]

pass : 0 / 0 (0%)

@HangyuanLiu HangyuanLiu merged commit 7cb4a4a into StarRocks:main Jul 12, 2024
49 of 50 checks passed
Copy link

@Mergifyio backport branch-3.3

@github-actions github-actions bot removed the 3.3 label Jul 12, 2024
Copy link
Contributor

mergify bot commented Jul 12, 2024

backport branch-3.3

✅ Backports have been created

mergify bot pushed a commit that referenced this pull request Jul 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants