Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix #623] region may be missed #624

Merged
merged 5 commits into from
Jun 30, 2022

Conversation

TrafalgarRicardoLu
Copy link
Contributor

What problem does this PR solve?

Issue Number: close #623

Problem Description: If there is a region split between two queries, it may miss some regions.

What is changed and how does it work?

I change the time of getting region from before scan to after scan.

When we build client, we will use region info in regionCache. After region split, it's outdated.
But when client try to scan the region, it will get the newest region info from pb.

In the original code, we get region info before scan. So the region info is still outdated and it will use the old endKey to get next region. Using the old endKey may miss some regions that split during the two query.

I get the region info after scan to make sure the return value is newest.

Check List for Tests

This PR has been tested by at least one of the following methods:

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)

@shiyuhang0
Copy link
Collaborator

@iosmanthus PTAL

Signed-off-by: TrafalgarRicardoLu <[email protected]>
@codecov
Copy link

codecov bot commented Jun 27, 2022

Codecov Report

Merging #624 (6da3e99) into master (3724e87) will decrease coverage by 0.02%.
The diff coverage is 75.00%.

@@             Coverage Diff              @@
##             master     #624      +/-   ##
============================================
- Coverage     34.76%   34.73%   -0.03%     
- Complexity     1421     1424       +3     
============================================
  Files           278      278              
  Lines         17356    17358       +2     
  Branches       1972     1972              
============================================
- Hits           6034     6030       -4     
- Misses        10713    10718       +5     
- Partials        609      610       +1     
Impacted Files Coverage Δ
...ommon/operation/iterator/ConcreteScanIterator.java 0.00% <0.00%> (ø)
...g/tikv/common/operation/iterator/ScanIterator.java 78.94% <ø> (ø)
...ikv/common/operation/iterator/RawScanIterator.java 83.33% <100.00%> (+0.57%) ⬆️
...java/org/tikv/common/region/RegionStoreClient.java 52.49% <100.00%> (+0.06%) ⬆️
...rc/main/java/io/grpc/netty/NettyClientHandler.java 56.89% <0.00%> (-5.61%) ⬇️
...ty/handler/codec/http2/Http2ConnectionHandler.java 49.63% <0.00%> (+0.97%) ⬆️
.../main/java/org/tikv/common/region/RegionCache.java 50.92% <0.00%> (+1.85%) ⬆️
src/main/java/io/grpc/netty/WriteQueue.java 76.69% <0.00%> (+2.25%) ⬆️
...va/org/tikv/common/region/StoreHealthyChecker.java 79.74% <0.00%> (+7.59%) ⬆️
src/main/java/org/tikv/common/region/TiStore.java 74.35% <0.00%> (+12.82%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3724e87...6da3e99. Read the comment docs.

BackOffer backOffer = ConcreteBackOffer.newScannerNextMaxBackOff();
currentCache = client.scan(backOffer, startKey, version);
region = client.getRegion();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might add some comments about this update since the scan has some implicit side effects. In addition, this issue might occur in

TiRegion loadCurrentRegionToCache() throws GrpcException {
, the raw scan of client-java. We might add some mock tests for this interface, at least a TODO should be added.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have add comment and fix it in RawScanIterator. I think test it is not easy, so a TODO is added.

Signed-off-by: TrafalgarRicardoLu <[email protected]>
Signed-off-by: TrafalgarRicardoLu <[email protected]>
Signed-off-by: TrafalgarRicardoLu <[email protected]>
iosmanthus
iosmanthus previously approved these changes Jun 29, 2022
Copy link
Member

@iosmanthus iosmanthus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Signed-off-by: TrafalgarRicardoLu <[email protected]>
@ti-srebot
Copy link
Collaborator

@zhangyangyu, Thanks for your review. The bot only counts LGTMs from Reviewers and higher roles, but you're still welcome to leave your comments. You are not a reviewer or committer or co-leader or leader.

@sunxiaoguang
Copy link
Member

Thanks for your contribution!

Could you please add test code to pass the code coverage check, thanks.

Copy link
Member

@iosmanthus iosmanthus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@iosmanthus iosmanthus merged commit 5a757d9 into tikv:master Jun 30, 2022
@iosmanthus
Copy link
Member

Thanks for your contribution!

Could you please add test code to pass the code coverage check, thanks.

We may add some mock tests in the next pull request.

ti-srebot pushed a commit to ti-srebot/client-java that referenced this pull request Jun 30, 2022
@ti-srebot
Copy link
Collaborator

cherry pick to release-3.3 in PR #625

iosmanthus pushed a commit that referenced this pull request Jun 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Table not found when TiDB has a lots of tables in a DB
6 participants