RawKV support SST Ingest using KVStream interface #233

marsishandsome · 2021-07-14T05:15:34Z

What problem does this PR solve?

This is a subtask of tikv/tikv#10563.

What is changed and how it works?

TiKV java client support Bulk Load using the ingest API.

support RawKV Region Split
support switch TiKV to import mode
support rawWrite API to ingest data

this PR depends on:

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Code changes

Has exported function/method change
Has exported variable/fields change
Has interface methods change
Has persistent data change

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Related changes

Need to cherry-pick to the release branch
Need to update the documentation
Need to be included in the release note

marsishandsome · 2021-07-19T08:25:02Z

/run-all-tests tikv=pr/10524

marsishandsome · 2021-07-20T02:30:37Z

/run-all-tests tikv=pr/10524

src/main/java/org/tikv/common/TiSession.java

andylokandy · 2021-07-20T08:43:44Z

src/main/java/org/tikv/common/TiSession.java

+    return splitRegion(splitKeys, backOffer, 1);
+  }
+
+  private List<Metapb.Region> splitRegion(


Why use recursive function to retry rather than a loop?

It's not a simple retry, e.g.

we need to split 6 keys

2 keys are split successfully, 4 keys are failed

retry to split the 4 failed keys

It's easy to use a recursive function to implement this logic.

andylokandy · 2021-07-20T09:02:17Z

src/main/java/org/tikv/common/importer/SwitchTiKVModeClient.java

+  public void keepTiKVToImportMode() {
+    ingestScheduledExecutorService.scheduleAtFixedRate(
+        this::switchTiKVToImportMode, 0, KEEP_TIKV_TO_IMPORT_MODE_PERIOD, TimeUnit.SECONDS);
+  }
+
+  public void stopKeepTiKVToImportMode() {
+    ingestScheduledExecutorService.shutdown();
+  }


I think it's better to keep this two function private and be called automatically by switchTiKVToXXXMode()

switchTiKVToNormalMode is called one time, while keepTiKVToImportMode will start a scheduler and call switchTiKVToImportMode every 2 minute.

The name keepTiKVToImportMode is used in order that

let users know that a scheduler will be started after calling keepTiKVToImportMode

stopKeepTiKVToImportMode must be called to stop the scheduler

why do we need to call keep import repeatedly?

Because I presumed that there will be multiple import clients. But now I know that the API is for example purpose.

src/main/java/org/tikv/raw/RawKVClient.java

src/main/java/org/tikv/common/importer/ImporterClient.java

andylokandy · 2021-07-20T09:26:12Z

src/main/java/org/tikv/common/importer/ImporterClient.java

+      returnNumber = 0;
+      for (ImporterStoreClient client : clientList) {
+        if (client.isRawWriteResponseReceived()) {
+          returnNumber++;


the client is not removed from clientList, will it count twice in the next iteration?

refactor this code, PTAL

andylokandy · 2021-07-20T09:28:28Z

src/main/java/org/tikv/common/importer/ImporterClient.java

+
+  private void ingest() throws GrpcException {
+    int returnNumber = 0;
+    while (returnNumber < clientList.size()) {


Will this dead loop if a client never succeed?

then will go to client.hasRawWriteResponseError() and throw new GrpcException

src/main/java/org/tikv/common/importer/ImporterStoreClient.java

marsishandsome · 2021-07-21T09:13:05Z

@andylokandy PTAL

Signed-off-by: marsishandsome <[email protected]>

marsishandsome · 2021-07-22T02:18:42Z

/run-all-tests

birdstorm

LGTM

marsishandsome force-pushed the feature/support-rawkv-import branch 5 times, most recently from fd126c5 to e03b641 Compare July 14, 2021 05:49

marsishandsome mentioned this pull request Jul 14, 2021

RawKV support Bulk Load tikv/tikv#10563

Open

9 tasks

marsishandsome force-pushed the feature/support-rawkv-import branch 17 times, most recently from a43e7e2 to 6ec7e09 Compare July 19, 2021 08:24

marsishandsome force-pushed the feature/support-rawkv-import branch 4 times, most recently from fca1963 to 3dc1da2 Compare July 20, 2021 07:13

marsishandsome changed the title ~~[WIP] RawKV support SST Ingest using KVStream interface~~ RawKV support SST Ingest using KVStream interface Jul 20, 2021

marsishandsome force-pushed the feature/support-rawkv-import branch from 3dc1da2 to 7b47468 Compare July 20, 2021 07:34

marsishandsome requested review from Little-Wallace, birdstorm and andylokandy and removed request for birdstorm July 20, 2021 07:35

marsishandsome force-pushed the feature/support-rawkv-import branch from 7b47468 to a24b2ed Compare July 20, 2021 07:40

andylokandy reviewed Jul 20, 2021

View reviewed changes

marsishandsome force-pushed the feature/support-rawkv-import branch 2 times, most recently from bdba26f to 40686fc Compare July 20, 2021 09:59

tikv deleted a comment from andylokandy Jul 21, 2021

marsishandsome requested a review from birdstorm July 22, 2021 02:10

marsishandsome added 4 commits July 22, 2021 10:12

support RawKV ingest API

e15661a

Signed-off-by: marsishandsome <[email protected]>

address code review

79a6e9e

Signed-off-by: marsishandsome <[email protected]>

address code review

66cc979

Signed-off-by: marsishandsome <[email protected]>

address code review

92471c2

Signed-off-by: marsishandsome <[email protected]>

marsishandsome force-pushed the feature/support-rawkv-import branch from 356fa69 to eecb406 Compare July 22, 2021 02:16

fix

4945902

Signed-off-by: marsishandsome <[email protected]>

marsishandsome force-pushed the feature/support-rawkv-import branch from eecb406 to 4945902 Compare July 22, 2021 02:17

test with tikv master branch

9fc01fe

Signed-off-by: marsishandsome <[email protected]>

andylokandy approved these changes Jul 22, 2021

View reviewed changes

ti-srebot added the status/LGT1 label Jul 22, 2021

birdstorm approved these changes Jul 22, 2021

View reviewed changes

ti-srebot removed the status/LGT1 label Jul 22, 2021

ti-srebot approved these changes Jul 22, 2021

View reviewed changes

ti-srebot added the status/LGT2 label Jul 22, 2021

marsishandsome merged commit 798244c into tikv:master Jul 22, 2021

marsishandsome mentioned this pull request Aug 2, 2021

TiKV Java Client support ingest API #245

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RawKV support SST Ingest using KVStream interface #233

RawKV support SST Ingest using KVStream interface #233

marsishandsome commented Jul 14, 2021 •

edited

Loading

marsishandsome commented Jul 19, 2021

marsishandsome commented Jul 20, 2021

andylokandy Jul 20, 2021

marsishandsome Jul 21, 2021

andylokandy Jul 20, 2021

marsishandsome Jul 21, 2021

birdstorm Jul 22, 2021

andylokandy Jul 22, 2021

andylokandy Jul 20, 2021

marsishandsome Jul 21, 2021

andylokandy Jul 20, 2021

marsishandsome Jul 21, 2021

marsishandsome commented Jul 21, 2021

marsishandsome commented Jul 22, 2021

birdstorm left a comment

RawKV support SST Ingest using KVStream interface #233

RawKV support SST Ingest using KVStream interface #233

Conversation

marsishandsome commented Jul 14, 2021 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

marsishandsome commented Jul 19, 2021

marsishandsome commented Jul 20, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marsishandsome commented Jul 21, 2021

marsishandsome commented Jul 22, 2021

birdstorm left a comment

Choose a reason for hiding this comment

marsishandsome commented Jul 14, 2021 •

edited

Loading