Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Support Backup/Restore for external catalog #52895

Merged
merged 6 commits into from
Nov 15, 2024
Merged

[Feature] Support Backup/Restore for external catalog #52895

merged 6 commits into from
Nov 15, 2024

Conversation

srlch
Copy link
Contributor

@srlch srlch commented Nov 14, 2024 •
edited by jaogoy
Loading

In this PR, we support Backup/Restore for External Catalog. User can Backup All of or part of External Catalog metadata into snapshot file and restore it into a new cluster.

The following summarizes the behavioral changes caused by this PR:

  1. New syntax to support Backup/Restore for External Catalog, here is the new part for it (old syntax is not shown here):
BACKUP { ALL EXTERNAL CATALOGS | EXTERNAL CATALOG[S] <catalog_name> [, ...] }
SNAPSHOT <snapshot_name> TO <repo_name>
[ PROPERTIES ]

RESTORE SNAPSHOT <snapshot_name> FROM <repo_name>
[ ALL EXTERNAL CATALOGS | EXTERNAL CATALOG[S] <catalog_name> AS <alias> [, ...] ]
[ PROPERTIES ]


-- CANCEL
CANCEL { BACKUP | RESTORE } FOR EXTERNAL CATALOG[S]
-- CANCEL { BACKUP | RESTORE } FROM [ DATABASE ] <db_name>
  1. For EXTERNAL CATALOG Backup/Restore, no db info and ON clause can be specified.
  2. Only one Backup/Restore job can be executed at the same time in the cluster.
  3. default_catalog is not support.

Fixes #52902

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 3.3
    • 3.2
    • 3.1
    • 3.0
    • 2.5

@srlch srlch requested review from a team as code owners November 14, 2024 06:42
@wanpengfei-git wanpengfei-git requested a review from a team November 14, 2024 06:42
@mergify mergify bot assigned srlch Nov 14, 2024
@srlch srlch changed the title [WIP] [Feature] Support Backup/Restore for external catalog [Feature] Support Backup/Restore for external catalog Nov 14, 2024
@srlch srlch force-pushed the support_backup_restore_external_catalog branch from eca2d7f to b174cba Compare November 14, 2024 10:57
@srlch
fix
22562a9 
Signed-off-by: srlch <[email protected]>
@srlch
fix
942c7e4 
Signed-off-by: srlch <[email protected]>
@srlch
fix
62c2452 
Signed-off-by: srlch <[email protected]>
@srlch
fix comment
8008217 
Signed-off-by: srlch <[email protected]>
jaogoy
jaogoy previously approved these changes Nov 15, 2024
View reviewed changes
@srlch
fix
dbd351b 
Signed-off-by: srlch <[email protected]>
@sonarcloud SonarCloud
Copy link

sonarcloud bot commented Nov 15, 2024

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@github-actions GitHub Actions
Copy link

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

@github-actions GitHub Actions
Copy link

[FE Incremental Coverage Report]

❌ fail : 130 / 228 (57.02%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 com/starrocks/catalog/Catalog.java 0 2 00.00% [74, 75]
🔵 com/starrocks/sql/analyzer/AuthorizerStmtVisitor.java 19 49 38.78% [2119, 2120, 2138, 2139, 2141, 2143, 2144, 2146, 2147, 2148, 2149, 2150, 2152, 2153, 2155, 2156, 2157, 2158, 2159, 2160, 2161, 2162, 2163, 2239, 2241, 2242, 2244, 2245, 2246, 2247]
🔵 com/starrocks/server/CatalogMgr.java 4 10 40.00% [100, 101, 165, 166, 168, 169]
🔵 com/starrocks/qe/ShowExecutor.java 4 10 40.00% [1819, 1820, 1821, 1857, 1858, 1859]
🔵 com/starrocks/backup/RestoreJob.java 13 25 52.00% [480, 481, 486, 487, 488, 489, 1492, 1493, 1494, 1496, 1498, 1499]
🔵 com/starrocks/backup/BackupJob.java 5 9 55.56% [475, 476, 477, 479]
🔵 com/starrocks/sql/ast/CancelBackupStmt.java 3 5 60.00% [31, 32]
🔵 com/starrocks/backup/BackupHandler.java 40 64 62.50% [292, 293, 294, 298, 300, 303, 304, 323, 326, 330, 526, 580, 581, 582, 583, 584, 587, 588, 589, 592, 595, 596, 599, 709]
🔵 com/starrocks/sql/ast/CatalogRef.java 13 20 65.00% [49, 53, 57, 58, 63, 68, 78]
🔵 com/starrocks/sql/ast/AbstractBackupStmt.java 7 9 77.78% [143, 144]
🔵 com/starrocks/sql/analyzer/BackupRestoreAnalyzer.java 13 16 81.25% [96, 308, 309]
🔵 com/starrocks/sql/ast/BackupStmt.java 2 2 100.00% []
🔵 com/starrocks/sql/ast/RestoreStmt.java 2 2 100.00% []
🔵 com/starrocks/backup/BackupMeta.java 5 5 100.00% []

@github-actions GitHub Actions
Copy link

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

@wyb wyb enabled auto-merge (squash) November 15, 2024 06:59
@andyziye andyziye merged commit 2225f3c into StarRocks:main Nov 15, 2024
55 of 59 checks passed
@wanpengfei-git wanpengfei-git mentioned this pull request Nov 15, 2024
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xiangguangyxg xiangguangyxg xiangguangyxg left review comments

@gengjun-git gengjun-git gengjun-git approved these changes

@jaogoy jaogoy jaogoy approved these changes

@kangkaisen kangkaisen kangkaisen approved these changes

@wyb wyb wyb approved these changes

Assignees

@srlch srlch

Labels
behavior_changed META-REVIEW version:3.4
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature] Support Backup/Restore for external catalog
8 participants
@srlch @wyb @kangkaisen @jaogoy @gengjun-git @xiangguangyxg @wanpengfei-git @andyziye