[Improve] support gz compress in streamload #434
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DongLiang-0
reviewed
Jul 18, 2024
| @@ -33,4 +33,6 @@ public class LoadConstants { | |||
| public static final String READ_JSON_BY_LINE = "read_json_by_line"; | |||
| public static final String GROUP_COMMIT = "group_commit"; | |||
| public static final String GROUP_COMMIT_OFF_MODE = "off_mode"; | |||
| public static final String COMPRESS_TYPE = "compress_type"; | |||
| public static final String COMPRESS_TYPE_GZ = "gz"; | |||
There was a problem hiding this comment.
Is it more clear to use gzip here?
There was a problem hiding this comment.
This depends on streamload and must be gz.
DongLiang-0
reviewed
Jul 18, 2024
| @@ -128,6 +132,7 @@ public DorisBatchStreamLoad( | |||
| && !loadProps | |||
| .getProperty(GROUP_COMMIT) | |||
| .equalsIgnoreCase(GROUP_COMMIT_OFF_MODE); | |||
| this.enableGzCompress = loadProps.getProperty(COMPRESS_TYPE, "").equals(COMPRESS_TYPE_GZ); | |||
There was a problem hiding this comment.
Do we need to check whether the current file format is csv? If the data here is json, will gzip be turned on?
There was a problem hiding this comment.
If json and compression are configured, streamload will report an error when writing.
CalvinKirs
approved these changes
Jul 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Issue Number: close #xxx
Problem Summary:
Currently streamload supports gz compression in csv format, which requires the client to compress the stream
Checklist(Required)
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...