[Enhancement] configure S3 client rename_file operation timeout (backport #48860) #49706
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I'm doing:
Currently, sr use object_storage_request_timeout_ms as the S3 client request timeout. But this parameter maybe too small in some case especially using by rename_file operation for large file (1GB, for example). Timeout maybe happen in this case
What I'm doing:
We introduce a new BE/CN config called
object_storage_rename_file_request_timeout_ms
and it is used as following:object_storage_rename_file_request_timeout_ms
>= 0, it will used for rename_file operation in S3 as the request timeout.object_storage_rename_file_request_timeout_ms
< 0, the timeout limit for rename_file operation in S3 will be determinded by object_storage_request_timeout_ms.Fixes #issue
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check:
This is an automatic backport of pull request #48860 done by [Mergify](https://mergify.com). ## Why I'm doing: Currently, sr use object_storage_request_timeout_ms as the S3 client request timeout. But this parameter maybe too small in some case especially using by rename_file operation for large file (1GB, for example). Timeout maybe happen in this case
What I'm doing:
We introduce a new BE/CN config called
object_storage_rename_file_request_timeout_ms
and it is used as following:object_storage_rename_file_request_timeout_ms
>= 0, it will used for rename_file operation in S3 as the request timeout.object_storage_rename_file_request_timeout_ms
< 0, the timeout limit for rename_file operation in S3 will be determinded by object_storage_request_timeout_ms.Fixes #issue
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist: