refactor: make fuse_amend
implementation more convenient for on-premise deployments
#17147
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
Since
fuse_amend
is currently implemented using the AWS Rust SDK, some behaviors may be inconsistent with other components based on OpenDAL, especially in on-premise deployment. e.g.Different default TLS cert verification strategy
For instance, if the HTTPS service's certificate is a CA certificate (rather than an end-entity certificate), OpenDAL (using OpenSSL) will not reject the connection. However, the default configuration of the AWS Rust SDK (using Rustls) will reject such certificates. Other similar issues may also occur.
The configuration of the endpoint URL may require a more flexible approach in on-premise deployment
For on-premise deployments of s3-compatible services, the choice of addressing style for buckets may vary (e.g., path-style vs. virtual-hosted-style addressing).
To better accommodate on-premise deployment scenarios, although this PR continues to use the AWS Rust SDK, it forces the usage of an
native_tls
based HTTPS connector.Besides, two new settings have been introduced to allow more flexible configuration:
premise_deploy_danger_amend_accept_invalid_cert
When set to
1
,fuse_amend
will accept invalid certificates during execution. The default value is0
.This parameter is intended solely for diagnosing issues in on-premise deployments and should not be enabled in production environments.
When enabled (set to
1
), domain name validation will also be skipped.premise_deploy_amend_force_path_style
When set to
1
,fuse_amend
will use path-style URIs to access buckets during execution. The default value is1
.Misc:
list_object_versions
failureTests
Type of change
This change is![Reviewable](https://camo.githubusercontent.com/1541c4039185914e83657d3683ec25920c672c6c5c7ab4240ee7bff601adec0b/68747470733a2f2f72657669657761626c652e696f2f7265766965775f627574746f6e2e737667)