Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: make fuse_amend implementation more convenient for on-premise deployments #17147

Merged

Conversation

dantengsky
Copy link
Member

@dantengsky dantengsky commented Dec 31, 2024

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Since fuse_amend is currently implemented using the AWS Rust SDK, some behaviors may be inconsistent with other components based on OpenDAL, especially in on-premise deployment. e.g.

  • Different default TLS cert verification strategy
    For instance, if the HTTPS service's certificate is a CA certificate (rather than an end-entity certificate), OpenDAL (using OpenSSL) will not reject the connection. However, the default configuration of the AWS Rust SDK (using Rustls) will reject such certificates. Other similar issues may also occur.

  • The configuration of the endpoint URL may require a more flexible approach in on-premise deployment

    For on-premise deployments of s3-compatible services, the choice of addressing style for buckets may vary (e.g., path-style vs. virtual-hosted-style addressing).

To better accommodate on-premise deployment scenarios, although this PR continues to use the AWS Rust SDK, it forces the usage of an native_tls based HTTPS connector.

Besides, two new settings have been introduced to allow more flexible configuration:

  • premise_deploy_danger_amend_accept_invalid_cert

    When set to 1, fuse_amend will accept invalid certificates during execution. The default value is 0.
    This parameter is intended solely for diagnosing issues in on-premise deployments and should not be enabled in production environments.

    When enabled (set to 1), domain name validation will also be skipped.

  • premise_deploy_amend_force_path_style

    When set to 1, fuse_amend will use path-style URIs to access buckets during execution. The default value is 1.

Misc:

  • Some unused crates are removed
  • Verbose error message for list_object_versions failure

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - use existing tests

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions bot added the pr-refactor this PR changes the code base without new features or bugfix label Dec 31, 2024
@dantengsky dantengsky force-pushed the chore-use-openssl-for-rust-aws-sdk branch 3 times, most recently from 43d3bc2 to 09ddd5e Compare December 31, 2024 10:56
@dantengsky dantengsky force-pushed the chore-use-openssl-for-rust-aws-sdk branch from 09ddd5e to 547f396 Compare December 31, 2024 10:57
@dantengsky dantengsky marked this pull request as ready for review January 2, 2025 01:18
@BohuTANG BohuTANG merged commit 749ca84 into databendlabs:main Jan 2, 2025
75 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-refactor this PR changes the code base without new features or bugfix
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants