Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Every setting is configurable per upload/download. #32

Open
graebm opened this issue Jul 18, 2024 · 1 comment
Open

Every setting is configurable per upload/download. #32

graebm opened this issue Jul 18, 2024 · 1 comment

Comments

@graebm
Copy link
Contributor

graebm commented Jul 18, 2024

If this is going to sit under other systems someday, it must be possible for 2 downloads to be happening at the same time, each with totally different settings (e.g. different buckets, different regions, different part sizes).

aws-c-s3 started with settings such as part-size living on the s3_client. When it was integrated into other pre-existing transfer managers, configured part-size per-object. Eventually, all these settings had to be made configurable per download/upload.

Seems like there are 3 approaches we can take here, using part_size as an example:

  • part-size is set per download()
  • part-size is set on the Downloader, but can be overridden per download()
  • part-size is set on the Downloader but there's no real downside to a user creating N uniqueDownloaders to do N parallel downloads

There might be exceptions, for settings that are more "global" like max-memory-usage, max-bandwidth, etc

@aajtodd
Copy link
Contributor

aajtodd commented Jul 19, 2024

Not sure which way we want to go but the SDK uses the last option.

Common settings (retries, http client, etc) are set on the client but can be overridden per operation.

use aws_config::{BehaviorVersion, Region};

let config = aws_config::defaults(BehaviorVersion::latest())
    .region("us-east-1")
    .load()
    .await;

let s3 = aws_sdk_s3::Client::new(&config);

// Will be sent to "us-east-1"
s3.list_buckets()
    .send()
    .await?;

// Unset fields will default to using the original config value
let modified = aws_sdk_s3::Config::builder()
    .region(Region::from_static("us-west-2"));

// Will be sent to "us-west-2"
s3.list_buckets()
     // Creates a CustomizableOperation
    .customize()
    .config_override(modified)
    .send()
    .await?;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants