feat(python): Improve auto-initialization of credential providers

[nameexhaustion]

• For AWS, this enables auto-initialization of CredentialProviderAWS if storage_options contains only an aws_region config key.
• In general, for all supported cloud types:
  • Allows credential providers to be auto-initialized if storage_options contains client config keys¹
• Adjusts checking of keys on the Python-side to be case-insensitive

Footnotes

1. https://docs.rs/object_store/latest/object_store/enum.ClientConfigKey.html ↩

[codecov]

Codecov Report

Attention: Patch coverage is 54.16667% with 11 lines in your changes missing coverage. Please review.

Project coverage is 79.63%. Comparing base (064e661) to head (deecd6b).
Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
py-polars/polars/io/cloud/credential_provider.py	54.16%	7 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #20843      +/-   ##
==========================================
- Coverage   79.70%   79.63%   -0.07%     
==========================================
  Files        1569     1571       +2     
  Lines      223017   223166     +149     
  Branches     2546     2554       +8     
==========================================
- Hits       177746   177724      -22     
- Misses      44687    44857     +170     
- Partials      584      585       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ion-elgreco]

@nameexhaustion any reason why you are not using the CredentialProvider trait from object store?

[nameexhaustion]

@nameexhaustion any reason why you are not using the CredentialProvider trait from object store?

It is being used here -

polars/crates/polars-io/src/cloud/credential_provider.rs

Lines 473 to 480 in 24fccc4

	impl IntoCredentialProvider for PythonCredentialProvider {
	#[cfg(feature = "aws")]
	fn into_aws_provider(self) -> object_store::aws::AwsCredentialProvider {
	use polars_error::{to_compute_err, PolarsResult};
	use crate::cloud::credential_provider::{
	CredentialProviderFunction, ObjectStoreCredential,
	};

We have it set up such that the Python credential providers eventually get converted into the *CredentialProvider trait objects from object_store. This lets us lean on the Python-side libraries for authentication logic.