Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

defaults for storage_options for the s3 protocol #222

Closed
keewis opened this issue Aug 20, 2024 · 3 comments
Closed

defaults for storage_options for the s3 protocol #222

keewis opened this issue Aug 20, 2024 · 3 comments

Comments

@keewis
Copy link
Contributor

keewis commented Aug 20, 2024

There are multiple different ways of how to authenticate S3: passing explicit kwargs to storage_options, but also environment variables and a session profile (where the credentials are read from a config file, like .aws/credentials).

The current defaults in

protocol_defaults = {"key": "", "secret": "", "anon": True}
assume we always want to use kwargs for S3, but that will actively prevent other methods from working. For example:

from aiobotocore.session import AioSession

open_virtual_dataset(url, reader_options={"storage_options": {"session": AioSession(profile="myprofile")}})

will not use the passed profile because s3fs / aiobotocore prioritize the kwargs.

The only thing I can think of to fix this is to simply remove key and secret, but not sure if that's the best option.

@TomAugspurger
Copy link
Contributor

IMO, virtualizarr shouldn't have default storage options. I don't think it's in a better position to pick defaults than the underlying library.

@norlandrhagen
Copy link
Collaborator

Totally agree. This PR takes a stab at removing those defaults.

@keewis
Copy link
Contributor Author

keewis commented Aug 26, 2024

should have been closed by #228

@keewis keewis closed this as completed Aug 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants