Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PrestoS3ClientFactory should use the AWS Java SDK default credential provider #625

Closed
ddrinka opened this issue Apr 11, 2019 · 3 comments · Fixed by #741
Closed

PrestoS3ClientFactory should use the AWS Java SDK default credential provider #625

ddrinka opened this issue Apr 11, 2019 · 3 comments · Fixed by #741

Comments

@ddrinka
Copy link
Member

ddrinka commented Apr 11, 2019

The Glue Hive Metastore provider uses the default AWSGlueAsyncClientBuilder provided by the AWS SDK for Java to locate AWS credentials. This is good. There are numerous ways to provide the credentials to the plugin, and these are the ways we're used to providing credentials to services that run on AWS. These configuration mechanisms are also container-friendly.

On the other hand, the S3 client handles this process its own way, with only two included options for configuration, hardcoded credentials in the configuration files, and utilizing the InstanceProfileCredentialsProvider.

The S3 client should be able to retrieve credentials from AWS using the default provider, as Glue does.

Here are a couple other folks struggling with this over the years:
https://stackoverflow.com/questions/51527973/configure-presto-connector-options-by-environment-variables
https://stackoverflow.com/questions/41554020/presto-fails-to-recognize-aws-credentials-both-iam-and-keys
https://prestodb.slack.com/archives/C07JH9WMQ/p1556085524072500

@electrum
Copy link
Member

electrum commented Apr 11, 2019

This seems reasonable. We could add a new config, hive.s3.use-default-credentials, that when true, would return null as the credential provider rather than throwing an exception when no other credentials are available. This should be done for both PrestoS3ClientFactory and PrestoS3FileSystem. This config could default to true, but be set to false if someone wants to assure they don't pick up credentials automatically (when they intended to configure them explicitly).

@ddrinka
Copy link
Member Author

ddrinka commented May 15, 2019

@anoopj thanks for #741, that should close this issue.

@anoopj
Copy link
Member

anoopj commented May 19, 2019

@ddrinka, Yep, I'm fixing PrestoS3FileSystem and PrestoS3ClientFactory as part of #741

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging a pull request may close this issue.

3 participants