Fallback to anonymous credentials in S3 client

[bentsherman]

Close #3989

Based on the default S3 credentials provider, which uses the default provider chain with a fallback to anonymous credentials. Since Nextflow doesn't use the default provider chain anymore, I think we have to add this fallback ourselves.

[pditommaso]

Nice point. However, I'm not understanding why it should happen. when the anonymous is specified this branch should be taken that uses the AnonymousAWSCredentials in explicit manner

nextflow/plugins/nf-amazon/src/main/nextflow/cloud/aws/nio/S3FileSystemProvider.java

Lines 838 to 839 in 1daebee

	log.debug("Creating AWS S3 client with anonymous credentials");
	client = new S3Client(new AmazonS3Client(new AnonymousAWSCredentials(), clientConfig));

Also worth adding an integration tests in the tests/ folder

[bentsherman]

I guess it's because the anonymous property isn't set:

nextflow/plugins/nf-amazon/src/main/nextflow/cloud/aws/nio/S3FileSystemProvider.java

Lines 836 to 840 in 1daebee

	final boolean anonymous = "true".equals(props.getProperty("anonymous"));
	if( anonymous ) {
	log.debug("Creating AWS S3 client with anonymous credentials");
	client = new S3Client(new AmazonS3Client(new AnonymousAWSCredentials(), clientConfig));
	}

As for the e2e test, how would we negate the CI runner's AWS creds before running the test?

[bentsherman]

I see, you're talking about aws.client.anonymous in the Nextflow config. But if you enable that option, it looks like you would only be able to access public buckets. What if you want to access public and private buckets in the same run?

[bentsherman]

Since I'll be out, if you can get it to work, a simple e2e test would be:

println file('s3://ngi-igenomes/igenomes/Homo_sapiens/Ensembl/GRCh37/Annotation/README.txt').text

[pditommaso]

Ok, added the integration tests and refactored the anonymous class to S3CredentialsProvider class

[bentsherman]

Where is the integration test? I don't see it in the PR

[pditommaso]

You are right. It went lost 👉 680f0a9

[bentsherman]

@pditommaso should we deprecate the aws.client.anonymous config option? It seems useless if Nextflow can fallback to anonymous creds anyway.

[pditommaso]

I still can be useful to avoid the normal auth flow when using non-aws cloud, I guess