Add prefix to _s3_exists_dir to avoid permissions issues

[ericphanson]

The comment here is wrong; you can use both start-after and prefix on AWS (see the examples here: https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#API_ListObjectsV2_Examples). And in fact, if you have per-prefix access control setup, you need to restrict your queries to the prefix to not get access denied errors.

I am guessing the comment refers to Minio, where this can be solved by a dispatch in Minio.jl. We can see what the tests say.

cc @christopher-dG @ExpandingMan

[ericphanson]

bors try

[bors]

try

Build failed:

• Julia 1.3 - ubuntu-latest - x64

[ExpandingMan]

Getting this to work reliably was extremely tricky and subtle, even if that comment is wrong it seems likely this will break. Are you sure it will still pick up the directory in all cases with both arguments?

Can you be more specific about what permission issue the existing code is causing?

[omus]

Using IAM permissions that use conditional s3:prefix restrictions is when this problem occurs. There's an example of an IAM permissions policy using that here: https://aws.amazon.com/premiumsupport/knowledge-center/iam-s3-user-specific-folder/

We should definitely be adding a test for this behaviour.

[omus]

For testing we should probably create an IAM role and an associated policy which creates the original problem. Ideally we'd do use something like a CFN template to create these resources similar to: https://github.com/JuliaCloud/AWS.jl/blob/v1.61.0/test/resources/aws_jl_test.yaml. Unfortunately this project doesn't have an existing setup for that.

[ExpandingMan]

Yikes. Man, AWS is so painful sometimes.

Ok, well of course after you do whatever it takes to get tests passing on AWS I will implement new (or rather old) methods in Minio.jl as needed.

[mattBrzezinski]

I feel like we need to vamp up our test cases for this stuff. From personal experience I've found it's very very tricky to get right.

[iamed2]

If the path already ends with /, shouldn't any results that show up with the prefix path indicate the dir exists? Could we just check for:

q = Dict("delimiter" => "", "max-keys" => 1, "prefix" => path)

?

[kleinschmidt]

Just to verify what's blocking this:

• we need more tests to reproduce this current behavior
• this maybe requires setting up a role with the necessary permissions (or lack thereof)

Is there anything else?

[omus]

Superseded by #289