Don't silently swallow forbidden AWS GET requests (sscache S3 always cache miss)

[ezyang]

We were recently debugging an issue with sccache where all of our compilations were missing the cache (even though we were successfully writing to the cache.) Inspecting the log revealed that sccache was receiving a FORBIDDEN response and deciding this was a cache miss. It would be much better if these problems were logged as a separate statistic so that it would be more easy to diagnose when they occur. Extra brownie points if the statistics page gives suggestions for debugging these types of errors.

For anyone else dealing with this issue, the problem is that sccache does not apply AWS credentials when performing a GET request, so your bucket needs to be readable from the public, otherwise sccache will not be able to see it. Maybe this is an independent bug, but making our bucket public solved the problem for us.

[glandium]

S3 doesn't return 404s for missing objects, it always returns 403s, which is forbidden.

[ezyang]

If I have list access, it will return a 404 for missing objects. If I don't have access, that is a misconfiguration ;)

[jrobsonchase]

@ezyang We had a similar problem and tried to make a contribution to enable authenticated S3 GET requests in #135. Until that gets merged, we've been using my fork of sccache in our builds, so I've been keeping it somewhat up-to-date.

[ezyang]

@jechase That is great to hear. In our case making the cache public was an acceptable outcome, but if we ever need a private cache I know where I'll be looking :)