[feature] s3: add option to treat bucket as public

[Follpvosten]

Is your feature request related to a problem ?

there have been recurring issues with signed S3 URL caches not being invalidated properly.

at the same time, for my specific usecase (public profile, no private media at all), it makes no sense to set my S3 bucket to private (it is in fact set to public right now).

Describe the solution you'd like.

for people with the same usecase as me, it would be nice to have an option like storage-s3-public-bucket with a default of false.

public buckets don't require URL signing, so generating a valid link that also doesn't expire becomes simple string concatenation, which means they probably don't even need to be cached because they're really cheap to create, and if they do get cached, it would be impossible for them to get invalidated, also fixing the caching problem.

Describe alternatives you've considered.

looking at the existing settings, the only real option to permanently fix this would be enabling the storage-s3-proxy, unnecessarily creating load on my server.

the other alternative is obviously "get the caching right and never break it again", but that sounds a bit like proposing to "just write memory safe C" :p

Additional context.

right now, on 0.7.0-rc1, the URL cache seems broken for avatars, leading to my profile looking like this:

and it's also broken for all federated avatars, not just mine, so I can watch the avatars of my follows slowly disappear as the URLs expire.

I've had similar issues (which have been fixed since) with custom emoji before.

[daenney]

I'm curious about something; GTS generates unguessable filenames (at least from looking at my own statusses with media attachments). Is there an additional benefit that's realised by doing S3 with URL pre-signing on top of that? As far as I can see we don't have such a mechanism for the boring/on-disk media storage and we're (implicitly) relying on the unguessable nature of the files in that case.

[LittleFox94]

I'm curious about something; GTS generates unguessable filenames (at least from looking at my own statusses with media attachments). Is there an additional benefit that's realised by doing S3 with URL pre-signing on top of that? As far as I can see we don't have such a mechanism for the boring/on-disk media storage and we're (implicitly) relying on the unguessable nature of the files in that case.

Access to objects could be limited at a later point by doing this and not have everything publicly reachable in S3. Consider e.g. an attachment to a followers-only post. All people following you can load it (as they authenticate on GTS) and now have the S3 URL of the attachment. You block a person, but they still have the S3 URL and can just load the attachment whenever they want, until it is deleted. With pre-signed URLs we can give time-limited access to objects. GTS only gives a now-valid URL when you authenticate (as a follower in this example), when someone is blocked (or even just unfollows), they can only use the S3 URL for a short (*) amount of time after that.

(*: iirc GTS uses a day to have one URL per day per object it gives out to all allowed requesters, which then allows browsers to cache media - another signature and expiration changes the URL, making browsers unable to recognize it as already cached)

[tsmethurst]

This is now closed by #3046, since you can use storage-s3-redirect-url to send requests directly to your S3 bucket without using pre-signed URLs.