Delete file from AWS when expired

[lufficc]

When uploaded a file, no one downloads it. After it expired, this record will delete from redis, but not delete from AWS , right?

[ehuggett]

I can't find it now but i remember seeing some discussion about this, i think it was suggesting something similar to https://aws.amazon.com/blogs/aws/amazon-s3-object-expiration/ was in use?

The expiry is rounded up to the "next midnight" and the link seems to suggest the only guarantee is that they will be deleted by the midnight after that one (as the rules are only applied once per day at an unspecified time).

[dannycoates]

@ehuggett is correct. We use S3 lifecycle expiration to delete the file from AWS.

Here's our exact portion of configuration for this bucket:

      LifecycleConfiguration:
        Rules:
        - ExpirationInDays: 1
          Status: Enabled

[ehuggett]

Would there be any financial benefit to deleting from AWS sooner?

• Delete rule only runs once every 24 hours
• Assuming every 24 hours, then up to <24 hours worth of expired data is in storage.
• so up to 48 hours "worth" of storage is used to run the service

It should be possible to avoid it

• Delete requests are free (and a single request can delete 1000 objects)
• Redis can send a notification upon expire events, but not in its default configuration as they seem to be resource expensive (might not be suitable/sensible, but its just one of many options)
• It would be a very simple way to trigger the deletion of the data on s3 or send push notifications about expiry etc
• would allow a storage backend with no concept of an expiry to be used in the future (or by others now)
• but leave the s3 expiry rule in place, just in case our own scheduled deletion fails for any reason

[dannycoates]

Would there be any financial benefit to deleting from AWS sooner?

Possibly. Cost cutting isn't our top priority at this point but I think we'd welcome patches that saved us money :)