How to manage file references not uploaded directly from app?

[mdbd01]

Hi,

Thanks for a great library! I was wondering if you have any idea on how to "connect" with new files not being uploaded from our application? I'll give you an example

Normal procedure

1. User is signed in and can upload a cover image for a book. File is stored on S3, and our model has all info we need in book.cover_image.

What I'm trying to achieve
2. Let's say we add files directly to S3, from a Lambda function or that someone sends an email with a cover image of an book. How would we go on to "connect" this to an already existing book in our system if the file is already in an bucket?

All the best,
Mattias

[benkoshy]

Hi Mattias,

It's simply a quesiton of accessing the:

• S3 bucket, and then
• the s3 object

In order to connect to the former, you need access credentials, and in order to "connect" to the file, you need, minimally, the object "key". This key, is mostly held (for Shrine use cases) in a database record. I don't have enough details, but from your lambda, you'll have to access the relevant database record, or otherwise get a hold of the s3 object's key, in order to "connect" to it.

Once you have those details, connecting to the file should be trivial.

[mdbd01]

Hi,

Thanks for a quick reply!

I have access to the S3 bucket and the S3 object. When the files is stored it pings us with the object key. The "problem" is that our database hasn't stored the key with the record yet.

I still can't figure out exactly how to achieve this, but would it be possible to do something like this?

existing_file = s3.object("key") #=> #Aws::S3::Object
book.cover_image.attach(existing_file)
book.save

[benkoshy]

existing_file = s3.object("key") #=> #Aws::S3::Object
book.cover_image.attach(existing_file)
book.save

# Where is this code running? On your app, or on in the lambda

If it's running on your lambda, how would it work?

• Wouldn't you need a connection to your database to access the book record?
• Besides, the lambda only gives you something like 30 seconds, before it times out. That might be enough, but i want to avoid this. if something happens, then you're going to start dropping attachments. you might need an external service to ensure that the files are successfully promoted on the s3 bucket.

To avoid the X-Y problem would you be opposed to stating your use case:

• why are you doing this in a lambda?
• what are you trying to do?

[mdbd01]

The main thing I'm trying to achieve here is to be able to use S3 both as it's intended with uploads from the app etc, but also without interactions with our main application. I'd like to build features outside of our main app (monolith) that can make use to customers. But of course that our application also can use.

Today I've set up an test-enrivonment on AWS where I could send in email attachments, and they are stored on S3 with the help of a Lambda function. I can access the files with the S3 client, but I'd like to attach them to the corresponding book object so that when the user sees the cover image of the book, he doesn't notice any difference if it's uploaded with shrine or connected in the background somehow.

I can of course trigger a function with a webhook or so. But I'm not sure how to actually set the connection between the book object and the already existing file on S3.

Does it make any sense? =)

[janko]

If I understand correctly, you want to save attachment data do the database for files not uploaded by Shrine? You should be able to do this by generating a Shrine::UploadedFile object, setting the id to object path minus the storage prefix, and calling Attacher#set to write the data to the column:

uploaded_file = Shrine.uploaded_file(id: "key", storage: :store, metadata: { ... })
book.cover_image_attacher.set(uploaded_file)
book.save

If you want to extract metadata in the process, you could use the refresh_metadata plugin:

Shrine.plugin :refresh_metadata

uploaded_file = Shrine.uploaded_file(...)
uploaded_file.refresh_metadata!
book.cover_image_attacher.set(uploaded_file)
book.save

[benkoshy]

Glad you found the solution. Out of curiousity, how are you going to persist the book from the lambda function?

[mdbd01]

That is a good question, it was most used as an example. We already have all the books in the database, this is just an extra way to add images to it.

I'm intending to add several ways for users to email, upload, submit by forms etc to S3, and files would be added to the customer for sales import, product images, requested files etc.

[benkoshy]

I'm intending to add several ways for users to email, upload, submit by forms etc to S3, and files would be added to the customer for sales import, product images, requested files etc.

Janko has written very excellent guides which tell you how to implement use cases where users to upload objects directly to s3 buckets, without any need of doing it in the lambda (but i think i'm misunderstanding your use case though): https://shrinerb.com/docs/direct-s3

If you haven't already considered it, I would urge you to consider background jobs over the lambda, any day of the week.

[mdbd01]

Thanks! Background jobs would indeed be a good idea, but that would require the application to watch S3 for changes, and attach files as you described earlier? Or create an endpoint to receive files?

Now I can use AWS SES to receive attachments without the application involved, and then deal with each file when I want to. Sendgrid also has this incoming files feature.

I have a feeling that I'm making things more difficult for myself than I have to. But many services out there can add files to S3, and I'd like the ability to connect with them just through S3 instead of having them posting to an API endpoint.

Does it sound weird? :)

[benkoshy]

a huge benefit of shrine is the life cycle management and persistence of objects - shrine takes care of everything:

• uploading to cache
• promoting to storage
• persisting to the record with all the metadata in tact.

if you're using a lambda in anyway --> how are you going to persist it?

book = Book.find(id)
# lambda has no conception of Sequel / AR models!
# you'd have to connect to your DB, require the model files

A better approach might be investigate the use of SQS / SNS (or equivalent) to notify your app that an object has been uploaded to the bucket - I cannot comment specifically because I have never used these services, and I cannot profess competence on this.

Once you're running the code on your app, jonathan and janko's advice would be what you want.

[jrochkind]

Here is another way to tell a shrine model about an existing file on S3:

    book.cover_image_attacher.change(  book.cover_image_attacher.uploaded_file(store: :storage, id: "some_id/something.jpg") )

That one will do more standard lifecycle management -- not sure if that will include extracting metadata and various other hooks normally run on a "normal" upload, etc. I think this method is as similar as you can get to uploading a file, except for an existing file, with all (maybe?) of the normal stuff that would happen on uploading a file still happening.

I asked a similar question in a larger more confusing thread I started here: #579 (reply in thread). I wound up finding this answer myself, with some hints from others.

Curious what @janko thinks about this method vs book.cover_image_attacher.set(uploaded_file) for this use case!