Managing a voluminous fileset -- HLS video set

[jrochkind]

So I have a use case I'm considering whether/how to use shrine for... let's dip my toe in the new Github Discussion forum, see what happens!

I will be creating HLS video -- a prepared HLS video consists of a manifest file ending in .m3u8, possibly some secondary .m3u8 manifests, and dozens (or more) of .ts segment files.

It's really only the primary manifest.m3u8 that my app needs to know about. I probably won't be uploading all the files through shrine/the app -- some external process will create the HLS files in a storage location.

So why might I want to use shrine at all?

• location/URL generation/management, in a mostly back-end-storage agnostic way
• Associate a produced HLS set with a given Rails model (ie database row) (yep, I'm using Rails/activerecord)
• Lifecycle management, where the HLS set will get deleted if the Rails model does -- or if a new HLS set is instead assigned.

I know there are ways to store multiple files in "derivatives" in an array/Hash-like structure. And indeed this HLS set could be considered the derivative of an existing file...

But for simplicity, what I'm thinking is creating an additional shrine attachment, and using it to store/reference only the primary .m3u8 file -- this is really all the app needs to refer to (although for lifecycle management... see below). It just seems like a simpler and more straightforward thing to implement and debug and maintain to me.

I think this will mostly work out, but there are two somewhat weird parts:

Can I assign an existing file to a shrine attachment?

An external process is going to create these files in a storage location. So I'd like to assign that location to my Rails model/shrine attachment, without doing an upload or anything, just assign the location attachment, like the hash with a storage and id key...

The file already exists inside a "storage" shrine knows about, I just need to save it's location on the model without triggering any upload process or other analysis.

It seems like this should be do-able, but I'm having trouble figuring out what way of doing it will play well with shrine.

Can I make delete/destroy delete the whole directory?

I'm thinking the attachment will only know about the primary manifest.m3u8 file, and this is ordinarily all the app needs to know about/refer to.

BUT, if shrine decides to delete the attachment for any reason... I want it to delete the contents of the whole directory that that referenced file is in.

Fortunately, at least some shrine adapters now have a delete_prefixed method that can be used to delete a directory, great. The ones I will be using do, great.

Is it totally insane to write a little local shrine "plugin" that will override deletion to trigger delete_prefixed on the prefix/directory the file is in, whenever a deletion happens? I guess by over-riding UploadedFile#delete? (Or maybe Attacher#destroy, but I think UploadedFile#delete is probably better, and the plugin system does seem to let us do it with the FileMethods module? )

Is this a reasonable idea?

In general, does this approach make sense? Is there a better way to do it?

[benkoshy]

I would be very interested to know what are the issues surrounding why a column cannot simply be saved to a new record?

new_record.image_data =   old_record.image_data

Is it totally insane to write a little local shrine "plugin" that will override deletion to trigger delete_prefixed on the prefix/directory the file is in, whenever a deletion happens?

Janko built Shrine for this very purpose: to allow for perfect flexibility in getting Shrine to do exactly what you want it to do.

I haven't looked too closely into it, but I would prefer Attacher#destroy overridden for a specific uploader. But this may be due to my incorrect understanding of the API that you want.

The only complication i can see is if two records are some how deleting the same folder?

I would be curious to see the finished result.

[jrochkind]

Hi, thanks for the reply @benkoshy!

If I just write to image_data, it's not immediately reflected in the shrine attachment....

new_record.image_data = { "id" => "something", "storage" => "store", "metadata" => {}}
new_record.image # => nil
new_record.save!
new_record.image # => nil, still!

Which just makes me nervous, and is inconvenient.

It looks like I can force this to sync with

widget.image_attacher.reload

But... if there already was an image, and I've changed image_data to point somewhere else... will the previous image be deleted? Looks like nope! That kind of lifecycle management is part of what I want.

In general this is just feeling like the wrong way to do it, and going behind shrine's back.

Here's an option that sort of seems to work:

new_record.image_attacher.set Shrine.uploaded_file(storage: :store, id: "whatever/here.jpg")
new_record.image #=> #<Shrine::UploadedFile storage=:store id="whatever/here.jpg" metadata={}>

The result is immediately apparent... but alas this one doesn't actually delete any previous attachment.

I really want life-cycle management along with being able to set to an existing file location. I haven't wrapped my head around what triggers the lifecycle management (deleting the replaced old file); I wonder if there's any way to do this.

[jrochkind]

OK! Looking at the source code (I've been poking in here since last week), THIS works well:

new_record.image_attacher.change(  new_record.image_attacher.uploaded_file(store: :storage, id: "some_id/something.jpg") )

It accepts the data I tell it, without re-uploading it or trying to. It DOES do the lifecycle management, deleting the old copy of the file. (You can assign a location that doesn't actually exist, and it won't complain on assignment -- which is expected).

So... I think that works?

As far as the next part about a plugin to delete a whole directory... I'm interested in why you're thinking you'd prefer overriding Attacher#destroy to UploadedFile#delete?

I'm not really sure, but what I'm thinking is... Attacher#destroy is written just to call out to to file delete.

shrine/lib/shrine/attacher.rb

Lines 210 to 212 in 7713f3b

	def destroy
	file&.delete
	end

But could there be other things that call file.delete without calling Attacher#destroy? And would then avoid my custom code? Which I don't want? It seems like there could be, if I override the lower-level thing I'm more assured of having my logic get in there.

On the other hand, as those experiments with creating UploadedFiles manually show... if I override teh specific ImageAttacher::UploadedFile#delete... in some cases I might not have the right UploadedFile subclass? Not sure.

[janko]

OK! Looking at the source code (I've been poking in here since last week), THIS works well:

new_record.image_attacher.change(  new_record.image_attacher.uploaded_file(store: :storage, id: "some_id/something.jpg") )

Yes, this is the way to do it 👍🏻. You can also use ImageUploader.uploaded_file.

As far as the next part about a plugin to delete a whole directory... I'm interested in why you're thinking you'd prefer overriding Attacher#destroy to UploadedFile#delete?

I'm not really sure, but what I'm thinking is... Attacher#destroy is written just to call out to to file delete.

I would generally put higher-level logic in attacher, and keep UploadedFile having lower-level logic. One advantage of being on the attacher level is that you have access to the record, which might be useful for determining which directory to delete. But in your case, I think UploadedFile#delete would be fine too, given that you're not storing other files in the attachment column.