Support for IO objects of unknown size

[willcosgrove]

I'm trying to implement a streaming processor. Something that can take the given IO, and stream it to a program that will convert it, and stream the output to be uploaded back to Shrine. But Shrine expects all valid IO to respond to size which is something that can't be known ahead of time in this case.

Wrote an extension of the S3 storage that uses multipart uploads when the size of the IO is unknown, but I found out the size restraint is placed further up the chain than that. First in _enforce_io then in extract_size called indirectly by get_metadata. Size could potentially be required in more places, but I stopped my search here.

I have two questions about this:

1. Is this something that sounds like it should be possible to build within Shrine's plugin framework, or would this require a restructuring of Shrine's core to be able to support IOs with no known size?
2. If it would require a restructuring of Shrine's core, would you see supporting this use case as a reasonable goal of Shrine, or as outside of the scope of what you want Shrine to be?

[janko]

Thank you for bringing this to my attention!

First of all, streaming support is something that's very important to me, because it's crucial when dealing with large files. I tried to design Shrine so that it supports streaming well. For example, Shrine::UploadedFile objects allow you to stream uploaded files directly from the storage.

Until now I haven't thought of the use case of uploading a stream of unknown size to a storage. This definitely makes sense, and I believe that no parts of Shrine rely on size to be known other than the ones you've found.

Do you think you could send me a pull request which addresses the things you've mentioned (making #size method optional and patching the S3 storage)?