How to post multipart form data?

[jakeonfire]

Trying to make a request that posts multipart form data including a file. The file comes from an external URL, and currently we're downloading the file to memory and then posting that though not sure that's the most efficient way.
Coming from faraday + multipart middeware, so i'm not too familiar with the details of a multipart request beyond having seen raw ones.
I saw this discussion: #141 but it doesn't give me enough clues to go on 😬

[ioquatix]

Here is an example of how to post the data directly: https://github.com/socketry/async-rest/blob/ae3d6da90d6fa8383304bfc66d334954e857dcb2/lib/async/rest/wrapper/form.rb#L27-L33

Regarding files, you'll need to construct a multipart body. Actually, there isn't a good mechanism for this but this might be a good start: https://github.com/socketry/multipart-post/blob/2b1235a25bc3ff03d907785caa83f09c69d6f501/lib/multipart/post/multipartable.rb

Doing it in-memory is okay if you have enough memory/swap to do so.

Beyond that, what we should probably introduce is Protocol::HTTP::Multipart which does streaming from disk/memory. multipart-post has most of the pieces but it was primarily designed for Net::HTTP.

[jakeonfire]

amazing. can lean on multipart-post quite a bit:

headers = {}
uri = URI("https://example.com/my_api")
data = URI("https://i.etsystatic.com/39938216/r/il/11dead/4579446101/il_fullxfull.4579446101_1irk.jpg").read
form_data = {
  file: ::Multipart::Post::UploadIO.new(StringIO.new(data), "image/jpg", "image.jpg")
}
multipart_post = Net::HTTP::Post::Multipart.new(uri.path, form_data, headers)
headers["content-type"] = multipart_post["content-type"]
headers["content-length"] = multipart_post["content-length"]
body = multipart_post.body_stream.read

task = Async do
  internet = Async::HTTP::Internet.instance
  internet.post(uri.to_s, headers, body)
end
response = task.wait

does that seem about right and optimal?

[jakeonfire]

should body always be an array?

[ioquatix]

Async::HTTP::Internet is more permissive on what it accepts as the body:

async-http/lib/async/http/internet.rb

Line 45 in 827a0a8

body = Body::Buffered.wrap(body)

In essence, it's okay to provide an Array of String chunks. Async::Client is less permissive and expects an instance of the a body class.

However, there are alternatives which can help you solve different problems, like streaming from disk, etc. You can pass an instance of those too, e.g. https://github.com/socketry/async-http/blob/main/lib/async/http/body/slowloris.rb is a body wrapper that simulates a slow upload/download.

[jakeonfire]

it's okay to provide an Array of String chunks

and i'm assuming it's okay to provide a string (to Async::HTTP::Internet).

do you see any advantages to doing what Net::HTTP::Post::Multipart is doing a bit more manually and providing string chunks? honestly uploading files is not a big part of what our falcon server is doing so i'm not too concerned about making it maximally optimal. but i would appreciate more clarity on what details affect concurrency and what areas i should be paying more attention to (like, is there a better time to call task.wait to handle the response?).

understandable if there are too many opaque/unspoken details, though. and thanks for all the guidance.

[ioquatix]

IMHO, if I was building a hypothetical Protocol::HTTP::Multipart, I'd provide wrappers for loading files from disk, and from memory, and then a container to put them all together. I'd probably avoid trying to chunk the entire thing instead breaking it into "multipart header", "file chunk1", "file chunk2", ...", "multipart header", "memory chunk1", "memory chunk2", "end of multipart" etc.

It's definitely optimal to chunk IO according to host page size and/or network MTU. It's also better to only load chunks from disk as they are being sent, as this would avoid using a lot of memory if you had to load the entire thing. However, if it's already in memory, there is very little point to write it out to disk first or try and behave identically to an IO.

In terms of chunking, Async::HTTP assumes you will break things into optimal chunks - i.e. if you provide an array of 1 chunk that's 1000MiB, it will try to write that all in one go. What that means, is that if you have HTTP/1.1 chunked encoding, it will try to write a 1000MiB chunk. HTTP/1.0 and 2 have different flow control mechanisms, so the data would be buffered and broken down into smaller pieces. So I wouldn't think about it too much and instead aim for page-aligned chunks. I tried to be a little helpful here by exposing this:

https://github.com/socketry/io-stream/blob/7adeff467e8d038a260b712e4d7c372fc2ff8fa3/lib/io/stream/generic.rb#L15-L19

I would recommend you use those constants for "streaming IO" chunk size, and in addition, I recommend you can experiment changing those values to see the impact on throughput. Certainly I've done that and it's very interesting to do a sweep from 1 byte -> 1MiB in small increments.

like, is there a better time to call task.wait to handle the response?

Basically, you should avoid waiting on things.

If your goal is to stream from one source to another, in theory you can do that but I'm not sure how effectively you can implement it with the multipart-post gem.

Basically, you should chain the data from the incoming file to the place you are trying to post it.

In simple cases you can literally write this:

::Protocol::HTTP::Middleware.build do |request|
  Async::HTTP::Internet.instance.post(..., request.body)

This will ensure that as soon as the client start sending the file, you will start streaming to the post request, i.e. the minimum TTFB possible. You have to be a little more mindful about error handling though. In theory we could do this with the hypothetical "Protocol::HTTP::Multipart" that doesn't exist yet... it would be pretty neat.

[jakeonfire]

@ioquatix how do you see falcon and async-http working together? if a request (to rails) is already being handled async by falcon, does making outgoing requests async via async-http (where the results of those requests need processing to respond to the outer (rails) request and thus we need to wait on the result before proceeding) add any efficiencies?