Detach response stream in Promise mode

[jordanbtucker]

What problem are you trying to solve?

Got stores the body of the response in the body property. This is great if you're working with relatively small payloads, like text, but it's not great for large files because it eats up your memory.

const res = await got('http://example.com/very-large-file.tar.gz', {encoding: null})
console.log(res.headers)
console.log(res.body.byteLength) // 1073741824 (1GB loaded in memory)
await fs.promises.writeFile('file.tar.gz', res.body)

Of course, you can use got.stream, but then you just get a stream instead of the response object. To get the response, you can listen to the response event, but then you've moved from promises to callbacks.

const stream = got.stream('http://example.com/very-large-file.tar.gz')

// Yuck
const res = await new Promise(resolve => {
  stream.on('response', res => { resolve(res) })
}

console.log(res.headers)
await pipeline(stream, fs.createWriteStream('file.tar.gz'))

Describe the feature

Having a feature that resolves with the response, but still lets you process the body as a stream would be great. Since IncomingMessage is a stream, maybe got.stream can just return the response instead of just a stream.

Checklist

• I have read the documentation and am fairly sure this feature doesn't already exist.

[szmarczak]

Duplicate of #771

[jordanbtucker]

This isn't really a duplicate of #771 because that issue just wanted the headers. I do want the body, I just want to pipe it from the request like http/https does it.

I modified the workaround given in #771, but that gives me an empty body when I try to pipe the response to something. Plus it's even uglier than the workaround in my initial comment.

const getResponse = () =>
  new Promise((resolve, reject) => {
    const promise = got('http://example.com').on('response', response => {
      resolve(response)
      promise.cancel()
    })

    promise.then(response => resolve(response)).catch(reject)
  })

;(async () => {
  const res = await getResponse()
  res.pipe(fs.createWriteStream('got-test.html'))
  // got-test.html is empty 😕
})()

In short, I want this:

const res = await got('http://example.com', {parseBody: false})
res.pipe(fs.createWriteStream('got-test.html'))

If you don't think this is a valuable feature, that is fine, but it is different than just wanting to grab the headers.

[szmarczak]

Hmm... As I see it you want to combine Promise and Stream mode. Currently, that's not possible. So either you need to do:

const combined = async (...args) => {
    const stream = got.stream(...args);
    const response = await pEvent(stream, 'response');
    return {stream, response};
};

or

got('https://example.com').on('response', response => {
    response.pipe(...); // Pipe first, then silently reject further pipes
    response.on('newListener', (event, listener) => {
        if (event === 'data') {
            response.removeListener(event, listener);
        }
    });
});

[jordanbtucker]

Thanks for the work arounds. This was meant as a feature request, so if you're not willing to discuss this as a new feature, then you can keep it closed, but I was hoping for more of a discussion.

[szmarczak]

Also it won't work if the response is cached, beware of that. So you need to disable cache. The stream approach seems to be the best option.

[szmarczak]

Closing because lack of activity. This is not an actual problem.