Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TypeError: fetch failed #1248

Closed
kyrylkov opened this issue Feb 23, 2022 · 83 comments
Closed

TypeError: fetch failed #1248

kyrylkov opened this issue Feb 23, 2022 · 83 comments
Labels
bug Something isn't working

Comments

@kyrylkov
Copy link
Contributor

kyrylkov commented Feb 23, 2022

Bug Description

Fetch throws 'TypeError: fetch failed' when posting

Reproducible By

Request with private production credentials to ERP

Expected Behavior

Successful response (ok). node-fetch works just fine.

Logs & Screenshots

 TypeError: fetch failed
    at Object.processResponse (C:\Users\kyryl\dev\entriwise\server\node_modules\undici\lib\fetch\index.js:183:23)
    at Fetch.fetchFinale (C:\Users\kyryl\dev\entriwise\server\node_modules\undici\lib\fetch\index.js:937:17)
    at Fetch.mainFetch (C:\Users\kyryl\dev\entriwise\server\node_modules\undici\lib\fetch\index.js:744:17)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

Environment

Linux and Windows, v17.5 v17.6

Additional context

Request takes up to several minutes until response

@kyrylkov kyrylkov added the bug Something isn't working label Feb 23, 2022
@mcollina
Copy link
Member

Thanks for reporting, would you like to include a code to reproduce your problem? We'd also need the server counterpart.

@kyrylkov
Copy link
Contributor Author

@mcollina We cannot share production environment credentials, but we could give you sandbox credentials via remote desktop. Is there any other way to give you more info for debugging?

@mcollina
Copy link
Member

I think it would be better if you produced a server that returned that kind of response. We do not provide support for private code.

@kyrylkov
Copy link
Contributor Author

kyrylkov commented Feb 24, 2022

Matteo, the server is NetSuite ERP, so we cannot provide it. Also note we've replaced undici fetch with node-fetch and it's working fine. So this error is not the server response.

We can provide you with isolated fetch request, no code required. But we cannot share the credentials to run this request.

@mcollina
Copy link
Member

There is something in that server response that it is making undici.fetch fail. What I'm asking is a way to reproduce this without touching your systems. Only you can write it.

@kyrylkov
Copy link
Contributor Author

kyrylkov commented Feb 24, 2022

Right but I don't understand how? The only way we can reproduce it ourselves is by executing that request to that account with those credentials.

@Linkgoron
Copy link
Member

There's supposed to be an additional cause property on that error, but I don't see it in the log. Does it exist?

@kyrylkov
Copy link
Contributor Author

@Linkgoron No. The code is like this. But 2nd line is never reached.

const fetchRes = await fetch(webservicesUrl, options);
if (!fetchRes.ok) throw new Error(`HTTP error! Status: ${fetchRes.status}`);

@ronag
Copy link
Member

ronag commented Feb 24, 2022

Sorry but with this little information it's impossible for us to do anything.

@kyrylkov
Copy link
Contributor Author

@ronag So I'm trying to get you more information. Can we add some debugging code in undici to help you diagnose it?

@janat08
Copy link

janat08 commented Feb 24, 2022

I also get some error, it's not the same:

"TypeError: fetch failed\n    at Object.processResponse (C:\\Users\\jeyan\\Documents\\boughtwithslowsearch\\node_modules\\undici\\lib\\fetch\\index.js:183:23)\n    at Fetch.fetchFinale (C:\\Users\\jeyan\\Documents\\boughtwithslowsearch\\node_modules\\undici\\lib\\fetch\\index.js:937:17)\n    at Fetch.mainFetch (C:\\Users\\jeyan\\Documents\\boughtwithslowsearch\\node_modules\\undici\\lib\\fetch\\index.js:744:17)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)"

@akauppi
Copy link

akauppi commented Feb 27, 2022

Maybe this can help?

I am experiencing the same with a Firebase Emulators host.

$ git clone https://github.com/akauppi/firebase-jest-testing.git
$ cd firebase-jest-testing

$ git checkout 0d2a5f43022453a34a3de9510f9e436b4d890078    # reproducible in this commit

$ npm install

In one terminal:

$ npm run start

In another:

$ npm run test:fns:greet
...
(node:37466) ExperimentalWarning: VM Modules is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
 FAIL  sample/test-fns/greet.test.js
  Cloud Function callables
    ✕ returns a greeting (15 ms)

  ● Cloud Function callables › returns a greeting

    TypeError: fetch failed

      at Object.processResponse (../../node_modules/undici/lib/fetch/index.js:183:23)

Test Suites: 1 failed, 1 total
Tests:       1 failed, 1 total
Snapshots:   0 total
Time:        0.324 s, estimated 1 s
Ran all test suites matching /greet.test.js/i.

Added: Once npm run start is running, you can exercise the server by:

$ curl -v -X POST -d '{"data":"abc"}' -H "Content-type: application/json" http://localhost:5002/demo-1/mars-cent
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying ::1:5002...
* connect to ::1 port 5002 failed: Connection refused
*   Trying 127.0.0.1:5002...
* Connected to localhost (127.0.0.1) port 5002 (#0)
> POST /demo-1/mars-central2/greet HTTP/1.1
> Host: localhost:5002
> User-Agent: curl/7.77.0
> Accept: */*
> Content-type: application/json
> Content-Length: 14
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< x-powered-by: Express
< vary: Origin
< content-type: application/json; charset=utf-8
< content-length: 28
< etag: W/"1c-90pTE8vnJrTHpo49hh2KVRcGoxw"
< date: Sun, 27 Feb 2022 12:33:58 GMT
< connection: close
< 
* Closing connection 0
{"result":"Greetings, abc."} 

Trying with 127.0.0.1 instead of localhost puts the problem at rest:

$ EMUL_HOST=127.0.0.1 npm run test:fns:greet

> test:fns:greet
> NODE_OPTIONS=--experimental-vm-modules jest --config sample/test-fns/jest.config.js -f greet.test.js --verbose --all

(node:38224) ExperimentalWarning: VM Modules is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:38224) ExperimentalWarning: buffer.Blob is an experimental feature. This feature could change at any time
 PASS  sample/test-fns/greet.test.js
  Cloud Function callables
    ✓ returns a greeting (33 ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        0.32 s, estimated 1 s
Ran all test suites matching /greet.test.js/i.

@akauppi
Copy link

akauppi commented Feb 27, 2022

As a continuation to my above report, from a CI run.

It looks like Undici (4.14.1) fetch may have a general problem with host names. Here, emul is supposed to point to a Docker container under Google Cloud Build. The run fails to find it.

Step #2: > ci:test
Step #2: > wait-for emul:6767 && wait-for emul:5002 && npm run -s _ci_warm-up && npm run -s _test1
Step #2: 
Step #2: Unexpected error: TypeError: fetch failed
Step #2:     at Object.processResponse (/workspace/node_modules/undici/lib/fetch/index.js:183:23)
Step #2:     at Fetch.fetchFinale (/workspace/node_modules/undici/lib/fetch/index.js:937:17)
Step #2:     at Fetch.mainFetch (/workspace/node_modules/undici/lib/fetch/index.js:744:17)
Step #2:     at processTicksAndRejections (node:internal/process/task_queues:96:5) {
Step #2:   cause: Error: getaddrinfo ENOTFOUND emul
Step #2:       at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26) {
Step #2:     errno: -3008,
Step #2:     code: 'ENOTFOUND',
Step #2:     syscall: 'getaddrinfo',
Step #2:     hostname: 'emul'
Step #2:   }

@ronag
Copy link
Member

ronag commented Feb 27, 2022

I don't think this has anything to do with undici per se. Does it work with the normal http client in node?

@akauppi
Copy link

akauppi commented Feb 28, 2022

I don't think this has anything to do with undici per se. Does it work with the normal http client in node?

It works with curl - I am not very versed in using the normal http client, so.. don't know.

Got some more light to why localhost isn't recognized by fetch.

My /etc/hosts (macOS 12.2.1 Intel):

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1       localhost
255.255.255.255 broadcasthost
::1             localhost
# Added by Docker Desktop
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
# End of section

If I comment out the line ::1 localhost, things work as normal.

@ronag
Copy link
Member

ronag commented Feb 28, 2022

@akauppi: Sorry but I don't think this is an issue we can resolve in undici. Sounds more like a node core dns thing. I suggest you open an issue in the node repo.

@ronag
Copy link
Member

ronag commented Mar 19, 2022

@akauppi: As i wrote above undici is probably not the place for the fix.
@kyrylkov: If you want help with private code you can mail me.

@ronag ronag closed this as completed Mar 19, 2022
@kyrylkov
Copy link
Contributor Author

@ronag emailed you

@kyrylkov
Copy link
Contributor Author

kyrylkov commented Apr 26, 2022

@ronag I'm still having these errors. Cannot use undici fetch at all. Here is updated trace from Node v18:

TypeError: fetch failed
    at Object.processResponse (/home/ubuntu/server/node_modules/undici/lib/fetch/index.js:200:23)
    at /home/ubuntu/server/node_modules/undici/lib/fetch/index.js:941:38
    at node:internal/process/task_queues:140:7
    at AsyncResource.runInAsyncScope (node:async_hooks:202:9)
    at AsyncResource.runMicrotask (node:internal/process/task_queues:137:8)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    ... 2 lines matching cause stack trace ...
    at process.processTimers (node:internal/timers:507:7) {
  cause: Error: [object Object]
      at makeNetworkError (/home/ubuntu/server/node_modules/undici/lib/fetch/response.js:366:11)
      at httpNetworkFetch (/home/ubuntu/server/node_modules/undici/lib/fetch/index.js:1751:12)
      at processTicksAndRejections (node:internal/process/task_queues:95:5)
      at runNextTicks (node:internal/process/task_queues:64:3)
      at listOnTimeout (node:internal/timers:533:9)
      at process.processTimers (node:internal/timers:507:7)
      at async httpNetworkOrCacheFetch (/home/ubuntu/server/node_modules/undici/lib/fetch/index.js:1440:29)
      at async httpFetch (/home/ubuntu/server/node_modules/undici/lib/fetch/index.js:1028:33)
      at async schemeFetch (/home/ubuntu/server/node_modules/undici/lib/fetch/index.js:888:14)
      at async /home/ubuntu/server/node_modules/undici/lib/fetch/index.js:569:16
      at async mainFetch (/home/ubuntu/server/node_modules/undici/lib/fetch/index.js:553:16) {
    [cause]: undefined
  }
}

@ronag Could you re-open or reply to my emails?

@s-bilic
Copy link

s-bilic commented Jul 19, 2022

Had the same issue, had to switch from node v18 to v16 as a temp workaround.

@ItzaMi
Copy link

ItzaMi commented Aug 15, 2022

Had the same issue, had to switch from node v18 to v16 as a temp workaround.

This worked for me. Also was having the same issue without any resolution and downgrading to 16 ended up doing the trick!

@mcollina mcollina reopened this Aug 15, 2022
@kyrylkov
Copy link
Contributor Author

We also keep getting this error from time to time in Node.js 18.7.0

@mcollina
Copy link
Member

I think I know what the problem is.

In node v17 we switched the default ordering of dns entries to follow what the OS is providing us vs reordering to put ipv4 always first.

I think we should try multiple addresses when dealing with localhost when establishing a new connection.

Wdyt @ronag?

@ronag
Copy link
Member

ronag commented Aug 15, 2022

I think I know what the problem is.

In node v17 we switched the default ordering of dns entries to follow what the OS is providing us vs reordering to put ipv4 always first.

I think we should try multiple addresses when dealing with localhost when establishing a new connection.

Wdyt @ronag?

SGTM!

@sangdth
Copy link

sangdth commented Jul 8, 2023

I removed the line ::1 localhost like this comment, things work again in my Nextjs case.

@kyrylkov
Copy link
Contributor Author

kyrylkov commented Jul 8, 2023

I removed the line ::1 localhost like this comment, things work again in my Nextjs case.

It's not related to localhost in many cases.

@BSN4
Copy link

BSN4 commented Jul 8, 2023

Im not sure why but seems the team decided to ignore this bug ?

Any news about this issue

@kyrylkov
Copy link
Contributor Author

kyrylkov commented Jul 8, 2023

Im not sure why but seems the team decided to ignore this bug ?

No one has provide a repro for them to look into

@BSN4
Copy link

BSN4 commented Jul 13, 2023

@kyrylkov ok tested this on v20.4.0 I think its fixed for now can't say for sure

@kyrylkov
Copy link
Contributor Author

ok tested this on v20.4.0 I think its fixed for now can't say for sure

We have this issue with v20.4.0 also

@jpcmf
Copy link

jpcmf commented Jul 16, 2023

Я сталкиваюсь с той же проблемой с Next.js и Strapi. На localhost Next.js приложение не может получить API из Strapi. Недавно я обновил Node.js до версии 18.15.0.

error - TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11413:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async fetchQuery (webpack-internal:///./src/utils/fetch.ts:18:22)
    at async Promise.all (index 0)
    at async getServerSideProps (webpack-internal:///./src/pages/index.tsx:47:20) {
  cause: Error: connect ECONNREFUSED ::1:1337
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16) {
    errno: -4078,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '::1',
    port: 1337
  },
  page: '/'
}

I implement authorization with strapi and next. I had a fetch error. The only thing that helped me was this combination of environment variables: NEXTAUTH_URL=http:localhost:3000 NEXT_PUBLIC_API_URL=http://127.0.0.1:1337

https://strapi.io/blog/user-authentication-in-next-js-with-strapi

I did the same. I replaced "localhost" with "127.0.0.1" 👎🏻

@gigantino
Copy link

Having the same issue, downgrading to 18.16.0 solved it.

@pauldraper
Copy link

pauldraper commented Jul 31, 2023

The problem with the debugging is that error.stack does not include the cause.

new Error("", { cause: new Error() }).stack
Error
    at [eval]:1:13
    at Script.runInThisContext (node:vm:128:12)
    at Object.runInThisContext (node:vm:306:38)
    at node:internal/process/execution:83:21
    at [eval]-wrapper:6:24
    at runScript (node:internal/process/execution:82:62)
    at evalScript (node:internal/process/execution:104:10)
    at node:internal/main/eval_string:50:3

Whereas

util.inspect(new Error("", { cause: new Error() }))
Error
    at [eval]:1:37
    at Script.runInThisContext (node:vm:128:12)
    ... 5 lines matching cause stack trace ...
    at node:internal/main/eval_string:50:3 {
  [cause]: Error
      at [eval]:1:60
      at Script.runInThisContext (node:vm:128:12)
      at Object.runInThisContext (node:vm:306:38)
      at node:internal/process/execution:83:21
      at [eval]-wrapper:6:24
      at runScript (node:internal/process/execution:82:62)
      at evalScript (node:internal/process/execution:104:10)
      at node:internal/main/eval_string:50:3

That cause will show you the real problem with the fetch call. E.g. if you try to send an expect header, or an invalid connection header, or a million other possible problems.

You'll need to fix the logging to include the cause, not just the stack.

@kyrylkov
Copy link
Contributor Author

kyrylkov commented Aug 1, 2023

@pauldraper You're right about the cause. But disregarding whether logging includes the cause or not, there is clearly something wrong with various versions of undici / Node.js. Because with some combinations of undici / Node.js versions TypeError: fetch failed doesn't happen and with some it happens consistently often, whatever the underlying cause is.

@pauldraper
Copy link

pauldraper commented Aug 1, 2023

whatever the underlying cause is

Well, that's actually pretty significant.

I suggest posting the cause.

@kyrylkov
Copy link
Contributor Author

kyrylkov commented Aug 1, 2023

It is significant, but even more significant is the repro, which no one could produce so far because the issue is intermittent and not easily reproducible. In our case, for example, when it fails, it fails in like 1 request out of 1000. And as soon as we log the request and retry it, it works just fine without producing the error.

@pauldraper
Copy link

It could be, for example, a network failure. No way to know here.

@kyrylkov
Copy link
Contributor Author

kyrylkov commented Aug 1, 2023

It can be, but it is not. Why? Because it happens only with a certain combination of Node / undici. And it doesn't happen at all with other combinations. Thus it is a Node / undici issue. And we are talking about sampling periods of weeks here.

@pauldraper
Copy link

This happened to me a few hours ago. I logged the cause.

cause: HeadersTimeoutError: Headers Timeout Error

@gigantino
Copy link

gigantino commented Aug 1, 2023

It could be, for example, a network failure. No way to know here.

I don't think it's a network failure. On SvelteKit (which is where I got this error), once it happens once it keeps on happening multiple times (even if I restart node) and then just randomly decides to start working again. I also thought it could have been a network issue but everything else worked perfectly...

edit: typo

@ronag
Copy link
Member

ronag commented Aug 2, 2023

This happened to me a few hours ago. I logged the cause.

That happens when the server hasn't replied with anything in a minute or so.

@ronag
Copy link
Member

ronag commented Aug 2, 2023

I think some of these issue might be related to keep-alive + server or client bugs. Try to disable keep-alive and see if the issue persists.

@kyrylkov
Copy link
Contributor Author

kyrylkov commented Aug 2, 2023

This happened to me a few hours ago. I logged the cause.

That happens when the server hasn't replied with anything in a minute or so.

For unidici fetch, at one time, HeadersTimeoutError was changed to 5 minutes. 5 minute header timeout is very unlikely. Unless this has been reverted to 1 minute again, but it doesn't look to be the case.

@gazhay
Copy link

gazhay commented Aug 2, 2023

Fwiw my report wasn't timeout related as the error was instantaneous in all cases.

@arun1940
Copy link

arun1940 commented Aug 7, 2023

I am facing an issue while sending a GET request with body data. The error disappears when I remove the body data. But when I send a POST request with a body, the error doesn't occur. My current version of Next.js is 13.4, and I suspect that the error is caused by trying to send a GET request with body data.

@kyrylkov
Copy link
Contributor Author

kyrylkov commented Aug 7, 2023

@arun1940 We had it both with GET without body and with many POSTs.

@nxnxu
Copy link

nxnxu commented Aug 7, 2023

@arun1940

I am facing an issue while sending a GET request with body data. The error disappears when I remove the body data. But when I send a POST request with a body, the error doesn't occur. My current version of Next.js is 13.4, and I suspect that the error is caused by trying to send a GET request with body data.

From the [HTTP 1.1 2014 Spec](https://www.rfc-editor.org/rfc/rfc7231#page-24):

    A payload within a GET request message has no defined semantics; sending a payload body on a GET request might cause some existing implementations to reject the request.

Try it check with node-fetch, import this

import fetch from 'node-fetch';

@arun1940
Copy link

arun1940 commented Aug 7, 2023

I managed to resolve my issue by removing the data from the body of the GET request. I was using Postman to test my API and my application is built on Next.js version 13.4. The error that was occurring has been resolved.- error Failed to handle a request for /api/xxx-xx
node version:18.16.1
My Error Was:-
TypeError: fetch failed
at Object.fetch (node:internal/deps/undici/undici:11457:11)
at async invokeRequest (/home//nextjs/auth-app/node_modules/next/dist/server/lib/server-ipc/invoke-request.js:21:12)
at async requestHandler (/home/xxxx/nextjs/auth-app/node_modules/next/dist/server/lib/start-server.js:336:33)
at async Server. (/home/xxxx/nextjs/auth-app/node_modules/next/dist/server/lib/start-server.js:152:13) {
cause: RequestContentLengthMismatchError: Request body length does not match content-length header
at write (node:internal/deps/undici/undici:9949:41)
at _resume (node:internal/deps/undici/undici:9927:33)
at resume (node:internal/deps/undici/undici:9829:7)
at connect (node:internal/deps/undici/undici:9818:7) {
code: 'UND_ERR_REQ_CONTENT_LENGTH_MISMATCH'
}
}

@0oj
Copy link

0oj commented Aug 12, 2023

I don't think it's a network failure. On SvelteKit (which is where I got this error), once it happens once it keeps on happening multiple times (even if I restart node) and then just randomly decides to start working again. I also thought it could have been a network issue but everything else worked perfectly...

@gigantino We're you able to resolve this within SvelteKit? I'm facing the same issue trying to access a certain LinkedIn endpoint (others work fine). Exact same behaviour. Randomly works sometimes when I'm not looking.

@gigantino
Copy link

@gigantino We're you able to resolve this within SvelteKit? I'm facing the same issue trying to access a certain LinkedIn endpoint (others work fine). Exact same behaviour. Randomly works sometimes when I'm not looking.

@0oj I did by switching to Node 18.16.0

@HermesKD123
Copy link

Hello, anyone got the fix to this error?

@BeanKorea
Copy link

This issue is happening in @microsoft/signalr

@ronag
Copy link
Member

ronag commented Aug 23, 2023

This is a generic error that can occur due to several different reasons. Please check the cause property on the error.

@nodejs nodejs locked as resolved and limited conversation to collaborators Aug 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests