Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fetch times out in under 5 seconds #1531

Closed
crackedpotato007 opened this issue Jul 7, 2022 · 30 comments · Fixed by #1914
Closed

fetch times out in under 5 seconds #1531

crackedpotato007 opened this issue Jul 7, 2022 · 30 comments · Fixed by #1914
Labels
bug Something isn't working

Comments

@crackedpotato007
Copy link

Bug Description

When trying to fetch a URL a fetch failed error is thrown with the code as
code: 'UND_ERR_CONNECT_TIMEOUT'
, This error is thrown at a request that barely takes 5 - 6 seconds to complete and other HTTP clients like axios and curl perform flawlessly on the same server

Reproducible By

Fetch any discord API url

Expected Behavior

The fetch should complete which is well below the 120s timeout

Logs & Screenshots

/home/arnav/Documents/tej.js/node_modules/undici/lib/fetch/index.js:197
        Object.assign(new TypeError('fetch failed'), { cause: response.error })
                      ^
TypeError: fetch failed
    at Object.processResponse (/home/arnav/Documents/tej.js/node_modules/undici/lib/fetch/index.js:197:23)
    at /home/arnav/Documents/tej.js/node_modules/undici/lib/fetch/index.js:930:38
    at node:internal/process/task_queues:141:7
    at AsyncResource.runInAsyncScope (node:async_hooks:201:9)
    at AsyncResource.runMicrotask (node:internal/process/task_queues:138:8)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  cause: ConnectTimeoutError: Connect Timeout Error
      at Timeout.onConnectTimeout [as _onTimeout] (/home/arnav/Documents/tej.js/node_modules/undici/lib/core/connect.js:108:24)
      at listOnTimeout (node:internal/timers:561:11)
      at processTimers (node:internal/timers:502:7) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}

Environment

Gentoo, Node v17.6.0

Additional context

Happen on every discord API url and used to work some time back but suddenly just fails everywhere.

@crackedpotato007 crackedpotato007 added the bug Something isn't working label Jul 7, 2022
@mcollina
Copy link
Member

mcollina commented Jul 7, 2022

Thanks for reporting!

Can you provide steps to reproduce? We often need a reproducible example, e.g. some code that allows someone else to recreate your problem by just copying and pasting it. If it involves more than a couple of different file, create a new repository on GitHub and add a link to that.

@crackedpotato007
Copy link
Author

Hey!
Here you go
a valid auth token can be generated by https://discord.com/developers

 const myuser = await fetch("https://discord.com/api/v10/users/@me", {
      headers: {
        authorization: `Bot xxxxx`,
        "User-Agent": "undici/tej.js",
        encoding: "json",
      },
    }).catch((err) => {
      throw new Error(err);
    });

@crackedpotato007
Copy link
Author

Undici fails if your DNS is set to your router which in turn uses google DNS but if i sed the google DNS directly on my machine it would work, this shouldn't be happening i believe.

@mcollina
Copy link
Member

mcollina commented Jul 7, 2022

Can you create a reproduction without using an external server? Something that we can run locally.

@crackedpotato007
Copy link
Author

I can give that a try but I believe that would just run as I believe that undici isn't able to resolve the webpage to an ip address which explains how a direct dns configuration resolved the issue.

@crackedpotato007
Copy link
Author

Can you try to replicate it with a external server and the dns set to your router?
For me it was configured as

search bbrouter
nameserver 192.168.1.1

@mcollina
Copy link
Member

mcollina commented Jul 8, 2022

I have a very similar setup, and there is no problem in contacting the discord API.

Can you try running your script with NODE_DEBUG=net and paste the output? Both when using undici and when using axios.

@ronag
Copy link
Member

ronag commented Jul 8, 2022

Are you sure it's 5 seconds? Our default connect timeout is 10s? I suspect your dns lookup or ssl negotiation is taking more than 10s or something...

@crackedpotato007
Copy link
Author

A direct DNS to 8.8.8.8 works flawlessly on undici but a DNS pointing to the router fails. I will get you the timestamp in a few hours

@ronag
Copy link
Member

ronag commented Jul 8, 2022

I don't think this is a undici problem. This works for axios probably because it has a longer (or no) timeout for establishing the connection.

@mcollina
Copy link
Member

mcollina commented Jul 8, 2022

The timeout should be configurable with:

import { fetch, setGlobalDispatcher, Agent } from 'undici'

setGlobalDispatcher(new Agent({ connect: { timeout: 60_000 } }) )

fetch(...)

@polyrainbow
Copy link

I am also getting UND_ERR_CONNECT_TIMEOUT after ~5 secs when IPv6 is not configured or working properly. With curl, Chrome and other clients, it still works via IPv4. This seems to be because Node does not implement Happy Eyeballs. There is already an issue for this: nodejs/node#41625

@EduApps-CDG
Copy link

EduApps-CDG commented Sep 11, 2022

I have a very similar setup, and there is no problem in contacting the discord API.

Can you try running your script with NODE_DEBUG=net and paste the output? Both when using undici and when using axios.
@arnav7633

I'm having the same issue. I'm using discord.js@14.3.0 which uses undici.

NET 56348: pipe false undefined
NET 56348: connect: find host discord.com
NET 56348: connect: dns options { family: undefined, hints: 32 }
NET 56348: _read
NET 56348: _read wait for connection
NET 56348: destroy
NET 56348: close
NET 56348: close handle
ConnectTimeoutError: Connect Timeout Error
    at onConnectTimeout (/home/eduardo/proj/DescicloBot/bot_stable/node_modules/undici/lib/core/connect.js:131:24)
    at /home/eduardo/proj/DescicloBot/bot_stable/node_modules/undici/lib/core/connect.js:78:46
    at Immediate._onImmediate (/home/eduardo/proj/DescicloBot/bot_stable/node_modules/undici/lib/core/connect.js:119:9)
    at processImmediate (node:internal/timers:471:21) {
  code: 'UND_ERR_CONNECT_TIMEOUT'
}
NET 56348: emit close

Is family: undefined correct?

@0xZoomEye
Copy link

Screenshot from 2022-10-08 18-30-29
I encounter the same problem the first time i use it.

@ronag
Copy link
Member

ronag commented Oct 8, 2022

@mcollina maybe we should consider increasing the default timeout?

@mcollina
Copy link
Member

mcollina commented Oct 8, 2022

That's not the problem. The problem is that all those folks have IPv6 misconfigured and we are missing Happy Eyeballs in Node.js nodejs/node#41625. Hopefully it will land before Node.js v18 goes LTS or shortly thereafter.

@cassitly
Copy link

cassitly commented Oct 23, 2022

ive gotten into this issues as well, i dont know which lines of code did its so heres all of it
const { Client, Intents, DiscordAPIError, Collection, GatewayIntentBits, REST, Routes } = require('discord.js');
const client = new Client({ intents: [GatewayIntentBits.Guilds] });
require("dotenv") .config();
const CLIENT_ID = 'your client id'
const http = require('http')

const fs = require('node:fs');

const axios = require('axios');

const https = require('https');
const res = require('res');

const data = JSON.stringify({
todo: 'Buy the milk',
});

const options = {
hostname: 'localhost',
port: 443,
path: '/app',
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Content-Length': data.length,
},
};

const error1 = process.env.INVALID_REQUESTED_URL
const title1 = process.env.INVALID_REQUESTED_URL_TITLE

const express = require('express');

const mongoose = require('mongoose');
const { linkSync, writeFile } = require('fs');

const alert = process.env.COTENT_DOWN || "Sorry But the content on this page for this web server is currently down, please try again later"
const al1 = process.env.ERROR || "404 Page Not Found"
const dashboard = process.env.dash || "Sorry But The DashBoard Is Still In Development"

const port = process.env.PORT || 443

const prefix = '/'

// then you may define it like so
client.event = new Collection();
client.commands = new Collection();

const commands = [
{
name: 'ping',
description: 'Replies with Pong!',
},
{
name: 'test',
description: 'Test Running OK',
},
{
name: 'channel',
description: 'Displays the Owners Channel',
},
{
name: 'reload',
description: 'Reload the bot datasets',
},
{
name: 'latest',
description: 'Display the latest video of the owners channel',
},
];

const rest = new REST({ version: '11' }).setToken('your token');

(async () => {
try {
console.log('Started refreshing application (/) commands.');

await rest.put(Routes.applicationCommands(CLIENT_ID), { body: commands });

console.log('Successfully reloaded application (/) commands.');

} catch (error) {
console.error(error);
}
})();

client.on('ready', () => {
console.log(Logged in as ${client.user.tag}!);
});

client.on('interactionCreate', async interaction => {
if (!interaction.isChatInputCommand()) return;

if (interaction.commandName == 'ping') {
await interaction.reply('Pong!');
} else if (interaction.commandName === 'test') {
await interaction.reply('Test Running OK');
} else if (interaction.commandName === 'channel') {
await interaction.reply('Channel not vaild yet');
} else if (interaction.commandName === 'reload') {
await interaction.reply('Reloading Dataset');
} else if (interaction.commandName === 'latest') {
await interaction.reply('Theres No Latest Video In this channel yet');
}
});

const server = http.createServer((req, res) => {
if (req.url == '/test') {
res.write("test");
res.end();
} else if (req.url == "/app") {
res.write(fs.readFileSync("/Users/jedik/Desktop/Discord.js Bot/Beta Solaris Bot/index.html"));
res.end();
} else if (req.url == "/app/service/etc/host/js/script/main/password=197719777") {
res.write(fs.readFileSync("/Users/jedik/Desktop/Discord.js Bot/Beta Solaris Bot/assets/js/main.js"));
res.end();
} else if (req.url == "/app/service/etc/host/js/script/jquery.min/password=1977719777") {
res.write(fs.readFileSync("/Users/jedik/Desktop/Discord.js Bot/Beta Solaris Bot/assets/js/jquery.min.js"));
res.end();
} else if (req.url == "/app/service/etc/host/js/script/jquery.poptrox.min/password=19771977") {
res.write(fs.readFileSync("/Users/jedik/Desktop/Discord.js Bot/Beta Solaris Bot/assets/js/jquery.poptrox.min.js"));
res.end();
} else if (req.url == "/web/server/main/github") {
res.write(fs.readFileSync("/Users/jedik/Desktop/Discord.js Bot/Beta Solaris Bot/index.html"));
res.end();
} else if (req.url == "/solarishost") {
res.write(fs.readFileSync("/Users/jedik/Desktop/Discord.js Bot/Beta Solaris Bot/index.html"));
res.write(fs.readFileSync("/Users/jedik/Desktop/Discord.js Bot/Beta Solaris Bot/assets/css/main.css"));
res.write(fs.readFileSync("/Users/jedik/Desktop/Discord.js Bot/Beta Solaris Bot/assets/js/main.js"));
res.write(fs.readFileSync("/Users/jedik/Desktop/Discord.js Bot/Beta Solaris Bot/assets/js/jquery.min.js"));
res.write(fs.readFileSync("/Users/jedik/Desktop/Discord.js Bot/Beta Solaris Bot/assets/js/jquery.poptrox.min.js"));
res.write(fs.readFileSync("/Users/jedik/Desktop/Discord.js Bot/Beta Solaris Bot/assets/js/jquery.scrolly.min.js"));
res.write(fs.readFileSync("/Users/jedik/Desktop/Discord.js Bot/Beta Solaris Bot/assets/js/skel.min.js"));
res.write(fs.readFileSync("/Users/jedik/Desktop/Discord.js Bot/Beta Solaris Bot/assets/js/util.js"));
res.end();
} else if (req.url == "/solarishost/contact") {
res.write(fs.readFileSync("/Users/jedik/Desktop/Discord.js Bot/Beta Solaris Bot/contact.html"));
res.end();
} else if (req.url == "/service/content/assets/js/access/pass/granted/content/load/developer-pass/main.js") {
res.write("{content:122, service:down, protection:down}")
} else if (req.url == '/service/password/1977') {
res.write("Sorry but this service content code is down");
res.end();
} else if (req.url == "/etc/server/env/password/19777") {
res.write("Welcome ");
res.write("Great Job you have entered the correct password into the url bar we're now loading the content on this page of the website")
res.write(" Opps It seems like the content on this page has been moved")
res.end();
} else if (req.url == "/service/password") {
res.write("We're sorry you do not have access to this content of the webserver");
res.end();
} else if (req.url == "/service") {
res.write("We're Sorry you do not have permission to access this content on this webserver");
res.end();
} else if (req.url == "/etc/server/env") {
res.write("We're Sorry but you do not have access to this content on this web server");
res.end();
} else if (req.url == "/etc/server/env/password") {
res.write("Please enter the correct password in the url bar ")
res.write("Waiting...")
res.end();
} else if (req.url == "/assets/js/access/password/1977") {
res.write("Good Job you have entered the correct password into the url bar we're now loading this page content on this webserver");
res.write("Content Loaded, it seem you will have to use another page on this webserver to access the content that here before");
res.write("It seem like the developer has moved the content from this webpage on this webserver to another page we will sent you the link here localhost:443/service/content/assets/js/access/pass/granted/content/load/developer-pass/main.js");
res.end();
} else if (req.url == "/assets/js/access/password") {
res.write("Please enter the correct password into the url bar for our operating system to grant you access");
res.end();
} else if (req.url == "/host/app") {
res.setHeader('Content-type', 'application/json');
res.setHeader('Access-Control-Allow-Or-Deny', "Allow from views");
res.writeHead(200); //status code HTTP 200 / OK

  let dataObj = {"id":443, "name":"App", "email":"app@work.org"};
  let data = JSON.stringify(dataObj);
  let index = {"id":443, "name":"index", "text":"Welcome"}
  res.end(data);
} else if (req.url == "/host/app/test") {
  res.setHeader('Content-type',  'application/json');
  res.setHeader('Access-Control-Allow-Or-Deny', "Allow from views");
  res.writeHead(300); //status code HTTP 300 / OK

  let rf2 = alert
  let index = {"id":"443", "name":"test", "text":"test"}
  let d1 = JSON.stringify(index);
  res.end(rf2);
} else if (req.url == "/host/app/test/index/password/1977/code/1") {
  let ht1 = al1
  let rf1 = alert
  res.end(rf1);
} else if (req.url == "/host/app/invalid-request") {
  let erp1 = error1
  let erpt1 = title1
  res.end(erp1);
} else if (req.url == "/host/app/dashboard") {
  let rf2 = alert
  let ht2 = al1
  let dtr1 = dashboard
  res.end(rf2);
}

});
axios.post('http://localhost:443/host/app', {
market: 'Buy the milk',
});
const app = express();

app.use(
express.urlencoded({
extended: true,
}),
);

app.use(express.json());

app.post('localhost:443/host/app', (req, res) => {
console.log(req.body.todo);
});

server.listen(port, () => {
console.log(Server running at port ${port})
});

client.on("message", msg => {
if (msg.content === "/ping") {
msg.reply("pong");
} else if (msg.content == "/test") {
msg.reply("This command is still under development please try again when the bot updated")
} else if (msg.content == "start") {
msg.reply("Unknown Commands, Did you entered correctly, try using /start")
} else if (msg.content == "/start") {
msg.reply("You have started your adventure, on solarishost, you can start mining by doing s!mine")
} else if (msg.content == "/mine") {
msg.reply("This command is still under development, please try again later when the bot has updated")
} else if (msg.content == "/help") {
msg.reply("You can use the command listed here: s!help, s!start, s!test, s!ping, s!mine, s!stop, s!pokemon find")
} else if (msg.content == "help") {
msg.reply("Please use /help to ask for help for this bot instead of using help to ask for help in command list")
} else if (msg.content == "/stop") {
msg.reply("You have stopped your adventure, to get a break")
} else if (msg.content == "/pokemon find") {
msg.reply("Sorry this feature is still in development")
}
});

client.login('Your Token');

@cassitly
Copy link

and also here the error

ConnectTimeoutError: Connect Timeout Error
at onConnectTimeout (C:\Users\jedik\IdeaProjects\NextUptime-Bot\node_module
s\undici\lib\core\connect.js:131:24)
at C:\Users\jedik\IdeaProjects\NextUptime-Bot\node_modules\undici\lib\core
connect.js:78:46
at Immediate.onImmediate (C:\Users\jedik\IdeaProjects\NextUptime-Bot\node
modules\undici\lib\core\connect.js:117:33)
at processImmediate (node:internal/timers:466:21) {
code: 'UND_ERR_CONNECT_TIMEOUT'

@JS-AK
Copy link

JS-AK commented Oct 26, 2022

i have same problem in undici.request() or undici.fetch().

Connect Timeout Error

But it only happens when i pass dispatcher field to options

import { Agent, request } from "undici";

...

const base64encodedData = Buffer.from(this.#username + ":" + this.#password).toString("base64");
const { body, statusCode } = await request(this.#url, {
  dispatcher: new Agent({
    connect: { rejectUnauthorized: false, timeout: 60_000 },
  }),
  headers: { Authorization: "Basic " + base64encodedData },
  method: "POST",
});

when i dont pass dispatcher to options it works fine without any Connect Timeout Error

import { Agent, request } from "undici";

...

const base64encodedData = Buffer.from(this.#username + ":" + this.#password).toString("base64");
const { body, statusCode } = await request(this.#url, {
  headers: { Authorization: "Basic " + base64encodedData },
  method: "POST",
});

But in my case i needed option rejectUnauthorized: false

update

if i pass globally at top



import { Agent, request } from "undici";
const dispatcher = new Agent({
  connect: { rejectUnauthorized: false, timeout: 60_000 },
})

...

const base64encodedData = Buffer.from(this.#username + ":" + this.#password).toString("base64");
const { body, statusCode } = await request(this.#url, {
  dispatcher,
  headers: { Authorization: "Basic " + base64encodedData },
  method: "POST",
});

app works much better

@longzheng
Copy link

longzheng commented Nov 17, 2022

Just wanted to make a note for anyone else that experiences this error UND_ERR_CONNECT_TIMEOUT running on Azure App Service, the actual root cause is because App Service enforces a SNAT outbound connection limit which also results in fetch failed if you try to make a lot of outbound requests very quickly.

Inspired by #1531 (comment) I found it is possible to configure the max connections limit (per host) using

import { fetch, setGlobalDispatcher, Agent, Pool } from "undici";

setGlobalDispatcher(
  new Agent({ factory: (origin) => new Pool(origin, { connections: 128 }) })
);

fetch(...)

which alleviates the problem on Azure App Service

@TheGalacticAce
Copy link

hey, i've run into this issue and I don't understand why. how can i fix it?

@Radiergummi
Copy link

Radiergummi commented Jan 11, 2023

I think I am affected by this issue, too - I can't make any requests to hosts with an AAAA record outside of my network:

> await fetch('https://cloudflare.com');
Uncaught TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11118:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async REPL7:1:33 {
  cause: ConnectTimeoutError: Connect Timeout Error
      at onConnectTimeout (node:internal/deps/undici/undici:6625:28)
      at node:internal/deps/undici/undici:6583:50
      at Immediate._onImmediate (node:internal/deps/undici/undici:6614:13)
      at process.processImmediate (node:internal/timers:471:21)
      at process.topLevelDomainCallback (node:domain:161:15)
      at process.callbackTrampoline (node:internal/async_hooks:128:24) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}

It works for services like GitHub, which refuse to implement IPv6 and only have A records:

> await fetch('https://github.com');
Response {
  [Symbol(realm)]: null,
  [Symbol(state)]: Proxy [
    {
      aborted: false,
      rangeRequested: false,
      timingAllowPassed: true,
      requestIncludesCredentials: true,
      type: 'default',
      status: 200,
      timingInfo: [Object],
      cacheState: '',
      statusText: 'OK',
      headersList: [HeadersList],
      urlList: [Array],
      body: [Object]
    },
    { get: [Function: get], set: [Function: set] }
  ],
  [Symbol(headers)]: HeadersList { /* ... */ }
}

Is this expected? Can I do anything for the time being?

@mcollina
Copy link
Member

mcollina commented Feb 3, 2023

You need to flip the setting in Node.js core:

node --dns-result-order=ipv4first

Actually I think we can ship a new default here and enable autoSelectFamily: true:

https://nodejs.org/docs/latest-v18.x/api/net.html#socketconnectoptions-connectlistener.

and enable happy eyeballs in Node v18.

@ronag wdyt?

@bcomnes
Copy link
Contributor

bcomnes commented Mar 5, 2023

Running into this with the official node:19-alpine docker image running on fly.io. It seems to have something to do with a configuration in the image making it impossible to resolve domains with AAAA IPv6 records. dns-result-order would obviously fix this, but it would also be nice to figure out how to make AAAA records resolve property. I'll post back if I find anything.

@mcollina
Copy link
Member

mcollina commented Mar 5, 2023

@bcomnes it's not a problem of DNS resolution but rather ipv6 connectivity. contact fly support. They had ipv6 issues in the past and they fixed them, so possibly it's on their side.

@bcomnes
Copy link
Contributor

bcomnes commented Mar 5, 2023

Odd, I just deployed with autoSelectFamily: true: which made connections to domains that resolve to ipv6 work, but on a subsequent deploy it did not work. I will open an issue with them.

EDIT: I believe this was a fly IPv6 connective issue that was resolved.

@ic-agouveia
Copy link

This error happened to me while running nodemon without making any fetch request.

My WiFi router emits at the same time a 5G and a 2.4G bands merged/mixed into one wireless network.

Due to coming close to the router, my Mac auto-connected to the 5G and this UND_ERR_CONNECT_TIMEOUT error started showing. I turned off the VPN and it did not, but when I turned it on again, even with a different DNS the error showed up again.

I manually disconnected from the wireless network and reconnected. The error does not show up any longer…

@MikeThuchchai
Copy link

In my case, switch node from 18 to 20 is solved
(on sveltekit server)

GabiGrin added a commit to flydelabs/flyde-discord-bot-boilerplate that referenced this issue Aug 6, 2023
synzen added a commit to synzen/MonitoRSS that referenced this issue Jan 28, 2024
undergroundwires added a commit to undergroundwires/privacy.sexy that referenced this issue Mar 29, 2024
This commit introduces the `force-ipv4` GitHub action to address
connectivity issues caused by the lack of IPv6 support in GitHub
runners. Details:
- actions/runner#3138
- actions/runner-images#668

This change solves connection problems when Node's `fetch` API fails due
to `UND_ERR_CONNECT_TIMEOUT` errors. Details:
- actions/runner-images#9540
- actions/runner#3213

This action disables IPv6 at the system level, ensuring all outging
requests use IPv4. Resolving connectivity issues when running external
URL checks and Docker build checks.

This solution is a temporary workaround until GitHub runners support
IPv6 or Node `fetch` API has a working solution such as Happy Eyeball.
Detais:
- nodejs/node#41625
- nodejs/undici#1531
@undergroundwires
Copy link

Here's my workaround (open-source and documented) that I hope that can help you too:

After days of research and trial/error, this is how I got this working:

  1. Create a script called force-ipv4.sh, that configures system to prefer IPv4 over IPv6, call it to configure the machine. It was not easy to find a reliable cross-platform solution and I went Cloudflare WARP for DNS resolution along with some system configurations.
  2. To easily use the script in GitHub workflows, create GitHub action called force-ipv4 and call it in GitHub runners.
  3. Fixes the IPv6 request issues, and you can happily run e.g. fetch API from Node.

Related commit introducing this fix: undergroundwires/privacy.sexy@52fadcd

@mahevstark
Copy link

In my case, switch node from 18 to 20 is solved (on sveltekit server)

This solved it for me too

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.