fetch('https://www.php.net') using official nodejs binaries results in 503 - but 200 on other binaries, in browser, curl, etc.

[amezin]

Node.js Version

v22.12.0

NPM Version

10.9.0

Operating System

Linux amezin-laptop.home.arpa 6.12.4-zen1-1-zen #1 ZEN SMP PREEMPT_DYNAMIC Mon, 09 Dec 2024 14:30:31 +0000 x86_64 GNU/Linux

Subsystem

https

Description

1. Download https://nodejs.org/dist/v22.12.0/node-v22.12.0-linux-x64.tar.xz and unpack
2. Run bin/node
3. await fetch('https://www.php.net')

Result:

Response {
  status: 503,
  statusText: 'Service Temporarily Unavailable',
  headers: Headers {
    server: 'myracloud',
    date: 'Mon, 16 Dec 2024 07:43:14 GMT',
    'content-type': 'text/html',
    'transfer-encoding': 'chunked',
    connection: 'keep-alive',
    'cache-control': 'no-cache, no-store, max-age=0',
    'x-content-type-options': 'nosniff',
    'x-xss-protection': '1; mode=block',
    'x-frame-options': 'SAMEORIGIN'
  },
  body: ReadableStream { locked: false, state: 'readable', supportsBYOB: true },
  bodyUsed: false,
  ok: false,
  redirected: false,
  type: 'basic',
  url: 'https://www.php.net/'
}

When I do the same await fetch('https://www.php.net') in nodejs-lts-iron-20.18.1-1, or nodejs-23.4.0-1, provided by Arch Linux, I get:

Response {
  status: 200,
  statusText: 'OK',
  headers: Headers {
    server: 'myracloud',
    date: 'Mon, 16 Dec 2024 07:43:56 GMT',
    'content-type': 'text/html; charset=utf-8',
    'transfer-encoding': 'chunked',
    connection: 'keep-alive',
    'last-modified': 'Mon, 16 Dec 2024 07:30:09 GMT',
    'content-language': 'en',
    'permissions-policy': 'interest-cohort=()',
    'x-frame-options': 'SAMEORIGIN',
    'set-cookie': 'LAST_NEWS=1734335036; expires=Tue, 16 Dec 2025 07:43:56 GMT; Max-Age=31536000; path=/; domain=.php.net',
    link: '<https://www.php.net/index>; rel=shorturl',
    'content-encoding': 'gzip',
    vary: 'accept-encoding',
    expires: 'Mon, 16 Dec 2024 07:43:56 GMT',
    'cache-control': 'max-age=0'
  },
  body: ReadableStream { locked: false, state: 'readable', supportsBYOB: true },
  bodyUsed: false,
  ok: true,
  redirected: false,
  type: 'basic',
  url: 'https://www.php.net/'
}

Also the page opens successfully in Firefox, curl successfully downloads it.

Why does it happen and can it be fixed?

Minimal Reproduction

No response

Output

No response

Before You Submit

• I have looked for issues that already exist before submitting this
• My issue follows the guidelines in the README file, and follows the 'How to ask a good question' guide at https://stackoverflow.com/help/how-to-ask

[preveen-stack]

I was able to reproduce this from raspberry pi 4B for node 22.12 and node 18.19.
But on macos laptop with nodejs 22.7 on the same network it was working fine!

@nodejs/net Can you check

~/pg/nodeissue/node-v22.12.0-linux-arm64 $ ./bin/node 
Welcome to Node.js v22.12.0.
Type ".help" for more information.
> await fetch("https://www.php.net")
Response {
  status: 503,
  statusText: 'Service Temporarily Unavailable',
  headers: Headers {
    server: 'myracloud',
    date: 'Wed, 18 Dec 2024 09:01:47 GMT',
    'content-type': 'text/html',
    'transfer-encoding': 'chunked',
    connection: 'keep-alive',
    'cache-control': 'no-cache, no-store, max-age=0',
    'x-content-type-options': 'nosniff',
    'x-xss-protection': '1; mode=block',
    'x-frame-options': 'SAMEORIGIN'
  },
  body: ReadableStream { locked: false, state: 'readable', supportsBYOB: true },
  bodyUsed: false,
  ok: false,
  redirected: false,
  type: 'basic',
  url: 'https://www.php.net/'
}
> await fetch("https://www.php.net")
Response {
  status: 503,
  statusText: 'Service Temporarily Unavailable',
  headers: Headers {
    server: 'myracloud',
    date: 'Wed, 18 Dec 2024 09:01:57 GMT',
    'content-type': 'text/html',
    'transfer-encoding': 'chunked',
    connection: 'keep-alive',
    'cache-control': 'no-cache, no-store, max-age=0',
    'x-content-type-options': 'nosniff',
    'x-xss-protection': '1; mode=block',
    'x-frame-options': 'SAMEORIGIN'
  },
  body: ReadableStream { locked: false, state: 'readable', supportsBYOB: true },
  bodyUsed: false,
  ok: false,
  redirected: false,
  type: 'basic',
  url: 'https://www.php.net/'
}
> .exit

$ node 
Welcome to Node.js v18.19.0.
Type ".help" for more information.
> await fetch("https://www.php.net")
Response {
  [Symbol(realm)]: null,
  [Symbol(state)]: Proxy [
    {
      aborted: false,
      rangeRequested: false,
      timingAllowPassed: true,
      requestIncludesCredentials: true,
      type: 'default',
      status: 503,
      timingInfo: [Object],
      cacheState: '',
      statusText: 'Service Temporarily Unavailable',
      headersList: [HeadersList],
      urlList: [Array],
      body: [Object]
    },
    { get: [Function: get], set: [Function: set] }
  ],
  [Symbol(headers)]: HeadersList {
    cookies: null,
    [Symbol(headers map)]: Map(9) {
      'server' => [Object],
      'date' => [Object],
      'content-type' => [Object],
      'transfer-encoding' => [Object],
      'connection' => [Object],
      'cache-control' => [Object],
      'x-content-type-options' => [Object],
      'x-xss-protection' => [Object],
      'x-frame-options' => [Object]
    },
    [Symbol(headers map sorted)]: null
  }
}
> await fetch("https://www.php.net")
Response {
  [Symbol(realm)]: null,
  [Symbol(state)]: Proxy [
    {
      aborted: false,
      rangeRequested: false,
      timingAllowPassed: true,
      requestIncludesCredentials: true,
      type: 'default',
      status: 503,
      timingInfo: [Object],
      cacheState: '',
      statusText: 'Service Temporarily Unavailable',
      headersList: [HeadersList],
      urlList: [Array],
      body: [Object]
    },
    { get: [Function: get], set: [Function: set] }
  ],
  [Symbol(headers)]: HeadersList {
    cookies: null,
    [Symbol(headers map)]: Map(9) {
      'server' => [Object],
      'date' => [Object],
      'content-type' => [Object],
      'transfer-encoding' => [Object],
      'connection' => [Object],
      'cache-control' => [Object],
      'x-content-type-options' => [Object],
      'x-xss-protection' => [Object],
      'x-frame-options' => [Object]
    },
    [Symbol(headers map sorted)]: null
  }
}
> .exit

Preveens-MacBook-Pro:preveen$ node
Welcome to Node.js v22.7.0.
Type ".help" for more information.
> await fetch("https://www.php.net")
Response {
  status: 200,
  statusText: 'OK',
  headers: Headers {
    server: 'myracloud',
    date: 'Wed, 18 Dec 2024 09:04:11 GMT',
    'content-type': 'text/html; charset=utf-8',
    'transfer-encoding': 'chunked',
    connection: 'keep-alive',
    'last-modified': 'Wed, 18 Dec 2024 08:50:09 GMT',
    'content-language': 'en',
    'permissions-policy': 'interest-cohort=()',
    'x-frame-options': 'SAMEORIGIN',
    'set-cookie': 'LAST_NEWS=1734512650; expires=Thu, 18 Dec 2025 09:04:11 GMT; Max-Age=31536000; path=/; domain=.php.net',
    link: '<https://www.php.net/index>; rel=shorturl',
    'content-encoding': 'gzip',
    vary: 'accept-encoding',
    expires: 'Wed, 18 Dec 2024 09:04:11 GMT',
    'cache-control': 'max-age=0'
  },
  body: ReadableStream { locked: false, state: 'readable', supportsBYOB: true },
  bodyUsed: false,
  ok: true,
  redirected: false,
  type: 'basic',
  url: 'https://www.php.net/'
}
> 

[pimterry]

I can reproduce this myself: requests with Node are all rejected, while other tools all work correctly. This happens with fetch and the https module, for Node v12+ (Node 10 works fine), making requests from Linux.

It looks like php.net (or more likely the CDN they're using) now blocks some of Node.js's TLS fingerprints completely. Not all of them (not the fingerprint from Macs, by the sounds of @preveen-stack's tests) but at least requests from Node on Linux. I'd guess they've seen some bot traffic with that signature, and some anti-bot mechanisms somewhere have marked it as suspicious.

Testing with HTTP Toolkit to check: making the request with any recent Node returns 503, making the request via HTTP Toolkit (which proxies the same content with identical header data etc, but does affect the TLS fingerprint) returns a 200. HTTP Toolkit itself is actually built with Node.js, but modifies its own fingerprint to roughly match Firefox here: https://github.com/httptoolkit/mockttp/blob/703fa1f06fdfa91ddc3ba8494c94747c90462c31/src/rules/passthrough-handling.ts#L25-L100

Copying that logic, if I make a request with the equivalent cipher list, it works:

// These are most of the standard ciphers Node.js uses by default, but reordered
// (Please check this list is correct for your case if you're doing anything
// important with this!)
const ciphers = [
  'TLS_AES_128_GCM_SHA256',
  'TLS_CHACHA20_POLY1305_SHA256',
  'TLS_AES_256_GCM_SHA384',
  'ECDHE-ECDSA-AES128-GCM-SHA256',
  'ECDHE-RSA-AES128-GCM-SHA256',
  'ECDHE-ECDSA-CHACHA20-POLY1305',
  'ECDHE-RSA-CHACHA20-POLY1305',
  'ECDHE-ECDSA-AES256-GCM-SHA384',
  'ECDHE-RSA-AES256-GCM-SHA384',
  'ECDHE-ECDSA-AES256-SHA',
  'ECDHE-ECDSA-AES128-SHA',
  'ECDHE-RSA-AES128-SHA',
  'ECDHE-RSA-AES256-SHA',
  'AES128-GCM-SHA256',
  'AES256-GCM-SHA384'
].join(':')

require('https').request('https://www.php.net', { ciphers }, r => console.log('response', r.statusCode)).end()

This prints response 200 while the same line without ciphers prints response 503.

I wrote an article about workarounds to solve TLS fingerprinting issues in general a while back here: https://httptoolkit.com/blog/tls-fingerprinting-node-js/. There's an issue discussing possible solutions in Node at nodejs/node#41112 but no real fix coming imminently there I'm afraid. For now, the easiest way to resolve this will be to tweak your request's TLS config somehow as above.