Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prevent 504 response caching #2166

Open
shark0der opened this issue Oct 12, 2021 · 0 comments
Open

Prevent 504 response caching #2166

shark0der opened this issue Oct 12, 2021 · 0 comments
Labels
investigate Under investigation and may be a bug.

Comments

@shark0der
Copy link

shark0der commented Oct 12, 2021 •
edited
Loading

Description
We're using ethers in a server-side worker that polls information from a contract every few minutes. From time to time Nginx that acts as a reverse proxy for the node fails to reach the node and returns a 504 Gateway Timeout response. Ethers (or maybe nodejs' http[s] client) caches the response forever and upon retrying will get the same response, even if the node came back online and Nginx would be able to fulfill the request without error. Even a few days later the error would show the nginx response from a few days ago when it was initially sent, without actually resending the request to Nginx.

Here's the stack trace that we get:

15|pinner  | 2021-10-12 09:31:43 +00:00: Error: bad response (status=504, headers={"date":"Thu, 30 Sep 2021 20:01:43 GMT","content-type":"text/html","content-length":"176","connection":"close","server":"Nginx"}, body="<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n", requestBody="{\"method\":\"eth_blockNumber\",\"params\":[],\"id\":23379,\"jsonrpc\":\"2.0\"}", requestMethod="POST", url="http://erigon.example.com", code=SERVER_ERROR, version=web/5.0.9)
15|pinner  | 2021-10-12 09:31:43 +00:00:     at Logger.makeError (/home/ubuntu/pinner/node_modules/@ethersproject/logger/lib/index.js:179:21)
15|pinner  | 2021-10-12 09:31:43 +00:00:     at Logger.throwError (/home/ubuntu/pinner/node_modules/@ethersproject/logger/lib/index.js:188:20)
15|pinner  | 2021-10-12 09:31:43 +00:00:     at /home/ubuntu/pinner/node_modules/@ethersproject/web/lib/index.js:227:36
15|pinner  | 2021-10-12 09:31:43 +00:00:     at step (/home/ubuntu/pinner/node_modules/@ethersproject/web/lib/index.js:33:23)
15|pinner  | 2021-10-12 09:31:43 +00:00:     at Object.next (/home/ubuntu/pinner/node_modules/@ethersproject/web/lib/index.js:14:53)
15|pinner  | 2021-10-12 09:31:43 +00:00:     at fulfilled (/home/ubuntu/pinner/node_modules/@ethersproject/web/lib/index.js:5:58)
15|pinner  | 2021-10-12 09:31:43 +00:00:     at runMicrotasks (<anonymous>)
15|pinner  | 2021-10-12 09:31:43 +00:00:     at processTicksAndRejections (internal/process/task_queues.js:97:5) {
15|pinner  | 2021-10-12 09:31:43 +00:00:   reason: 'bad response',
15|pinner  | 2021-10-12 09:31:43 +00:00:   code: 'SERVER_ERROR',
15|pinner  | 2021-10-12 09:31:43 +00:00:   status: 504,
15|pinner  | 2021-10-12 09:31:43 +00:00:   headers: {
15|pinner  | 2021-10-12 09:31:43 +00:00:     date: 'Thu, 30 Sep 2021 20:01:43 GMT',
15|pinner  | 2021-10-12 09:31:43 +00:00:     'content-type': 'text/html',
15|pinner  | 2021-10-12 09:31:43 +00:00:     'content-length': '176',
15|pinner  | 2021-10-12 09:31:43 +00:00:     connection: 'close',
15|pinner  | 2021-10-12 09:31:43 +00:00:     server: 'Nginx'
15|pinner  | 2021-10-12 09:31:43 +00:00:   },
15|pinner  | 2021-10-12 09:31:43 +00:00:   body: '<html>\r\n' +
15|pinner  | 2021-10-12 09:31:43 +00:00:     '<head><title>504 Gateway Time-out</title></head>\r\n' +
15|pinner  | 2021-10-12 09:31:43 +00:00:     '<body bgcolor="white">\r\n' +
15|pinner  | 2021-10-12 09:31:43 +00:00:     '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
15|pinner  | 2021-10-12 09:31:43 +00:00:     '<hr><center>nginx</center>\r\n' +
15|pinner  | 2021-10-12 09:31:43 +00:00:     '</body>\r\n' +
15|pinner  | 2021-10-12 09:31:43 +00:00:     '</html>\r\n',
15|pinner  | 2021-10-12 09:31:43 +00:00:   requestBody: '{"method":"eth_blockNumber","params":[],"id":23379,"jsonrpc":"2.0"}',
15|pinner  | 2021-10-12 09:31:43 +00:00:   requestMethod: 'POST',
15|pinner  | 2021-10-12 09:31:43 +00:00:   url: 'http://erigon.example.com'
15|pinner  | 2021-10-12 09:31:43 +00:00: }

Notice that the timestamp when this was logged was on 12th of October but the date header returned by nginx is 30th of September. I've also inspected the logs and it confirms that the requests are not being sent to nginx.

Environment:
Node version: v12.22.1
Ethers version: 5.0.9
Node: Erigon (but most likely irrelevant).

Although the ethers version is older, the web package hasn't changed in the last year so I suppose we might hit the same issue with the latest. I've just updated the package to the latest version and I'll add a comment whenever we hit the issue again.

Potential solution
One important thing that I noticed is that the request id that's being "sent" is the same so bumping the id whenever an error is encountered should probably avoid the caching issue.
@shark0der shark0der added the investigate Under investigation and may be a bug. label Oct 12, 2021
@Woodpile37 Woodpile37 mentioned this issue Oct 5, 2023
Merged
Woodpile37 added a commit to Woodpile37/ethers.js that referenced this issue Oct 5, 2023
@Woodpile37
[Snyk] Upgrade ws from 8.14.0 to 8.14.1 (#94)
8b0090f 
<p>This PR was automatically created by Snyk using the credentials of a
real user.</p><br /><h3>Snyk has created this PR to upgrade ws from
8.14.0 to 8.14.1.</h3>

:information_source: Keep your dependencies up-to-date. This makes it
easier to fix existing vulnerabilities and to more quickly identify and
fix newly disclosed vulnerabilities when they affect your project.
<hr/>

- The recommended version is **1 version** ahead of your current
version.
- The recommended version was released **a month ago**, on 2023-09-08.


<details>
<summary><b>Release notes</b></summary>
<br/>
  <details>
    <summary>Package name: <b>ws</b></summary>
    <ul>
      <li>
<b>8.14.1</b> - <a
href="https://snyk.io/redirect/github/websockets/ws/releases/tag/8.14.1">2023-09-08</a></br><h1>Bug
fixes</h1>
<ul>
<li>Improved the reliability of two tests for <a
href="https://snyk.io/redirect/github/nodejs/citgm">CITGM</a> (<a
class="commit-link" data-hovercard-type="commit"
data-hovercard-url="https://github.com/websockets/ws/commit/fd3c64cbd60606f75763350133ba2757b6a64545/hovercard"
href="https://snyk.io/redirect/github/websockets/ws/commit/fd3c64cbd60606f75763350133ba2757b6a64545"><tt>fd3c64c</tt></a>).</li>
</ul>
      </li>
      <li>
<b>8.14.0</b> - <a
href="https://snyk.io/redirect/github/websockets/ws/releases/tag/8.14.0">2023-09-06</a></br><h1>Features</h1>
<ul>
<li>The <code>WebSocket</code> constructor now accepts HTTP(S) URLs (<a
class="issue-link js-issue-link" data-error-text="Failed to load title"
data-id="1858169494" data-permission-text="Title is private"
data-url="websockets/ws#2162"
data-hovercard-type="pull_request"
data-hovercard-url="/websockets/ws/pull/2162/hovercard"
href="https://snyk.io/redirect/github/websockets/ws/pull/2162">#2162</a>).</li>
<li>The <code>socket</code> argument of
<code>server.handleUpgrade()</code> can now be a generic<br>
<code>Duplex</code> stream (<a class="issue-link js-issue-link"
data-error-text="Failed to load title" data-id="1873628991"
data-permission-text="Title is private"
data-url="websockets/ws#2165"
data-hovercard-type="pull_request"
data-hovercard-url="/websockets/ws/pull/2165/hovercard"
href="https://snyk.io/redirect/github/websockets/ws/pull/2165">#2165</a>).</li>
</ul>
<h1>Other notable changes</h1>
<ul>
<li>At most one event per microtask is now emitted (<a class="issue-link
js-issue-link" data-error-text="Failed to load title"
data-id="1857764370" data-permission-text="Title is private"
data-url="websockets/ws#2160"
data-hovercard-type="pull_request"
data-hovercard-url="/websockets/ws/pull/2160/hovercard"
href="https://snyk.io/redirect/github/websockets/ws/pull/2160">#2160</a>).</li>
</ul>
      </li>
    </ul>
from <a href="https://snyk.io/redirect/github/websockets/ws/releases">ws
GitHub release notes</a>
  </details>
</details>


<details>
  <summary><b>Commit messages</b></summary>
  </br>
  <details>
    <summary>Package name: <b>ws</b></summary>
    <ul>
<li><a
href="https://snyk.io/redirect/github/websockets/ws/commit/7460049ff0a61bef8d5eda4b1d5c8170bc7d6b6f">7460049</a>
[dist] 8.14.1</li>
<li><a
href="https://snyk.io/redirect/github/websockets/ws/commit/397b89e3db6782022bbcf328b1191f5a1eb7800f">397b89e</a>
[ci] Update actions/checkout action to v4</li>
<li><a
href="https://snyk.io/redirect/github/websockets/ws/commit/fd3c64cbd60606f75763350133ba2757b6a64545">fd3c64c</a>
[test] Fix flaky tests on Windows</li>
<li><a
href="https://snyk.io/redirect/github/websockets/ws/commit/ae60ce0d1eaa239844bc8d60d220b47e302c3d45">ae60ce0</a>
[ci] Cache downloaded npm dependencies (ethers-io#2166)</li>
<li><a
href="https://snyk.io/redirect/github/websockets/ws/commit/511aefece49ee38c6fcca19d230c115fbfeaefd8">511aefe</a>
[pkg] Silence npm warning</li>
<li><a
href="https://snyk.io/redirect/github/websockets/ws/commit/ddba690ab8c5da2da2fc9af3131d5e5629cbdbd4">ddba690</a>
[doc] Fix the type of the &#x60;socket&#x60; argument</li>
    </ul>

<a
href="https://snyk.io/redirect/github/websockets/ws/compare/d30768405fc295f0365c4bad8b7e14a9ad54c64b...7460049ff0a61bef8d5eda4b1d5c8170bc7d6b6f">Compare</a>
  </details>
</details>
<hr/>

**Note:** *You are seeing this because you or someone else with access
to this repository has authorized Snyk to open upgrade PRs.*

For more information: <img
src="https://api.segment.io/v1/pixel/track?data=eyJ3cml0ZUtleSI6InJyWmxZcEdHY2RyTHZsb0lYd0dUcVg4WkFRTnNCOUEwIiwiYW5vbnltb3VzSWQiOiJjZmMwZDE2Ni0zY2Y0LTQ3NjgtYjBiNi03MjEzMTBjYjBiZWQiLCJldmVudCI6IlBSIHZpZXdlZCIsInByb3BlcnRpZXMiOnsicHJJZCI6ImNmYzBkMTY2LTNjZjQtNDc2OC1iMGI2LTcyMTMxMGNiMGJlZCJ9fQ=="
width="0" height="0"/>

🧐 [View latest project
report](https://app.snyk.io/org/woodpile37/project/0f28f541-dd84-4216-97e4-d1b22c099c64?utm_source&#x3D;github&amp;utm_medium&#x3D;referral&amp;page&#x3D;upgrade-pr)

🛠 [Adjust upgrade PR
settings](https://app.snyk.io/org/woodpile37/project/0f28f541-dd84-4216-97e4-d1b22c099c64/settings/integration?utm_source&#x3D;github&amp;utm_medium&#x3D;referral&amp;page&#x3D;upgrade-pr)

🔕 [Ignore this dependency or unsubscribe from future upgrade
PRs](https://app.snyk.io/org/woodpile37/project/0f28f541-dd84-4216-97e4-d1b22c099c64/settings/integration?pkg&#x3D;ws&amp;utm_source&#x3D;github&amp;utm_medium&#x3D;referral&amp;page&#x3D;upgrade-pr#auto-dep-upgrades)

<!---
(snyk:metadata:{"prId":"cfc0d166-3cf4-4768-b0b6-721310cb0bed","prPublicId":"cfc0d166-3cf4-4768-b0b6-721310cb0bed","dependencies":[{"name":"ws","from":"8.14.0","to":"8.14.1"}],"packageManager":"npm","type":"auto","projectUrl":"https://app.snyk.io/org/woodpile37/project/0f28f541-dd84-4216-97e4-d1b22c099c64?utm_source=github&utm_medium=referral&page=upgrade-pr","projectPublicId":"0f28f541-dd84-4216-97e4-d1b22c099c64","env":"prod","prType":"upgrade","vulns":[],"issuesToFix":[],"upgrade":[],"upgradeInfo":{"versionsDiff":1,"publishedDate":"2023-09-08T16:04:44.943Z"},"templateVariants":[],"hasFixes":false,"isMajorUpgrade":false,"isBreakingChange":false,"priorityScoreList":[]})
--->
@CharliesAngel1 CharliesAngel1 mentioned this issue Oct 16, 2023
Open
@Jircs1 Jircs1 mentioned this issue Mar 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
investigate Under investigation and may be a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@shark0der