Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: devtools is causing a severe memory leak #761

Closed
mjlehrke opened this issue Dec 9, 2024 · 16 comments
Closed

fix: devtools is causing a severe memory leak #761

mjlehrke opened this issue Dec 9, 2024 · 16 comments
Labels
bug Something isn't working

Comments

@mjlehrke
Copy link

mjlehrke commented Dec 9, 2024

🐛 The bug

Creating a bare Nuxt project with npx nuxi@latest init project and starting it with npm run dev causes severe memory leak (most of the time). This happens on Windows with Node v22+. This happens on a bare Nuxt starter with no additional Nuxt modules or npm packages added. Importantly, the memory leak occurs without any incoming requests happening or the app open in a browser. You only need to npm run dev after doing the nuxi project init and it will start leaking immediately. This is going to be a pain in the ass because I noticed it usually doesn't happen on the first run of npm run dev but if you stop the dev server and run the command again it will happen (you know you have it when the CPU is high and the memory starts incrementing).

I believe this is different than nuxt/nuxt#30164 as that is in a built project with requests.

Observations:

  • It does not leak when building and running npm run preview.
  • Requests to the dev server are not needed to leak memory.
  • Disabling devtools prevents the memory leak.
  • It does not leak every time the dev server is started, maybe 10% of the time it will run normally on the bare project.
  • It also leaks with Node v22.12.0 in addition to v23.3.0.
  • Abnormally high CPU usage is occurring.
  • It ticks up by 2.7-2.9 MB per second.

Proof

These images were taken after enabling/disabling Nuxt devtools and running npm run dev and waiting for exactly 5 minutes.

devtools: { enabled: true }
Image

devtools: { enabled: false}
Image

Snapshots

These make it seem like there is something with \node_modules\unstorage\node_modules\readdirp\index.js and/or are-we-there-yet\node_modules\readable-stream. But also defineNuxtSchema.

Here's how I did this:

  1. npx nuxi dev --inspect with devtools disabled
  2. Open Chrome and go to chrome://inspect and inspect the Node project
  3. Let it run 5 minutes, collect garbage and take another snapshot (Snapshot 1, control)
  4. Edit nuxt.config.ts and enable devtools, let it restart with HMR
  5. Let it run 10 minutes, then collect garbage and take another snapshot (Snapshot 2)
  6. Let it run 10 more minutes, then collect garbage and take final snapshot (Snapshot 3)

Snapshot 2 vs Snapshot 3 (devtools enabled short vs longer leak)
Image
Image
Image
Image
Image
Image

Full reproduction

Since I don't think stackblitz is going to show this issue:

  1. npx nuxi init@latest project
  2. cd project
  3. npm run dev
  4. Quit the process and re-run npm run dev
  5. Observe memory leak
  6. npx nuxi devtools disable
  7. npm run dev
  8. Observe no memory leak

🛠️ To reproduce

https://stackblitz.com/edit/github-4exzk3tl

🌈 Expected behavior

devtools should not be causing a memory leak in a bare project created with npx nuxi@latest init project

ℹ️ Additional context

npx nuxi info

  • Operating System: Windows_NT
  • Node Version: v23.3.0
  • Nuxt Version: 3.14.1592
  • CLI Version: 3.16.0
  • Nitro Version: 2.10.4
  • Package Manager: npm@10.9.1
  • Builder: -
  • User Config: default
  • Runtime Modules: -
  • Build Modules: -

Full nuxt.config.ts (devtools enabled)

// https://nuxt.com/docs/api/configuration/nuxt-config
export default defineNuxtConfig({
  devtools: {
    enabled: true,
    timeline: { enabled: true },
  },

  compatibilityDate: '2024-11-01',
})

Full nuxt.config.ts (devtools disabled)

// https://nuxt.com/docs/api/configuration/nuxt-config
export default defineNuxtConfig({
  devtools: {
    enabled: false,
    timeline: { enabled: true },
  },

  compatibilityDate: '2024-11-01',
})
@mjlehrke mjlehrke added the bug Something isn't working label Dec 9, 2024
@antfu
Copy link
Member

antfu commented Dec 9, 2024

Hey Michael, thanks a lot for the detailed investigation of the issue.

First of all, I would love to have a consensus about the definition of "Memory Leak", could you explain a bit of your interopation here?

To me, I think a memory leak is about a long-running production server that holds stable data from requests/loops that make the memory grow consistently and eventually hit the physical limit causing the crash. While in a way, I don't even think the concept applies to dev servers as they are not designed to be long-running. From what I read of your repo, I didn't see the memory growing over time (isn't Snapshot 2 & 3 has the same memory cost?), tho I agree it indeed consumes more memory - but that sounds reasonable to me and isn't really a "memory leak".

There are multiple reasons DevTools would cause more memory:

I am not an expert of this for sure, so feel free to point out anything I might be wrong about. I'd also love to hear more context of your side of the story, like why you are looking into this and what troubles you, etc.

Thanks!

@mjlehrke
Copy link
Author

mjlehrke commented Dec 10, 2024

Hey Anthony,

Sorry I should have been more clear. The memory consumption continues to grow over time until it crashes, not just a one time bump with the devtools enabled. I was looking into this because of my dev environment crashing. At first I thought it was due to something in the Nuxt app I was working on. At first I tried removing modules to try and pinpoint it with no success. I flipped the approach and tried to start with a fresh Nuxt init and slowly add modules from there, but I noticed right away that the problem was occurring in the newly initialized Nuxt app!

This was my first time capturing snapshots (and I only dev as a hobby). The size in the snapshot is not the memory consumption, I think that is the size of the snapshot data itself. If I download the snapshot data the file sizes are what was indicated in the screenshots.

But I would consider this a memory leak. If I leave the dev server running for about 2 hours while I'm working it will crash the Node process (and VSCode) when my machine runs out of memory. To mitigate this I've been restarting the Nuxt dev server frequently.

So I guess to better describe the images under the "proof" section. Without devtools enabled the bare Nuxt app uses about 230 MB of memory. Enabling devtools, it will immediately jump to ~800 MB of memory usage and then grow from there. In the screenshot with devtools enabled that was running for 5 minutes, it was already up to 1233 MB memory usage. I did try testing with timeline: true disabled and it still occurs.

I have 16 GB in my machine, and it will continue to grow until it crashes.

@kingyue737
Copy link
Contributor

Downgrade to node v22.11.0 can be another workaround without disabling devtools.

@antfu
Copy link
Member

antfu commented Dec 11, 2024

Downgrade to node v22.11.0 can be another workaround without disabling devtools.

That's interesting. @kingyue737 Do you have any more details to share why Node version would make a difference here?

@Tofandel
Copy link

Just FYI the node versioning system is weird and for all intent and purposes node 22 should not be considered a downgrade over 23 because 22 is the LTS and the odd number (23) is not considered stable AKA the beta

In the snapshot I see windows, do you also get a leak on linux?

It might be a memory leak in node 23 with windows

@kingyue737
Copy link
Contributor

@Tofandel Not only v23, the latest v22.12.0 on Windows also has this issue. That’s why I propose a “downgrade”

@Tofandel
Copy link

Tofandel commented Dec 13, 2024

I see, I thought you meant only downgrade from node v23

In this case having a look into the commit history of node to find the culprit might be a good idea

Especially around those
[c17601557b] - fs: prevent unwanted dependencyOwners removal (Carlos Espa) nodejs/node#55565 (the most likely culprit given that this seems to be a file watcher issue)
[5357338b8e] - fs: use wstring on Windows paths (jazelly) nodejs/node#55171

If you know how, maybe try to checkout node in v22.12.0 and start reverting commit, build and start nuxt with the built node until you see the memory leak gone so we can identify the culprit

@Tofandel
Copy link

I also see an increase of 200Mb of memory use on every nuxt reload in node 22.12.0 in linux but not in 22.11.0

So node issue and not specific to windows. I'll try to see without the commit I suspect to see if this is the culprit

@mjlehrke
Copy link
Author

I can confirm the the bug is introduced between node v22.11.0 and v22.12.0. 22.11.0 has stable memory usage with devtools on while 22.12.0 has the memory leak.

@mjlehrke
Copy link
Author

mjlehrke commented Dec 15, 2024

This was a fun learning experience. I methodically built node commits between v22.11.0 and v22.12.0 and tested them to narrow in on the culprit.

I identified nodejs/node@41c50bc as the commit that introduces the memory leak. Unfortunately, that commit introduces a lot of changes so hard to narrow in.

This is 100% out of my expertise, if someone else wants to confirm this is the commit.

I wonder if this might be related, it is already fixed in the upstream libuv nodejs/node#52769
nodejs/node#56243

@Tofandel
Copy link

If you suppress the noise in the update which are tests doc change and platform specific changes. The changes become minimal.

I find the changes in uv__stream_recv_cmsg a bit suspicious, they are tingling my developer senses, can you try to test with just the changes in this function reverted?

We'll need to open an issue in the libuv repo once we find the cause

@mjlehrke
Copy link
Author

I reverted the changes to uv__stream_recv_cmsg in deps/uv/src/unix/stream.c but it still leaked.

@mjlehrke
Copy link
Author

mjlehrke commented Dec 16, 2024

Got it!

The minor logic update in deps/uv/src/win/fs-event.c introduces the memory leak. When I revert this change it no longer leaks memory. I do not know C, so this is above my pay grade for next steps.

https://github.com/nodejs/node/pull/55114/files#diff-f0e94526ff1778190e753978e54f805b05a9c1c2851f261375ffb01b4935918b

@Tofandel
Copy link

Tofandel commented Dec 16, 2024

Seems it might already have been fixed libuv/libuv#4647

Edit: maybe different memory leak, reported on an earlier version. But try the fix anyways as maybe the fix to the function now makes it so it goes in where the other memory leak is

@Tofandel
Copy link

Tofandel commented Dec 16, 2024

@mjlehrke Thanks for investigating and finding the cause of the leak

A PR has been merged that should fix it
libuv/libuv#4656

It might take a month or so until node picks it up though, so you may want to build node yourself with the patch for now and use that built version to avoid the memory leak

Though I will have to investigate why it's leaking on linux as well then as there must be yet a different memory leak somewhere

@mjlehrke
Copy link
Author

Sounds good, closing this in anticipation of the upstream fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants