Memory allocation past what is available in the container

[svennam92]

I've recently started investigating a memory issue with Node.js apps running in Cloud Foundry environments (I'm using CFv210). As an example, if I push a Node.js app with a mem leak and a 512MB memory limit, the Node.js V8 engine tries to allocate more and more memory until it passes that memory limit and the application crashes. The behavior I expect to see is that it will stop trying to allocate more memory when it reaches the limit, and instead try to GC more aggressively (and then crash at a later time). Has anyone else seen this behavior?

By default, on 64 bit machines, the Node.js v8 engine has a 1GB heap limit, so I can see why the engine tries to allocate more than is really available. There should be some way to prevent the Node.js v8 engine from trying to allocate more than is available. In Java, you can use JVM opts to set heap limits, maybe something similar?

I did find one thing that might help, --max-old-space-size. But... has any one done any investigation as to how to set that space size? "--max-old-space-size" only accounts for the v8 engine's heap, not the buffers or other processes. For example, should that limit be set to 50% of the memory_limit? 75%? Maybe that's something the Node.js buildpack should set as a reasonable default?

There is a separate issue that might be related to this. When I run 'free' or 'top' as a shell command from within the container spun up for my application, I am seeing "32gb" total. When I run commands like "os.totalmem()" from within Node.js, I'm also seeing 32gb. Although this is a problem that probably cannot be avoided due to use of cgroups, I wonder if this contributes to the memory issue.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/101495382.

[jtarchie]

@svennam92 there is no guarantee that the GC would invoke one it hit the memory limit. The memory leak you describe could be causing garbage that the GC needs to hold on to even beyond the limit.

You can do two things currently to support larger processes:

1. Allocate more memory to the droplet that you are deploying using the --memory option with cf push.
2. Define your start command with the --max-old-space-size defined.

The options are available so you can easily tweak it. The buildpack's responsibility is to compile your app with dependencies. It could make a best guess for the runtime of your application and memory limits, but we'd need better metrics to verify we were doing it right.

As for the separate issue. This sounds like something that might need to be filed with warden or garden. Maybe best to bring it up on the cf-dev mailing list.

Kind Regards,

JT

[flavorjones]

@svennam92 - Thanks for asking these questions.

I replied to your email to cf-dev@ on July 29th, but I don't know if you ever saw it, as I didn't receive a response. So here's my initial response:

Thanks for asking these questions. The buildpacks team, who currently maintains the nodejs-buildpack, is totally open to improving the node.js developer experience.

I'd love to hear about anyone's experience managing the total heap size within the node.js interpreter. If you have played with this, let us know, and we'd be happy to work with you on how it might work in conjunction with container memory limits.

Chris Piraino responded to your "separate issue" question, as well:

Running free/top from inside the container is going to report the VM's memory/cpu statistics in which the container is running. The correct stats are located in the appropriate cgroups filesystem, which is where Cloud Foundry pulls stats from when using the CLI.

From inside the container it is actually very hard to figure out how much memory the cgroup is using (see this article for more information).

and he linked to http://fabiokung.com/2014/03/13/memory-inside-linux-containers/

Do you have any additional ideas on how to manage memory in v8? This seems to be beyond the scope of the buildpacks team mandate; though if there's a way to reliably do it, I'm happy to consider pulling it into the nodejs-buildpack.

[svennam92]

Thanks for your feedback! -- I was hoping to reach a different audience that may have known more about how heap settings in Node.js work. The Java buildpack does it with JVM opts, which led me to believe that maybe the Node.js buildpack should do it as well; I guess Node.js doesn't quite have a clear-cut way of setting a heap limit.

This may eventually be an issue that the buildpack needs to address. During development, Node.js apps are run in large containers, where memory limits are almost always greater than the default limit for the v8 engine. People really only start running into this issue in low-memory environments such as those often spun up by CF.

Feel free to close this issue, I'll keep investigating.

[jotielim]

@jtarchie If I want to start the app with the memory limit, can I do this?
node --max_old_space_size=$MEMORY_LIMIT app.js