Memory leak in server?

[mrdavidlaing]

I'm running Kibana 4.2.beta2 as a Cloud Foundry app with 384MB of memory assigned to each app instance container.

I'm querying against a fairly small elasticsearch cluster, at a rate of about 20 requests / minute - basically very little load.

There appears be be some form of memory leak that causes the server side nodejs portion of Kibana to gradually consume more and more memory until it exhausts the 384M of memory allocated to it in the container.

Couple of follow on questions:

1. What is the correct amount of RAM to allocate to Kibana4 server?
2. Is this a known issue? Are other people running into something similar?
3. What information / test case if most helpful for me to provide to help debug further?

Thanks!

[nestorsamelo]

I am seeing the same. Memory consumption keeps increasing. I had to restart it at around 1.2 GB.

[rashidkpc]

What exactly occurs? Does the oom_killer slay the process? While its possible there could be a memory leak, the server doesn't do all that much. It possible this is just cache.

[nestorsamelo]

I have one instance running for 4 days and 18 hours:

root@host# ps -p [pid] -o etime=
 4-18:09:40

top shows:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
[pid] root      20   0 1900m 1.1g 8316 S  0.0 27.6  18:31.08 node

proc memory statistics:

root@host# cat /proc/[pid]/statm
486466 279120 2079 4044 0 477276 0

According to the documentation:

    /proc/[pid]/statm
              Provides information about memory usage, measured in pages.
              The columns are:
                  size       (1) total program size (same as VmSize in /proc/[pid]/status)
                  resident   (2) resident set size (same as VmRSS in /proc/[pid]/status)
                  share      (3) shared pages (i.e., backed by a file)
                  text       (4) text (code)
                  lib        (5) library (unused in Linux 2.6)
                  data       (6) data + stack
                  dt         (7) dirty pages (unused in Linux 2.6)

In my case, it didn't get to a point where oom_killer had to step in, as it is running in a 4 GB VM.

[Bennell]

@rashidkpc: We can also confirm the memory leak.

We started the dev server locally with esvm also running. No extra config, no custom dashboards, clean slate from freshly cloned kibana repo.

We monitored the memory usage of the process for about half an hour (we will let it run some more) and the memory usage is slowly creeping up even without running the kibana UI. It leaks a lot faster if I keep refreshing the UI. So far the amount of memory used by the process has doubled since starting it.

I'm not sure what other details we could share. We have seen the memory leak happening on OSX and Linux, nodejs 0.12.7.

[rashidkpc]

Growing memory isn't necessarily a sign of a leak.

The VM will only garbage collect at signs of memory pressure. You can confirm this by taking a heapdump of the process, which will force a garbage collection. If you then wait some time, or perform some actions, then take another heap dump, you can compare them.

These were taken ~ 10 minutes apart, as you can see, the memory actually goes down. Unless the process is dying I wouldn't worry about the growing memory. It will be freed as needed.

[cromega]

@rashidkpc I can also confirm that freeing up memory doesn't happen. The app container has 384MB memory assigned which slowly keeps filling up until either the CloudFoundry runtime kills the container and starts a new one, or, and this happens more often, the kibana server just silently stops serving requests, breaking the UI.

I will try running the app with significantly more memory and see what happens.

[rashidkpc]

@cromega did 4.1.x behave differently? 4.2 switched to node 0.12 which is slightly more relaxed about GCing. I'm not familiar with how cloud foundry works unfortunately.

[cromega]

@rashidkpc: I am running 4.2-beta2, before that I used a 4.2-snapshot. I have no experience with 4.1. Thanks for the info, I will play around with different node versions.

[nestorsamelo]

@rashidkpc: The memory footprint of 4.1.2 (103 MB) is smaller than 4.2.0-beta2 (473 MB).

top for 4.1.2 (uptime 1-03:33:45):

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1716 root      20   0  998m 103m 7136 S  0.0  2.6   0:02.31 node

top for 4.2.0-beta2 (uptime 6-00:28:29):

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
20998 root      20   0 1280m 473m 8316 S  0.3 12.0  23:33.94 node

This last one is the same instance as my previous comment, so (resident) memory consumption has indeed decreased as you suggested above, from 1.1 GB to 473 MB.

[PSiAU]

I'm not running Kibana as a Cloud Foundry app but am seeing the same issue 4.2.0 (final release). Kibana is using 646m of memory. Surely Kibana shouldn't use that much memory?

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND                                                                                                                  
11150 root      20   0 1183m 646m 8720 S   0.0 32.1   1:01.90 node

Update:

a 8 hours later, it has grown to this:

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND                                                                                         
11150 root      20   0 1389m 854m 5264 S   0.0 42.5   2:08.03 node

Running on Debian 7.9

[PSiAU]

As a test, I restarted Kibana and left it running on the server overnight without logging into it at all on the browser. Memory usage still grows nonetheless.

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND                                                                               
18161 root      20   0 1426m 799m 4324 S   0.3 39.7   1:17.29 node 

[nfnty]

I'm having the same problem running in a Docker container with 4.2.0 and Elasticsearch 2.0.0. No problem with 4.1.2. 1GiB OOM killed after approximately 14 hours.

nfnty/arch-kibana:latest with arguments

[ricardojdsilva87]

Hello,

Same problem here.
After upgrading to kibana 4.2.0 (ES and logstash upgrade too), I had some problem with ES crashing after a few hours, the OOM killer was invoked. I tried to lower the ES_HEAP to 25% of available RAM yesterday, but today the memory of the server was too low again (50MB remained)

After taking a better look to the top command, I've noticed that the node process was taking about 40%of the available RAM, meanwhile ES and logstash remained stable at ~40% and 7.5% (check below)

On the previous version of kibana (4.1.2) wit ES 1.7.2 and Logstash 1.5.4 I had no problem, the processes were running with no problems for 1 month until the upgrade. ES was configured with the recommended settings:

https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html

I'm going to leave it running for the rest of the day to check on the memory usage for each process, I'll update the thread later

Thanks for the help

[ricardojdsilva87]

´Hello,

Just a quick update, the kibana 4.2.0 is running for about 2 hours and it seems that the memory usage of node increased 10% of the total RAM

Meanwhile ES and logstash remained pratically the same.

Best Regards

[ricardojdsilva87]

Hello

After leaving the services running all night today morning I have this scenario:

The node memory usage climbed to 40% of the server RAM.
In a previous comment @rashidkpc said that the kibana server is using now the node v0.12. I'm going to try and investigate if there are any issues with this versions, maybe try a new node version.

Best regards

[peterholak]

Same here, see the attached picture, kibana was started at the 22:00 mark (no other change on the server happened since then) and I had it opened in browser for only about 2 mintues in total. Kibana 4.2.0, ES 2.0, Ubuntu 15.10.

[rvrignaud]

Hello,
Same problem here. Kibana 4.2.0 deployed on a container (using Ubuntu 15.04) with Elasticsearch 2.0. We have 0 usage of Kibana (no one is using it) however we see a linear growing usage of memory.
Here is the memory usage:

The first part of the graph show the behaviour with Kibana 4.1. The top green line is the memory threshold of the container. Each time Kibana reach the memory limit, it gets an OOM. As we can see on the graph, we have increased a few time the memory limit up to 1,5GB with no change of behaviour.

[mrdavidlaing]

@rashidkpc - You've labeled this not reproducible; how can we team together to get a test case you can use that reproduces the behaviour?

What development / test environment do you guys use - perhaps those of us on this ticket could put together something that replicates the error in your environment?

[spalger]

@mrdavidlaing we primarily use the chrome dev tools for memory profiling. I'm not sure how Rashid was collecting his snapshots (perhaps by signalling the node process) but I was doing the same by running the server with iron-node.

npm install -g iron-node
cd ~/dev/kibana
iron-node ./src/cli

With iron-node you can use the memory profiling tools to collect snapshots and see the items that are not being garbage collected, or collect memory over time and cause constant garbage collection. These types of metrics are a lot more meaningful than the total process memory level since v8 will only do light garbage collection until the it detects serious memory pressure.

[PSiAU]

Curiosity got the better of me so I left the node process growing just to see what would happen. It started consuming just about all free memory on the instance (not cache memory but Active memory). IO Wait hit 100% as the OS tried to deal with the low memory situation. I had to restart node before the OOM killer kicked in.

[jraby]

Just for the record, we're running Kibana 4.2.0 on ubuntu 12.04 and we see the same behaviour.

At startup, the kibana node process takes ~100mb (resident size seen from ps aux) and it grows steadily until the OOM killer kills something: Kibana or ES if we're not lucky.

Our usage pattern is very light, I'm the only user of the kibana installation at the moment and I don't have any dashboard open.

I've enabled verbose logging, it logs the memory usage every 5 seconds, I'll post it later.
For the moment, kibana reported using 70mb on startup and it now reports 107mb after 30 minutes.

RSS size as reported from ps aux started at ~100mb and it is now at 170mb.

[mrdavidlaing]

@jraby, @PSiAU - Out of interest, are you guys running Kibana inside a container, on a Ubuntu VM, or on baremetal?

[PSiAU]

I'm running it on Debian 7.9 (64-bit), in an EC2 instance.

[fpighi]

Having the same issue reported by other people here with Kibana 4.2.0. Memory usage on the server is going up with a linear pattern.
We are using kibana inside a docker container running on a Amazon Linux AMI in EC2.

[jraby]

Ubuntu 12.04 on ec2.
On Nov 10, 2015 7:10 AM, "Francesco Pighi" notifications@github.com wrote:

Having the same issue reported by other people here with Kibana 4.2.0.
Memory usage on the server is going up with a linear pattern.
We are using kibana inside a docker container running on a Amazon Linux
AMI in EC2.

—
Reply to this email directly or view it on GitHub
#5170 (comment).

[danrimal]

I have OOM killer log (VM had 2GB RAM):

[9827667.988558] Out of memory in UB 5517: OOM killed process 8709 (node) score 0 vm:5478276kB, rss:1712056kB, swap:1977324kB

Now, server has 6GB RAM and kibana 4.2.0 with sense and marvel app goes up to 6GB RAM after 24hours.

[dngferreira]

Using the --max-old-space-size=150 seems to fix the problem using docker.
Just as a safe side I have a 300mb limit on the container but it hasn't been triggered yet.

[PSiAU]

Looks like --max-old-space-size is working nicely, thanks @mrdavidlaing
Note: I was originally adding --max-old-space-size to /etc/init.d/kibana in the args parameter instead of /opt/kibana/bin/kibana. You need to add it to /opt/kibana/bin/kibana or it will not work.

[thomasmarkus]

I'm also happy to report that the PR from @mrdavidlaing also fixes the memory leak for us. 👍

[FlorinAndrei]

Can we get this fix into a release please? I've noticed that bin/kibana in the repo still doesn't limit memory usage.

[dbason]

We've also recently run into a similar issue with Kibana 4.3.

Edit: Will try the max-old-space-size fix.

[dyakhnov]

Same issue here: Kibana 4.3 in Docker 1.6.2 on AWS EC2
It works like Cookie Monster -- nom, nom, nom...

[ricardclau]

Same here, in my case I had a t2.micro instance running both Kibana 4.3 and ElasticSearch 2.1

With this particular setup ElasticSearch ended up crashing (presumably by the OOM killer) but it seems that the issue was with the nodejs application

@mrdavidlaing saved the day for me :)

Any plans for a 4.3.1 with this patch any soon? 👍

[FlorinAndrei]

--max-old-space-size=250 appears to fix it for me, process size seems to hover around 300 MB, give or take a few, no more OOM. Using Kibana 4.2.1

[mrdavidlaing]

I'm closing this because #5451 addresses this issue by allowing you to explicitly set node's GC settings via a NODE_OPTIONS environment variable. Should hopefully make the 4.4 release.

FWIW, in a low memory environment you should be able to prevent OOM errors by starting using:

NODE_OPTIONS="--max-old-space-size=200" bin/kibana

Adjusting 200 such that 100MB + 200MB < the the memory in MB available in your container.

[rashidkpc]

Adding for posterity: nodejs/node#2683

[austinchambers]

Given this issue was reproducible, perhaps the "not reproducible" tag should be removed?

[ageis]

See latest comment in #5451, the systemd service is broken due to this. Who thought > was a valid comment character?

[tylersmalley]

@ageis, not us. Make sure you're using our APT repository.