Telegraf seems to have a memory leak

[mattk42]

In my cluster running the 2.0 official release I have noticed that telegraf works its way up to 2GB of memory usage before it seemingly gets cleaned up and starts over.

[jchauncey]

Are you noticing the pod restarting a lot? There was an issue in the 0.13.x version of telegraf that caused the binary to panic when sending data to influx - influxdata/telegraf#1268

Check the pod logs and see if you see a similar erorr message.

[mattk42]

Unfortunately I actually killed off the DS, I have my own monitoring stuff in place so I shut down most of the deis-monitor components this morning.

I don't believe that the container is getting restarted though, if that was the case the container ids in the chart above would have changed.

[titilambert]

Hello, I think I got the same issue.
I suspect prometheus plugin. It seems it doesn't closing connection :/
Run on you apiservers: netstat -ntp | grep 8080 | wc -l

[jchauncey]

So we have seen this happen (especially on larger clusters) so we disabled the prometheus plugin by default in the image (although the chart turns it on). This means you will lose out on k8s metrics and container metrics. I will open an issue with telegraf and see if we can get it fixed.

[jchauncey]

See here - influxdata/telegraf#1405

[titilambert]

PR influxdata/telegraf#1406 created

[jchauncey]

So I have rebuilt the image to include latest master changes. It seems to have fixed the memory leak problem but Im not 100% on that. If you want to redeploy telegraf and check it out that would be awesome.

[titilambert]

@jchauncey testing today or tomorrow. I let you know when I get results ;)

[titilambert]

The issue is fix for me !