Skip to content
This repository has been archived by the owner on Jan 31, 2018. It is now read-only.

Not safe for running in a node cluster (like PM2) #36

Open
blak3r2 opened this issue Aug 18, 2016 · 5 comments
Open

Not safe for running in a node cluster (like PM2) #36

blak3r2 opened this issue Aug 18, 2016 · 5 comments

Comments

@blak3r2
Copy link

blak3r2 commented Aug 18, 2016

Learned the hardway today that this will crash PM2.

There is a bug in node related to UDP and I believe this is the issue.
nodejs/node-v0.x-archive#9261

I do not know why this statsd client causes it but the node-statsd one does not.
I wanted to post here before I forget in case it helps others.

@Raynos
Copy link

Raynos commented Aug 18, 2016

Interesting. We've not seen this edge case with node cluster & UDP itself.

Sorry about that, you should look into using a different statsd client or not using cluster.

At Uber, we do not use cluster.

@Raynos
Copy link

Raynos commented Aug 18, 2016

I updated the README with a caveat, thanks for pointing this out.

@blak3r2
Copy link
Author

blak3r2 commented Aug 19, 2016

Just to clarify... their is a way to use UDP... as npm node-statsd doesn't seem to cause this issue (at least anymore). I will dig in deeper when i get time and report back.

Thanks for putting this buffered statsd library together. I definitely had less packets dropped when i was running it (unclustered).

@Raynos
Copy link

Raynos commented Aug 19, 2016

@blak3r2 looks like the issue is that node-statsd creates a single UDP socket. Where as uber-statsd-client can create multiple UDP sockets, it will actually de-allocate the UDP socket on inactivity and re-allocate it lazily once needed. ( where inactivity, by default is 1000 milliseconds of no UDP writes )

@blak3r2 I suspect that if you use a module like process-reporter ( https://github.com/rf/process-reporter ) which continiously sends stats in an interval, you can "avoid" the issue.

You could also tweak the socket_timeout ( https://github.com/uber/node-statsd-client#optionssocket_timeout ), setting it to 10 minutes or 10 hours should avoid the multiple UDP sockets issue.

@blak3r2
Copy link
Author

blak3r2 commented Aug 22, 2016

Hi @Raynos I just wanted to thank you for your reply!

I am not sure that is quite it because we would have been sending a flood of packets.
I am going to setup some further tests this week and will report back.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants