-
Notifications
You must be signed in to change notification settings - Fork 227
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Crashlooping on start #479
Comments
I’ve never seen this and there’s no info which gives me any more info. Can you start Faktory manually, outside of k8s, on the same machine? |
Because this is running on a kubernetes cluster and we've locked down access to the nodes directly this is not something I can easily test. Additionally it has been deleted and created a number of times which causes it to schedule to 1 of the 7 nodes in this cluster and it hasn't worked on any of them. The only difference i see on the node pools between the cluster it does work in and this one is kernal and amazon linux version. Cluster it works in
Cluser it doesn't work in
I'll keep double checking configuration between the two but is it possible the same license file cannot be used for I think we ordered 2 licenses but the e-mail i was sent from my procurement team only had 1 license in it. And i noticed the log line says 200 connections when i expected each to only be 100. |
You can share a single license between two servers, giving each 100 connections. I believe the Licensing wiki page covers that. |
And yes, the same license can be used in staging and production. |
@mperham what do you mean by |
https://github.com/contribsys/faktory/wiki/Licensing#staging "Faktory has the notion of environment -- you start Faktory in development, staging or production and only pay for production servers." You have a total of 200 connections in production. You can have two production servers sharing the same license, each with FAKTORY_CONN_MAX=100. You can have unlimited staging servers but each staging server is limited to 100 connections automatically when you use |
@mperham
and i get these logs
and then the pod shuts down. I've also tried giving it the |
Sounds like you have a good clue. I have to assume there's some annoyance in your licensing configuration preventing startup, like unwanted whitespace or something. |
@mperham I FOUND IT. It totally is the statsd settings. I switched it back to using the |
That still doesn't make sense to me, Datadog should be more resilient than that.The Statsd protocol uses UDP which is connection-less. The existence of the remote side should be irrelevant. |
@mperham - well i've confirmed it. Adding back the Here is my config
So it seems a few things:
I've replicated this both on 1.8.0 and 1.9.0 using the enterprise image |
Yep, I can reproduce this. Fix coming... |
I've found and fixed the issue with Faktory stopping with no error message.
|
I suspect your underlying issue is that the datadog statsd host is not DNS resolvable. Datadog resolves the IP address when it creates the Statsd connection. |
Thanks @mperham - I assume this will be in 1.10 ? Also what about the |
It will be in 1.9.1 or 1.10. A license is always required outside of the development environment. |
Are you using an old version? Yes by 1 release
Have you checked the changelogs to see if your issue has been fixed in a later version? Yes
When starting up the faktory server we get a constant crashlooping in kubernetes and the logs don't indicate why when bumpped to a debug level.
This is for our enterprise license faktory we are trying to stand up.
Here are the logs it prints out.
Additionally kubernetes reports the process as exiting fully and cleanly and with exit code 0
The text was updated successfully, but these errors were encountered: