Kapacitor doesn't always create subscriptions when brought up with influxdb #187
Milestone
Comments
|
@jacobcase I think the real problem is the lack of retrying to connect to InfluxDB on startup. We should add some kind of retry with back off mechanism and only fail after a longer time period. As for the missing SUBSCRIPTIONS my only guess is that Kapacitor was able to connect to InfluxDB before it was ready to serve requests and got some kind of error that was ignored. I will try to reproduce locally and see if I can fix the root cause. Thanks for the detailed report. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
OS Host: mint 17
Docker Base image: Ubuntu 14.04
Docker version: 1.9.1
Docker compose version: 1.5.2
Influxdb version: 0.9.6.1
Kapacitor version: 0.10.0
From: .deb packages
I'm setting up an experiment using docker-compose, and I'm having an issue where Kapacitor requires restart manually for it to create subscriptions to influxdb. I'll try and step through the process to replicate.
To start, I'm using docker-compose with a few services, but the focus is on influxdb and kapacitor. When I bring up new containers (old ones completely removed), they come up at the same time. I have the kapacitor container automatically restarting when it fails as it will when it can't connect to influxdb for the first few tries as influxdb starts. However, after a few tries, it no longer generates the error message of
connection refusedwhen connecting to influxdb and stops restarting. At this point I would suspect it was able to reach influxdb and create subscriptions, but when I check influxdb withshow subscriptionsin the cli, I don't have any results.To fix the issue, I then manually restart kapacitor with
docker-compose restart kapacitor, and when I check the subscriptions, it is now shown.I'll provide some logs with debugging enabled. Viewing the logs of kapacitor, I can see connection refused the first few tries that looks like this.
After docker automatically restarts it about 5 times, which happens rapidly, I no longer see the connection refused error at the end and the container continues to run.
And that's where the logs stop, but there are not subscriptions in influxdb. At this point I restart kapacitor manually, and the subscriptions show up.
Let me know if I can provide any other details, or if I just screwed up somewhere...
Here is the project I'm doing it in, for reference: https://github.com/jacobcase/docker-files/tree/master/compose/tick_service_check
You MUST use --x-networking when bringing it up with compose for the containers to be on the same bridge have have others in hosts.
Note: It changes a lot as I'm working on this. Commit at this time is 749ed7fe5fe4d171544f3d5152eababaf51a7fbe
The text was updated successfully, but these errors were encountered: