Kibana docker instance stops running

[nabheet]

I need help/advice in debugging this issue. I start a kibana docker instance and it runs fine for a few days. But then it stops running and I am not sure how to debug this issue.

sudo docker logs kibana has this:

{"type":"log","@timestamp":"2015-11-14T08:49:15+00:00","tags":["status","plugin:elasticsearch","info"],"pid":1,"name":"plugin:elasticsearch","state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Request Timeout after 1500ms"}
{"type":"log","@timestamp":"2015-11-14T08:50:15+00:00","tags":["status","plugin:elasticsearch","error"],"pid":1,"name":"plugin:elasticsearch","state":"red","message":"Status changed from green to red - Request Timeout after 1500ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2015-11-14T08:50:30+00:00","tags":["status","plugin:elasticsearch","info"],"pid":1,"name":"plugin:elasticsearch","state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Request Timeout after 1500ms"}
{"type":"log","@timestamp":"2015-11-14T08:51:05+00:00","tags":["status","plugin:elasticsearch","error"],"pid":1,"name":"plugin:elasticsearch","state":"red","message":"Status changed from green to red - Request Timeout after 1500ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2015-11-14T08:51:08+00:00","tags":["status","plugin:elasticsearch","info"],"pid":1,"name":"plugin:elasticsearch","state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Request Timeout after 1500ms"}
{"type":"log","@timestamp":"2015-11-14T08:52:56+00:00","tags":["status","plugin:elasticsearch","error"],"pid":1,"name":"plugin:elasticsearch","state":"red","message":"Status changed from green to red - Request Timeout after 1500ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2015-11-14T08:53:40+00:00","tags":["status","plugin:elasticsearch","info"],"pid":1,"name":"plugin:elasticsearch","state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Request Timeout after 1500ms"}
{"type":"log","@timestamp":"2015-11-14T08:55:39+00:00","tags":["status","plugin:elasticsearch","error"],"pid":1,"name":"plugin:elasticsearch","state":"red","message":"Status changed from green to red - Request Timeout after 1500ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2015-11-14T08:55:42+00:00","tags":["status","plugin:elasticsearch","info"],"pid":1,"name":"plugin:elasticsearch","state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Request Timeout after 1500ms"}
{"type":"log","@timestamp":"2015-11-14T08:59:30+00:00","tags":["status","plugin:elasticsearch","error"],"pid":1,"name":"plugin:elasticsearch","state":"red","message":"Status changed from green to red - Request Timeout after 1500ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2015-11-14T08:59:33+00:00","tags":["status","plugin:elasticsearch","info"],"pid":1,"name":"plugin:elasticsearch","state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Request Timeout after 1500ms"}
{"type":"log","@timestamp":"2015-11-14T09:00:18+00:00","tags":["status","plugin:elasticsearch","error"],"pid":1,"name":"plugin:elasticsearch","state":"red","message":"Status changed from green to red - Request Timeout after 1500ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2015-11-14T09:00:22+00:00","tags":["status","plugin:elasticsearch","info"],"pid":1,"name":"plugin:elasticsearch","state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Request Timeout after 1500ms"}
{"type":"log","@timestamp":"2015-11-14T09:04:30+00:00","tags":["status","plugin:elasticsearch","error"],"pid":1,"name":"plugin:elasticsearch","state":"red","message":"Status changed from green to red - Request Timeout after 1500ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2015-11-14T09:04:33+00:00","tags":["status","plugin:elasticsearch","info"],"pid":1,"name":"plugin:elasticsearch","state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Request Timeout after 1500ms"}
{"type":"log","@timestamp":"2015-11-14T09:08:07+00:00","tags":["status","plugin:elasticsearch","error"],"pid":1,"name":"plugin:elasticsearch","state":"red","message":"Status changed from green to red - Request Timeout after 1500ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2015-11-14T09:08:10+00:00","tags":["status","plugin:elasticsearch","info"],"pid":1,"name":"plugin:elasticsearch","state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Request Timeout after 1500ms"}
{"type":"log","@timestamp":"2015-11-14T09:09:37+00:00","tags":["status","plugin:elasticsearch","error"],"pid":1,"name":"plugin:elasticsearch","state":"red","message":"Status changed from green to red - Request Timeout after 1500ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2015-11-14T09:09:40+00:00","tags":["status","plugin:elasticsearch","info"],"pid":1,"name":"plugin:elasticsearch","state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Request Timeout after 1500ms"}
{"type":"log","@timestamp":"2015-11-14T09:10:45+00:00","tags":["status","plugin:elasticsearch","error"],"pid":1,"name":"plugin:elasticsearch","state":"red","message":"Status changed from green to red - Request Timeout after 1500ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2015-11-14T09:10:48+00:00","tags":["status","plugin:elasticsearch","info"],"pid":1,"name":"plugin:elasticsearch","state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Request Timeout after 1500ms"}
{"type":"log","@timestamp":"2015-11-14T09:13:20+00:00","tags":["status","plugin:elasticsearch","error"],"pid":1,"name":"plugin:elasticsearch","state":"red","message":"Status changed from green to red - Request Timeout after 1500ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2015-11-14T09:13:24+00:00","tags":["status","plugin:elasticsearch","info"],"pid":1,"name":"plugin:elasticsearch","state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Request Timeout after 1500ms"}
{"type":"log","@timestamp":"2015-11-16T19:01:38+00:00","tags":["status","plugin:kibana","info"],"pid":1,"name":"plugin:kibana","state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2015-11-16T19:01:38+00:00","tags":["status","plugin:elasticsearch","info"],"pid":1,"name":"plugin:elasticsearch","state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2015-11-16T19:01:38+00:00","tags":["status","plugin:kbn_vislib_vis_types","info"],"pid":1,"name":"plugin:kbn_vislib_vis_types","state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2015-11-16T19:01:38+00:00","tags":["status","plugin:markdown_vis","info"],"pid":1,"name":"plugin:markdown_vis","state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2015-11-16T19:01:38+00:00","tags":["status","plugin:metric_vis","info"],"pid":1,"name":"plugin:metric_vis","state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2015-11-16T19:01:38+00:00","tags":["status","plugin:spyModes","info"],"pid":1,"name":"plugin:spyModes","state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2015-11-16T19:01:38+00:00","tags":["status","plugin:statusPage","info"],"pid":1,"name":"plugin:statusPage","state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2015-11-16T19:01:38+00:00","tags":["status","plugin:table_vis","info"],"pid":1,"name":"plugin:table_vis","state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2015-11-16T19:01:38+00:00","tags":["listening","info"],"pid":1,"message":"Server running at http://0.0.0.0:5601"}
{"type":"log","@timestamp":"2015-11-16T19:01:38+00:00","tags":["status","plugin:elasticsearch","info"],"pid":1,"name":"plugin:elasticsearch","state":"green","message":"Status changed from yellow to green - Kibana index ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

I looked at the ElasticSearch container logs:

[2015-11-10 22:05:42,556][INFO ][node                     ] [Googam] version[2.0.0], pid[1], build[de54438/2015-10-22T08:09:48Z]
[2015-11-10 22:05:42,558][INFO ][node                     ] [Googam] initializing ...
[2015-11-10 22:05:42,660][INFO ][plugins                  ] [Googam] loaded [], sites []
[2015-11-10 22:05:42,808][INFO ][env                      ] [Googam] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vda1)]], net usable_space [32.7gb], net total_space [39.2gb], spins? [possibly], types [ext4]
[2015-11-10 22:05:46,067][INFO ][node                     ] [Googam] initialized
[2015-11-10 22:05:46,087][INFO ][node                     ] [Googam] starting ...
[2015-11-10 22:05:46,212][INFO ][transport                ] [Googam] publish_address {192.168.180.45:9300}, bound_addresses {192.168.180.45:9300}, {[fe80::42:c0ff:fea8:b42d]:9300}
[2015-11-10 22:05:46,223][INFO ][discovery                ] [Googam] elasticsearch/Z05hfVF5RfOVQ2TPL6RZRg
[2015-11-10 22:05:49,309][INFO ][cluster.service          ] [Googam] new_master {Googam}{Z05hfVF5RfOVQ2TPL6RZRg}{192.168.180.45}{192.168.180.45:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2015-11-10 22:05:49,375][INFO ][gateway                  ] [Googam] recovered [0] indices into cluster_state
[2015-11-10 22:05:49,381][INFO ][http                     ] [Googam] publish_address {192.168.180.45:9200}, bound_addresses {192.168.180.45:9200}, {[fe80::42:c0ff:fea8:b42d]:9200}
[2015-11-10 22:05:49,382][INFO ][node                     ] [Googam] started
[2015-11-10 22:06:01,463][INFO ][cluster.metadata         ] [Googam] [.kibana] creating index, cause [api], templates [], shards [1]/[1], mappings [config]
[2015-11-10 22:06:10,307][INFO ][cluster.metadata         ] [Googam] [.kibana] deleting index
[2015-11-10 22:06:10,862][INFO ][cluster.metadata         ] [Googam] [jenkins_build_output] creating index, cause [api], templates [], shards [1]/[0], mappings [build_output]
[2015-11-10 22:06:11,391][INFO ][cluster.metadata         ] [Googam] [.kibana] creating index, cause [api], templates [], shards [1]/[0], mappings [index-pattern, visualization, search, config, dashboard]
[2015-11-10 22:06:12,421][INFO ][rest.suppressed          ] /.kibana/config/4.2.0 Params: {index=.kibana, op_type=create, id=4.2.0, type=config}
[.kibana][[.kibana][0]] DocumentAlreadyExistsException[[config][4.2.0]: document already exists]
    at org.elasticsearch.index.engine.InternalEngine.innerCreateNoLock(InternalEngine.java:411)
    at org.elasticsearch.index.engine.InternalEngine.innerCreate(InternalEngine.java:369)
    at org.elasticsearch.index.engine.InternalEngine.create(InternalEngine.java:341)
    at org.elasticsearch.index.shard.IndexShard.create(IndexShard.java:517)
    at org.elasticsearch.index.engine.Engine$Create.execute(Engine.java:789)
    at org.elasticsearch.action.support.replication.TransportReplicationAction.executeIndexRequestOnPrimary(TransportReplicationAction.java:1073)
    at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:170)
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.performOnPrimary(TransportReplicationAction.java:579)
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase$1.doRun(TransportReplicationAction.java:452)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[2015-11-10 22:15:10,739][INFO ][cluster.metadata         ] [Googam] [jenkins_build_output] create_mapping [build_event]
[2015-11-10 22:23:43,953][INFO ][cluster.metadata         ] [Googam] [.kibana] deleting index
[2015-11-10 22:23:45,002][INFO ][cluster.metadata         ] [Googam] [.kibana] creating index, cause [api], templates [], shards [1]/[0], mappings [index-pattern, visualization, search, config, dashboard]
[2015-11-10 22:53:28,989][INFO ][cluster.metadata         ] [Googam] [.kibana] deleting index
[2015-11-10 22:53:30,240][INFO ][cluster.metadata         ] [Googam] [.kibana] creating index, cause [api], templates [], shards [1]/[0], mappings [index-pattern, visualization, search, config, dashboard]
[2015-11-12 16:52:36,310][INFO ][cluster.metadata         ] [Googam] [.kibana] deleting index
[2015-11-12 16:52:37,304][INFO ][cluster.metadata         ] [Googam] [.kibana] creating index, cause [api], templates [], shards [1]/[0], mappings [index-pattern, visualization, search, config, dashboard]

Basically, there is no action after 11/12. And both containers are running on the same host and the kibana instance is linked to the elasticsearch instance.

Also, here is the output for sudo docker images:

REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
jenkins_master      latest              99783f1cb836        8 days ago          768.4 MB
elasticsearch       latest              8112755253f1        2 weeks ago         345.2 MB
jenkins             latest              5da1f62066bc        2 weeks ago         707.3 MB
kibana              latest              e2d1404d4133        2 weeks ago         260.1 MB
nginx               latest              914c82c5a678        2 weeks ago         132.7 MB

I have had to start kibana instance a couple of times now. I don't think I encountered this issue in the previous version. And also I am not sure where else to look into why kibana instance stops running.

Any advice/help would be greatly appreciated.

Thanks.

[nabheet]

I just realized that there was a new image that was pushed 5 days ago. I will try that image.

[nabheet]

I think this is related to elastic/kibana#5170.

[stouf]

I have the same issue. I'm going to give a try to that NODE_OPTIONS="--max-old-space-size=200" environment variable and see how it goes.