"Unexpected EOF Error" when using Snap with influxdb container

[nichelle-hall]

I am currently running Snap version v0.14.0-beta and Docker version 1.12.0-rc3 (for both client and server). I am also using the most recent update of the influxdb plugin that was merged after resolving issue #61 last week. When I load my plugins and run my Snap task, both snapd and my containers stall (meaning that they are running but not collecting telemetry). This said, I am confident that this issue is related to Snap's communication with the influxdb container because I have no errors when running the influxdb daemon (influxd) with Snap. It is only when I force quit out of snapd using ctrl+c that I get the following error. I have included my task file and docker commands to aid troubleshooting.

Error

ERRO[0125] error with publisher job                      _module=scheduler-job block=run content-type=snap.gob error=unexpected EOF job-type=publisher plugin-config=map[host:{127.0.0.1} password:{admin} port:{8086} user:{admin} database:{snap}] plugin-name=influx plugin-version=-1
WARN[0125] Publish job failed                            _block=submit-publish-job _module=scheduler-workflow parent-node-type=processor publish-name=influx publish-version=-1 task-id=5776df3e-1148-420e-9272-325c9e9e62d7 task-name=Task-5776df3e-1148-420e-9272-325c9e9e62d7
DEBU[0125] Batch submission complete                     _block=work-jobs _module=scheduler-workflow count-process-nodes=0 count-publish-nodes=1 parent-node-type=processor task-id=5776df3e-1148-420e-9272-325c9e9e62d7 task-name=Task-5776df3e-1148-420e-9272-325c9e9e62d7
DEBU[0125] Batch submission complete                     _block=work-jobs _module=scheduler-workflow count-process-nodes=1 count-publish-nodes=0 parent-node-type=collector task-id=5776df3e-1148-420e-9272-325c9e9e62d7 task-name=Task-5776df3e-1148-420e-9272-325c9e9e62d7
WARN[0125] Task failed                                   _block=spin _module=scheduler-task consecutive failure limit=10 consecutive failures=1 error=unexpected EOF task-id=5776df3e-1148-420e-9272-325c9e9e62d7 task-name=Task-5776df3e-1148-420e-9272-325c9e9e62d7

Snap Task

{
    "version": 1,
    "schedule": {
        "type": "simple",
        "interval": "1s"
    },
    "workflow": {
        "collect": {
            "metrics": {
                "/intel/psutil/load/load1": {},
                "/intel/psutil/load/load5": {},
                "/intel/psutil/load/load15": {},
                "/intel/psutil/vm/available": {},
                "/intel/psutil/vm/free": {},
                "/intel/psutil/vm/used": {}
            },
            "config": {
                "/intel/mock": {
                    "password": "secret",
                    "user": "root"
                }
            },
            "process": [
                {
                    "plugin_name": "passthru",
                    "process": null,
                    "publish": [
                        {
                            "plugin_name": "influx",
                            "config": {
                                "host": "127.0.0.1",
                                "port": 8086,
                                "database": "snap",
                                "user": "admin",
                                "password" : "admin"
                            }
                        }
                    ],
                    "config": null
                }
            ],
            "publish": null
        }
    }
}

Docker commands

docker run -d -p 8083:8083 -p 8086:8086 --name="influxdb"  influxdb
docker run -d -p 3000:3000 --link=influxdb  --name grafana grafana/grafana

[jcooklin]

@NichelleBot can you look in the influxdb.stderr log in /tmp for an error
(possibly a stack trace).

On Thu, Jul 7, 2016 at 5:07 PM NichelleBot notifications@github.com wrote:

I am currently running Snap version v0.14.0-beta and Docker version
1.12.0-rc3 (for both client and server). I am also using the most recent
update of the influxdb plugin that was merged after resolving issue #61
#61 last week. When I load my
plugins and run my Snap task, both snapd and my containers stall (meaning
that they are running but not collecting telemetry). This said, I am
confident that this issue is related to Snap's communication with the
influxdb container because I have no errors when running the influxdb
daemon (influxd) with Snap. It is only when I force quit out of snapd using
ctrl+c that I get the following error. I have included my task file and
docker commands to aid troubleshooting.

Error

ERRO[0125] error with publisher job _module=scheduler-job block=run content-type=snap.gob error=unexpected EOF job-type=publisher plugin-config=map[host:{127.0.0.1} password:{admin} port:{8086} user:{admin} database:{snap}] plugin-name=influx plugin-version=-1
WARN[0125] Publish job failed _block=submit-publish-job _module=scheduler-workflow parent-node-type=processor publish-name=influx publish-version=-1 task-id=5776df3e-1148-420e-9272-325c9e9e62d7 task-name=Task-5776df3e-1148-420e-9272-325c9e9e62d7
DEBU[0125] Batch submission complete _block=work-jobs _module=scheduler-workflow count-process-nodes=0 count-publish-nodes=1 parent-node-type=processor task-id=5776df3e-1148-420e-9272-325c9e9e62d7 task-name=Task-5776df3e-1148-420e-9272-325c9e9e62d7
DEBU[0125] Batch submission complete _block=work-jobs _module=scheduler-workflow count-process-nodes=1 count-publish-nodes=0 parent-node-type=collector task-id=5776df3e-1148-420e-9272-325c9e9e62d7 task-name=Task-5776df3e-1148-420e-9272-325c9e9e62d7
WARN[0125] Task failed _block=spin _module=scheduler-task consecutive failure limit=10 consecutive failures=1 error=unexpected EOF task-id=5776df3e-1148-420e-9272-325c9e9e62d7 task-name=Task-5776df3e-1148-420e-9272-325c9e9e62d7

Snap Task

{
"version": 1,
"schedule": {
"type": "simple",
"interval": "1s"
},
"workflow": {
"collect": {
"metrics": {
"/intel/psutil/load/load1": {},
"/intel/psutil/load/load5": {},
"/intel/psutil/load/load15": {},
"/intel/psutil/vm/available": {},
"/intel/psutil/vm/free": {},
"/intel/psutil/vm/used": {}
},
"config": {
"/intel/mock": {
"password": "secret",
"user": "root"
}
},
"process": [
{
"plugin_name": "passthru",
"process": null,
"publish": [
{
"plugin_name": "influx",
"config": {
"host": "127.0.0.1",
"port": 8086,
"database": "snap",
"user": "admin",
"password" : "admin"
}
}
],
"config": null
}
],
"publish": null
}
}
}

Docker commands

docker run -d -p 8083:8083 -p 8086:8086 --name="influxdb" influxdb
docker run -d -p 3000:3000 --link=influxdb --name grafana grafana/grafana

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#1054, or mute the thread
https://github.com/notifications/unsubscribe/AA0q-MGA_maEd2TYBoGG0gOxzUdtuPMdks5qTZSvgaJpZM4JHnZM
.

[nichelle-hall]

Here is what was logged.

2016/07/08 09:11:00 2016/07/08 09:11:00 method Decode reply type not a pointer: interface {}
2016/07/08 09:11:00 2016/07/08 09:11:00 method Decrypt has wrong number of ins: 2
2016/07/08 09:11:00 2016/07/08 09:11:00 method DecryptKey has wrong number of ins: 2
2016/07/08 09:11:00 2016/07/08 09:11:00 method Encode has wrong number of ins: 2
2016/07/08 09:11:00 2016/07/08 09:11:00 method Encrypt has wrong number of ins: 2
2016/07/08 09:11:00 2016/07/08 09:11:00 method EncryptKey has wrong number of ins: 1
2016/07/08 09:11:00 2016/07/08 09:11:00 method KillChan has wrong number of ins: 1
2016/07/08 09:11:00 2016/07/08 09:11:00 method ListenAddress has wrong number of ins: 1
2016/07/08 09:11:00 2016/07/08 09:11:00 method ListenPort has wrong number of ins: 1
2016/07/08 09:11:00 2016/07/08 09:11:00 method Logger has wrong number of ins: 1
2016/07/08 09:11:00 2016/07/08 09:11:00 method ResetHeartbeat has wrong number of ins: 1
2016/07/08 09:11:00 2016/07/08 09:11:00 method SetEncrypter has wrong number of ins: 2
2016/07/08 09:11:00 2016/07/08 09:11:00 method SetListenAddress has wrong number of ins: 2
2016/07/08 09:11:00 2016/07/08 09:11:00 method Token has wrong number of ins: 1
'''

[marcin-krolik]

@NichelleBot I have just repeated your experiment with snap-plugin-collector-psutil, snap-plugin-publisher-influxdb and snap-processor-passthru. I used task manifest you provided. InfluxDB running with docker, started with command you mentioned. I cannot reproduce your issue.

Docker version:
Client:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:47:50 2016
OS/Arch: linux/amd64

Server:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:47:50 2016
OS/Arch: linux/amd64

Snap:
93eb909

 $ curl -GET 'http://localhost:8086/query?pretty=true' --data-urlencode "db=snap" --data-urlencode 'q=SELECT last(value) FROM "intel/psutil/load/load15"'
{
    "results": [
        {
            "series": [
                {
                    "name": "intel/psutil/load/load15",
                    "columns": [
                        "time",
                        "last"
                    ],
                    "values": [
                        [
                            "2016-07-11T19:43:50Z",
                            0.28
                        ]
                    ]
                }
            ]
        }
    ]
}                                                                                     

Can you check if you are able to add some data to your db and query it from host?

[nichelle-hall]

I ran the same docker protocol on a vm with ubuntu (I am assuming that you are not using OS X) and was not able to replicate the issue either. I think the error has to do with the new docker for mac beta. Thanks for all your help!

[andrzej-k]

Closing this issue as non reproducible.

[nichelle-hall]

I will update the example with a note about the complications with Docker for Mac.

[mbbroberg]

👍 thanks @NichelleBot! Great work figuring this tough one out. Having it documented would be awesome. It may be worth sending that feedback to the Docker on Mac Beta team too if you have a chance.