Docker changing default syslog format causes nomad logs not work as expected

[wuub]

Nomad version

v0.4.0

Operating system and Environment details

$ docker version
Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 22:00:43 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 22:00:43 2016
 OS/Arch:      linux/amd64

Issue

When using newer docker versions, mismatch in syslog formats causes all logs to have their prefixes lstriped as shown below (moby/moby#20121)

28:59 docker/7a0ecb6376d5[1332]: [2016-07-04 11:28:59,523: INFO/gunicorn.error] Listening at: http://0.0.0.0:5000 (1)
28:59 docker/7a0ecb6376d5[1332]: [2016-07-04 11:28:59,523: INFO/gunicorn.error] Using worker: sync
28:59 docker/7a0ecb6376d5[1332]: [2016-07-04 11:28:59,526: INFO/gunicorn.error] Booting worker with pid: 9
29:20 docker/7a0ecb6376d5[1332]: [2016-07-04 11:29:20,474: INFO/gunicorn.error] Handling signal: term
29:20 docker/7a0ecb6376d5[1332]: [2016-07-04 11:29:20,475: INFO/gunicorn.error] Worker exiting (pid: 9)
29:21 docker/7a0ecb6376d5[1332]: [2016-07-04 11:29:21,015: INFO/gunicorn.error] Shutting down: Master

It also probably causing all logs to land in .stderr output (bad serverity detection?)
I'll try to setting --log-opt syslog-format=rfc3164 and let you know if this is reliable short term hotfix.

Reproduction steps

1. Launch any docker job with nomad v.0.4.0 and docker >= 1.11.0

Nomad Server logs (if appropriate)

n/a

Nomad Client logs (if appropriate)

n/a

Job file (if appropriate)

n/a

[camerondavison]

Just as an FYI. It does not look like docker actually switched to rfc5424. But does look like they changed from the DefaultFormatter to the UnixFormatter formatter.go

https://github.com/docker/docker/blob/v1.11.0/daemon/logger/syslog/syslog.go#L238

The problem with the switch to the UnixFormatter is that instead of the time looking like time.Now().Format(time.RFC3339) it looks like time.Now().Format(time.Stamp)

[wuub]

@a86c6f7964 thanks! I updated the title accordingly.

[camerondavison]

Not sure which approach y'all wanted to take, but I put in a pull request where I try to parse both types of logs.

[camerondavison]

@wuub they merged my pull request to master. any chance you can confirm that it fixes your problem?

[wuub]

@a86c6f7964 I won't be able to build and roll out my own binary this week, sorry :( but it seems that hashicorp is baking 0.4.1 as we speak and I'll definitely test that ASAP

[diptanu]

@wuub Did you get a chance to test this?

[wuub]

@diptanu I'm testing it right now

[wuub]

👍 I can confirm that current master branch works as expected with docker 1.11.2.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.