Kubernetes daemonset can't parse logs - #0 [in_tail_container_logs] pattern not matched

[joshk132]

Describe the bug

When using the Kubernetes daemonset from here the fluentd has issues with "#0 [in_tail_container_logs] pattern not matched" followed by a long string of "////////"

To Reproduce

• Deploy Kubernetes cluster
• Deploy Nodejs/Express api which uses a mix of console.log and winston (logging package found on NPM).
• Deploy linked daemonset
• Check logs and see error

Expected behavior

Expect logs to be sent to Loggly without error

Your Environment

Kubernetes latest

Your Configuration

See daemonset for config nothing has change from latest as of June 21 2021

Your Error Log

fluentd-ndmwr:2021-06-21 15:05:17 +0000 [warn]: #0 [in_tail_container_logs] pattern not matched: "2021-06-21T15:05:16.793

Additional context

Example log line from express app

express-api-687bd59447-jn9jg:::ffff:100.96.1.11 - - [21/Jun/2021:15:02:14 +0000] "GET /api/v1/messages/chat/1b9a2714:15195789:09ed:762e HTTP/1.1" 304 75.086 ms - content-length 

[Josh-Klassen-Dejero]

I am running into something similar. I found issue #434 which seems to say that they solution is to use the newest version of the daemonset (v1.12) and use the new cri parser. Not sure if the same thing would help you.

[antoinep92]

See also #412
I understand it's supposed to be fixed with 1.12 with

env:
  - name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
    value: "cri"

So far I haven't been able to make it work, but I hope it work for others

[stieler-it]

Maybe related: I received a similar error with fluent/fluentd-kubernetes-daemonset:v1.13.2-debian-graylog-1.0 with the same configuration that worked for fluent/fluentd-kubernetes-daemonset:v1.13.1-debian-graylog-1.0.

I used to replace tail_container_parse.conf, following the documentation here: https://github.com/fluent/fluentd-kubernetes-daemonset#use-cri-parser-for-containerdcri-o-logs with this content:

<parse>
 @type cri
</parse>

Now I use the environment variable instead:

- name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
  value: "cri"

which seems to generate an additional config value time_format, e.g. here https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/docker-image/v1.13/debian-graylog/conf/tail_container_parse.conf#L3

After this change, v1.13.2 also works for me.

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions]

This issue was automatically closed because of stale in 30 days

[SebastienPi]

which seems to generate an additional config value time_format, e.g. here https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/docker-image/v1.13/debian-graylog/conf/tail_container_parse.conf#L3

After this change, v1.13.2 also works for me.

For time format issue, I had to add:

- name: FLUENT_CONTAINER_TAIL_PARSER_TIME_FORMAT # Default is “%Y-%m-%dT%H:%M:%S.%NZ” (https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/docker-image/v1.16/debian-elasticsearch8/conf/tail_container_parse.conf) but containerd generates “invalid time format: value = 2024-03-07T18:08:35.145463706+01:00” which is “%Y-%m-%dT%H:%M:%S.%N%:z”!
  value: "%Y-%m-%dT%H:%M:%S.%N%:z"

Otherwise, I got this error: 2024-03-07 17:04:29 +0000 [warn]: #0 [in_tail_container_logs] invalid line found file="/var/log/containers/*****" line="2024-03-07T18:01:36.619372894+01:00 stdout F 5430: Thu Mar 7 17:01:36 UTC 2024" error="invalid time format: value = 2024-03-07T18:01:36.619372894+01:00, error_class = ArgumentError, error = string doesn't match"