You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Graylog is not accepting a newline "\n" as a valid GELF short_message, throwing an error message that is not very helpful.
Expected Behavior
I would prefer graylog to accept the newline, or to throw a warning instead of an error.
Many programs (I had the issue with mongodb) log empty lines and it's hard for general log collectors to parse and filter all the possible whitespace and non-useful strings. I understand the error if the field is not set, but maybe empty strings, whitespace and the like could be a warning or info instead of an error if you don't want to accept them.
What I would really like to see is a more descriptive error message and maybe better documentation on what is not allowed in the short_message field.
Current Behavior
Trying to ingest a gelf json with "short_message":"\n" throws an error
The only clue for debugging is the message has empty mandatory "short_message" field:
2018-06-12 14:40:45,775 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Unable to decode raw message RawMessage{id=972f15f0-6e4e-11e8-9b74-a28a23afbb85, journalOffset=17468033, codec=gelf, payloadSize=107, timestamp=2018-06-12T14:40:45.775Z, remoteAddress=/127.0.0.1:45498} on input <5b1f239f2e4b68000198bd9c>.
2018-06-12 14:40:45,775 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{id=972f15f0-6e4e-11e8-9b74-a28a23afbb85, journalOffset=17468033, codec=gelf, payloadSize=107, timestamp=2018-06-12T14:40:45.775Z, remoteAddress=/127.0.0.1:45498}
java.lang.IllegalArgumentException: GELF message <972f15f0-6e4e-11e8-9b74-a28a23afbb85> (received from <127.0.0.1:45498>) has empty mandatory "short_message" field.
at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:252) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:134) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:74) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
I was collecting messages from a very busy fluentd instance, and it has been hard to find what kind of messages were throwing the errors, since I was looking blindly for empty strings.
Possible Solution
I have 3 suggestions:
Accept "\n" as valid short_message or lower its log level.
Print the full message in the error stack trace, at least in DEBUG or TRACE log levels. I think this will be most useful. It's very hard to know what messages are the problematic ones when you have a big volume of them
short_message string (UTF-8)
a short descriptive message; MUST be set by client library.
Context
I am collecting logs from a kubernetes cluster using fluentd daemonset. I am sending the logs to graylog using https://github.com/bodhi-space/fluent-plugin-gelf-hs, which checks for the short_message being not empty, but is not parsing whitespace. I think now the plugin is doing right, since the docs doesn't specify what to check for.
Your Environment
Graylog Version: 2.4.5
Elasticsearch Version: 5.6.2
MongoDB Version: 3.4.6
Operating System:
Browser version:
Thank you very much!
The text was updated successfully, but these errors were encountered:
I'm running into the exact same issue here. Fluentd sending logs to Graylog in a Kubernetes cluster. We have a large number of Java applications that spew empty lines in their logs, just "\n" all the time.
Rignt now, I'm having a hard time configuring fluentd to ignore these.
Graylog is not accepting a newline "\n" as a valid GELF short_message, throwing an error message that is not very helpful.
Expected Behavior
I would prefer graylog to accept the newline, or to throw a warning instead of an error.
Many programs (I had the issue with mongodb) log empty lines and it's hard for general log collectors to parse and filter all the possible whitespace and non-useful strings. I understand the error if the field is not set, but maybe empty strings, whitespace and the like could be a warning or info instead of an error if you don't want to accept them.
What I would really like to see is a more descriptive error message and maybe better documentation on what is not allowed in the short_message field.
Current Behavior
Trying to ingest a gelf json with
"short_message":"\n"
throws an errorThe only clue for debugging is the message
has empty mandatory "short_message" field
:I was collecting messages from a very busy fluentd instance, and it has been hard to find what kind of messages were throwing the errors, since I was looking blindly for empty strings.
Possible Solution
I have 3 suggestions:
Context
I am collecting logs from a kubernetes cluster using fluentd daemonset. I am sending the logs to graylog using https://github.com/bodhi-space/fluent-plugin-gelf-hs, which checks for the short_message being not empty, but is not parsing whitespace. I think now the plugin is doing right, since the docs doesn't specify what to check for.
Your Environment
Thank you very much!
The text was updated successfully, but these errors were encountered: