You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
When parsing long/multiline log entries (usually due to stack traces and debug information) they often exceed the maximum size resulting in the message being rejected with the following error:
[2022/10/20 00:29:28] [ warn] [output:stackdriver:stackdriver.0] error
{
"error": {
"code": 400,
"message": "Log entry with size 679.5K exceeds maximum size of 256.0K",
"status": "INVALID_ARGUMENT"
}
}
When using a look ahead regex expression with the parse_regex processor there isn't an easy way that I'm aware of to limit the size of the message.
To Reproduce
Steps to reproduce the behaviour:
Create a log receiver with a parse_regex processor with a lookahead expression such as the following: ^(?<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2},\d{3}) (?<severity>[^ ]*) (?<msg>[\s\S]*?)(?=\g<time>|\Z)
2.Read in a log entry that exceeds the 256KB limit:
2022-10-20T00:22:29,900 INFO [main] Test message1
2022-10-20T00:22:29,941 INFO [main] **** PAD THIS OUT WITH A 256KB MULTILINE PAYLOAD ***
2022-10-20T00:22:29,969 INFO [main] Test message2
Expected behaviour
A message should be written to stackdriver to indicate that a log had been dropped rather than it just be silently lost and/or a processor should be provided to allow specific fields in the payload to be truncated based on a maximum length so we can at least get a log entry and part of the message.
The text was updated successfully, but these errors were encountered:
Hi @jgtr-blis thanks for opening an issue. I am actively working on this. I don't know if we have a public issue tracker for it at the moment, but I will keep this issue updated with progress.
Hi @braydonk, is this issue still being worked on? I've been running into a similar issue for some time now. However we're only using "parse_json" and "modify_fields", not the "parse_regex_ processor OP mentioned.
Describe the bug
When parsing long/multiline log entries (usually due to stack traces and debug information) they often exceed the maximum size resulting in the message being rejected with the following error:
When using a look ahead regex expression with the parse_regex processor there isn't an easy way that I'm aware of to limit the size of the message.
To Reproduce
Steps to reproduce the behaviour:
^(?<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2},\d{3}) (?<severity>[^ ]*) (?<msg>[\s\S]*?)(?=\g<time>|\Z)
2.Read in a log entry that exceeds the 256KB limit:
Expected behaviour
A message should be written to stackdriver to indicate that a log had been dropped rather than it just be silently lost and/or a processor should be provided to allow specific fields in the payload to be truncated based on a maximum length so we can at least get a log entry and part of the message.
The text was updated successfully, but these errors were encountered: