-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Gelf input should use codecs #37
Comments
To make multiline work, stream identity support is required. |
@Teudimundo - so as I see it, there are a few problems with using multiline codec with this input.
From the GELF doc
I have some confusion about the Are you expecting to receive many GELF messages with one or some lines that make up the logical multiline text? If the logical mutiline text is an error message + stacktrace why are they not in the |
I see the problems. In my case I'm using the docker log driver, and so I cannot control the short full message fields, because the driver just creates an event for each line (and obviously is completely unaware of what there is in the line). This would make also 'host' not a good solution either for the streaming, that I would like to be based on a custom filed (in my case If the multiline was just collecting events in groups, and delegated the "merging" to the input driver, then it would allow the driver to be able to take some decision, possibly guided by some configuration (like the field that will contain data to be merged). But that would be a complete different approach and I don't expect it to be implemented. |
Is there any update on this? I'd also like to be able to use multiline codec for gelf input, instead of using multiline filter. |
I need this option too. |
It is very unlikely that this will be implemented anytime soon. We are working on a different way of processing raw source data in an input. We call this Event Milling (as in a Saw Mill that transforms raw logs). See this discussion issue. |
Hi, we would really love to use gelf with the docker log driver and get our java stack traces as multiline events. Please keep us posted once there is any progress that does not involve using the multiline filter. Thanks! |
Hey, I've been trying to find out how to use any input with the multiline codec from multiple sources but all the issues are still open ( Some got created last summer ?? ). Is anyone working on these issues? Can't use this if its not supported, we cant create 100s of different logstashes to make a 1-1 relation between logstash and an application. Tried to put multiline in the filter but its terrible slow. Started to lose logs instantly when I pushed it a bit. Thanks! |
Yes, any updates here? We're also hitting this with docker log output, and I would appreciate any info about the current state (even if it might be disappointing ;) ) |
We gave up finding any existing solution to the stacktrace problem and implemented a 800 LoC "Lokstash" (it's in kotlin), that uses the dockerApi project and elasticsearch java lib. It means we need one pr docker server unfortunately |
Hm. Any docs on this? Did you open-source it? Is it usable for others (read: me)? :) |
@flypenguin Not really. Guess it could be open sourced, but it's pretty special cased to our use case. It was just to say, that we used less time implementing our selves than we have spent searching and testing existing solution |
For reference, a GELF-format message from Docker is below:
container_id is perfect for identifying lines from the same source. With this data is there a reason the 'message' field couldn't be passed off to a multiline codec for collapsing? |
yup, a solution for this would be just awesome. the use case is pretty clear to me, first hand. |
Agreed, we are also stuck due to this exact issue. Merging multi-line gelf messages from the docker log driver in logstash seems impossible. |
+1 big problem for me too. I need the multiline support for GELF, preferrably with stream identity support |
@jhmartin, @flypenguin, @mancej, @vbohata I will try to explain why this is not so easy in an attempt to stop all the +1's. It is on my radar but we are few in the LS team and there is much preparatory work we need to do first before a sustainable solution for this problem can be realised. The API for a codec is: The API for the identity_map_codec (AKA IMC, a codec wrapper with identity support - the multiline codec is usually the wrapped codec) is: Each new identity seen will create a fresh copy of the multiline codec because we have to keep the accumulations separated. The main piece of complexity is that the multiline codecs via the IMC need to flush their accumulated lines when a timeout occurs and no new multiline boundary is seen. As this timeout may occur asynchronously (not when the decode closure is available to call - to enqueue the flushed event) we have to do a bit of hoop-jumping to make the closure available to all. These timeout flushes operate in a different thread than the input thread so we have to take thread safety into account. If the LS input design was different (called Event Milling, linked above) we could simply add a new processor between the source (gelf UDP listener) and the IMC to extract the identity and the message from the event built by the source and pass those to the IMC and the IMC will pass to the queue. |
I wanted to note that since the multiline filter is deprecated in logstash-5.x this is a blocker for anyone who wants to upgrade and keep multiline intact. |
Is there any news on this issue? Being able to handle multiline GELF input is crucial |
I solve that problem by logging errors in json (one line) from my apps |
any updates on this ? |
filebeat can do that job well too |
No real solution since the issue was posted 2016. I think the conclusion is to look into another solution to aggregate docker logs. |
Looking for a workaround. If I have control of the log formatter used by the application, is there any way I can encode newlines and then replace the encoded character with newline using some sort of logstash filter? |
Well I ended up urlencoding the logs in the application. But it is ugly - would appreciate a real solution that didn't obfuscate the original logs |
Still nothing on this? |
I'm trying to use a multiline codec with the gelf input. Unfortunately looks like it is ignored. I've checked the code, and it looks like codec is completely ignored (I can barely read Ruby, though). But the same conclusions looks to be reached in the logstash issue #4308
The text was updated successfully, but these errors were encountered: