-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
LS 6.2.3: JVM heap size filling up with default jvm.options (invokedynamic and jit.threshold is set) #9346
LS 6.2.3: JVM heap size filling up with default jvm.options (invokedynamic and jit.threshold is set) #9346
Comments
@marioluan I take it this started after an upgrade? Which version were you using previously? |
@andrewvc Yes, it started after an upgrade from LS 5.6.3.
|
1. Regarding the heap size, the following error message is shown at the logs:
2. How many messages per second could you send using The way I could reproduce the issue was by running loggen with the following settings:
|
Would you mind sharing a heap dump? https://dzone.com/articles/memory-analysis-how-to-obtain-java-heat-dump I'll try and repro on a linux box, loggen isn't available on OSX AFAICT |
@andrewvc sent you an email. Let me know when you have the linux box results in place. |
@andrewvc I included a grok filter to that config and got different error messages:
Errors:
PS: This grok pattern runs OK in 5.6.3. |
Another thing I noticed: the issue is intermittent, that is, after restarts it can happen (or not). |
@marioluan what configurations options are you passing into LS? I'm especially interested in the batch size and the number of workers. If you aren't setting the number of workers, how many cores are you using? |
Looking at the heap dump I see I lot of RubyString objects, ~93,400 of them. They appear to be 65535 bytes apiece. The weird thing here is that the Ruby strings are not really that long, they're zero padded to reach that size. The actual message is only 300 bytes. So, that's quite a bit of overhead. I was wondering if it is the case that your |
Also, which queue are you using? Memory or persistent? |
@andrewvc thanks for taking a look at this. This is the settings from logstash.yml: pipeline.workers: 8
pipeline.batch.size: 500
metric.collect: true I'm also setting heap space to 1GB in jvm.options: -Xms1g
-Xmx1g And this is the packet I'm sendint to the UDP listener: {
"@timestamp": "2018-04-09T15:46:03.071Z",
"@version": "1",
"host": "127.0.0.1",
"message": "<38>2018-04-09T15:46:03 localhost prg00000[1234]:
seq: 0000057360, thread: 0000, runid: 1523288758, stamp:
2018-04-09T15:46:03 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPAD
DPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPA
DDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD",
"type": "syslog"
} And I'm using default config for queues. |
The UDP plugin is likely the cause of the 65535 byte strings you are seeing: |
Since logstash 5.6.3 used logstash-input-udp 3.1.2 and you're able to replicate this with some consistency, it may be worth downgrading the plugin: |
This is definitely a JRuby behavior issue. Not necessarily a bug. It looks like we (or the ruby UDP lib) is allocating 64k for the read, then trimming the string to actual length. Unfortunately JRuby tries to optimize this by just changing the It'll take a little looking to see where the UDP lib is doing that, but it should be fixable by re-allocating the string. |
Decreasing the buffer size (a logstash-input-udp option) to your actual maximum message size (1500 bytes?) should be a quick workaround. |
A syslog message has to fit into one UDP packet so we'll set to Guess change in jruby behavior would explain why we don't have this issue when we benchmark 5.6. 5.6 uses |
Thanks for the tip. Setting 20% increase! |
@andrewvc thought about this, and a possible hack/fix to get the Then we'd hand a properly resized (in terms of it's underlying |
I took a first pass at just porting the UDP input to use the java APIs for UDP directly. You can find the result here. logstash-plugins/logstash-input-udp#38 @marioluan if you have a chance to try that out and benchmark it that'd be awesome. If you checkout that folder and add replace the UDP input entry in your gemfile with:
|
tested on LS Tests were run for 15minutes with 300 byte message.
|
As
|
@IrlJidel that seems to be the same as doing |
btw I ran standard plugin on 6.2.3 with I got 64k eps! |
@jsvd We already use a codec I was just trying to use
|
The .b essentially gives you a ASCII copy of the string, so setting the charset to that should have the same effect |
My findings so far:
|
6.2.3
RHEL
jruby 9.1.13.0 (2.3.3) 2017-09-06 8e1c115 OpenJDK 64-Bit Server VM 25.162-b12 on 1.8.0_162-b12 +jit [linux-x86_64]
This same piece of come used to handle up to 40k events per second.
UPDATE: I've just confirmed the issue raises even when only the udp input is set.
The text was updated successfully, but these errors were encountered: