-Xdump:heap creates heapdump on kill -3 causing issues with performance data gathering

[dspath]

Could we get that changed to not trigger the heapdump on kill -3?

Some clients are setting up JVM arguments like this:
-Xdump:heap:defaults:file=heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd
-Xdump:system:defaults:file=core.%Y%m%d.%H%M%S.%pid.%seq.dmp

or this:
-Xdump:system:label=PRDWSCSCC306.aoins.com.WPCCSCC306A1.%Y%m%d.%H%M%S.%pid.%seq.dmp
-Xdump:java:label=PRDWSCSCC306.aoins.com.WPCCSCC306A1.%Y%m%d.%H%M%S.%pid.%seq.txt
-Xdump:heap:label=PRDWSCSCC306.aoins.com.WPCCSCC306A1.%Y%m%d.%H%M%S.%pid.phd

The "-Xdump:heap" args will cause a heapdump to be created on every kill -3. Our standard MustGather scripts for Linux and AIX perform a kill -3 every 30 seconds, but we often need clients to reduce that to a much smaller number, so the heapdump generations and accompanying global gc's are extremely disruptive when gathering performance data with our MustGather scripts and can bring a client's JVM down.

"-Xdump:system" doesn't cause a core dump to be created on kill -3, so why does "-Xdump:heap"?

For a standalone java process, if someone needs a heapdump on kill -3 could add and environment variable "export IBM_HEAPDUMP=TRUE".

In WebSphere, if they need to create a heapdump manually they have two good options:

1. Via the Admin Console per "Collecting Java dumps and core files using the administrative console"
   https://www.ibm.com/docs/en/was-nd/8.5.5?topic=ts-collecting-java-dumps-core-files-using-administrative-console

2. Using wsadmin per "Generating heap dumps manually"
   https://www.ibm.com/docs/en/was/8.5.5?topic=generation-generating-heap-dumps-manually

In Liberty, they can run:
server dump server_name --include=heap

[pshipton]

There is no heap dump agent by default other than for OutOfMemory. The option -Xdump:heap:defaults:file=hd doesn't add a heap dump agent, but -Xdump:heap:label=hd is adding a dump agent using the default options. The user event is a default and is causing the heapdump on each kill -3, but you can change this default.

-Xdump:heap:
    events=gpf+user,
    file=/home/peter/hd,
    range=1..0,
    priority=500,
    request=exclusive+compact+prepwalk,
    opts=PHD

i.e. The -Xdump:heap:defaults are as follows. I'm not sure why user is a default, but somebody might be depending on it, changing it isn't something to be done lightly.

  events=gpf+user
  filter=
  file=/home/peter/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd
  range=1..0
  priority=500
  request=exclusive+compact+prepwalk
  opts=PHD

You can change the default events via -Xdump:heap:defaults:events=gpf or -Xdump:heap:defaults:file=heapdump,events=gpf to change both the file and events at the same time. Then doing -Xdump:heap:label=hd won't create an agent for the user event.

-Xdump:heap:
    events=gpf,
    file=/home/peter/hd,
    range=1..0,
    priority=500,
    request=exclusive+compact+prepwalk,
    opts=PHD

To see all the dump agents in affect after setting options, use -Xdump:what
For example: java -Xdump:heap:defaults:file=heapdump,events=gpf -Xdump:heap:label=hd -Xdump:what

@keithc-ca fyi

[dspath]

I don't think we should have to have clients add things like -Xdump:heap:defaults:events=gpf or -Xdump:heap:defaults:file=heapdump,events=gpf then add their -Xdump:heap:label. If there is no -Xdump:heap, then a heapdump is not created on kill -3, so just adding -Xdump:heap:label shouldn't change the behavior of kill -3, it should just create heapdumps on OOM's per the "label".

[pshipton]

Why is -Xdump:heap:label= being set, what are they trying to accomplish? I expect it's just wrong and they should be doing something else.

[dspath]

We prefer they not do this and we discourage them at every opportunity, but some clients have adopted some settings we don't like I think because heapdumps didn't used to be so easy to get "settings like JAVA_DUMP_OPTS...., or IBM_HEAPDUMP=true". Some are extremely resistant to removing these. It doesn't change the fact that setting it shouldn't change the behavior when performing kill -3.

[yathamravali]

@dspath

Below are my observations after testing:

1. Aș @pshipton already mentioned, Below options will not trigger any default events. These will only change the file name which is inline with the documentation https://www.ibm.com/docs/en/sdk-java-technology/8?topic=options-xdump#defaults

-Xdump:heap:defaults:file=heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd
(or)
-Xdump:heap:defaults:label=heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd

So if customer would like to change default file name settings, then setting above options will do the job without generating any dumps on user event.

2. Now coming to below options as there is no events specified explicitly these will result in triggering the default events set for heap

-Xdump:heap:file=PRDWSCSCC306.aoins.com.WPCCSCC306A1.%Y%m%d.%H%M%S.%pid.phd
(or)
-Xdump:heap:label=PRDWSCSCC306.aoins.com.WPCCSCC306A1.%Y%m%d.%H%M%S.%pid.phd

And the default events set for heapdump are shown below, which is why on user event heap dump is getting generated

> java -Xdump:heap:defaults

Default -Xdump:heap settings:
  
events=gpf+user
  filter=
  file=/root/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd
  range=1..0
  priority=500
  request=exclusive+compact+prepwalk
  opts=PHD

To answer your question "-Xdump:system" doesn't cause a core dump to be created on kill -3, so why does "-Xdump:heap"?

This is because default settings for system dump has no user event set which is why no coredump is generated on user event

java -Xdump:system:defaults

Default -Xdump:system settings:

  events=gpf+abort+traceassert+corruptcache
  filter=
  file=/root/core.%Y%m%d.%H%M%S.%pid.%seq.dmp
  range=1..0
  priority=999
  request=serial
  opts=

It is important that we understand what customer would like to achieve using the options specified in point 2 above, If their requirement is only to have their label/file name settings modified then it is recommended to use the options specified in point1 as per the documentation.

[dspath]

Hi Ravali, thank you very much for the detailed explanation. Closing this item.

[yathamravali]

@pshipton @keithc-ca

1)We are trying to find the difference between Xdump file and label options. Couldn’t find any documentation for label option nor find it in-Xdump:help. As per our understanding both have the same functionality, could you please confirm?

2)Noticed that there was an issue raised to change the heap dump defaults #520. And through the PR #563 the defaults for heap dump has changed from gpf\usr to systhrow.

Above change is reverted back through #4257

Could we also get more details on why the PR change was reverted back, Are there any implications?(From the service perspective we don’t see a need for a heap dump on gpf event, shouldn’t it be on systhrow?)

[pshipton]

file and label appear to be quite similar. I'm not sure if label is an old published option that was deprecated and removed from the docs, or just an internal option that escaped to customers. Most platforms use file, z/OS uses dsn for the system core, but label can be used in their place.

#563 was reverted because it was only supposed to fix the help output to match the defaults, but it changed the behaviour. There is no reason given for the revert, I'm not sure if it was a service issue or just something Keith noticed.

[yathamravali]

Thanks @pshipton.

1. There isn't problem with below option and it is clearly documented that this option will not add any agent Xdump defaults
   -Xdump:<agent>:defaults:file=<filename>

2. Issue is with below option, By now it's clear that this is actually adding dump agents using defaults but we don't see it documented.
   -Xdump:heap:file=PRDWSCSCC306.aoins.com.WPCCSCC306A1.%Y%m%d.%H%M%S.%pid.phd

From the Dump agent suboptions table, could see below but this is not just updating the file name it’s actually adding defaults set for the agent as well.

-Xdump:<agent>:file=<filename> Specifies where to write the dump for the dump agent.

For eg:

java -Xdump:heap:file=ry.hcd -Xdump:what

-Xdump:heap:
    events=gpf+user,
    file=/root/ry.hcd,
    range=1..0,
    priority=500,
    request=exclusive+compact+prepwalk,
    opts=PHD
----------------------
-Xdump:heap:
    events=systhrow,
    filter=java/lang/OutOfMemoryError,
    file=/root/ry.hcd,
    range=1..4,
    priority=500,
    request=exclusive+compact+prepwalk,
    opts=PHD

 java -Xdump:heap -Xdump:what

----------------------
-Xdump:heap:
    events=gpf+user,
    file=/root/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd,
    range=1..0,
    priority=500,
    request=exclusive+compact+prepwalk,
    opts=PHD
----------------------
-Xdump:heap:
    events=systhrow,
    filter=java/lang/OutOfMemoryError,
    file=/root/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd,
    range=1..4,
    priority=500,
    request=exclusive+compact+prepwalk,
    opts=PHD

And similar behaviour is observed for other Dump agent suboptions in the table:

java -Xdump:heap:OPTS=CLASSIC -Xdump:what
-Xdump:heap:
    events=gpf+user,
    file=/root/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd,
    range=1..0,
    priority=500,
    request=exclusive+compact+prepwalk,
    opts=CLASSIC
----------------------
-Xdump:heap:
    events=systhrow,
    filter=java/lang/OutOfMemoryError,
    file=/root/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd,
    range=1..4,
    priority=500,
    request=exclusive+compact+prepwalk,
    opts=CLASSIC

@pshipton Could we have this information updated in the documentation i.e if no events are specified explicitly then defaults for the corresponding agent will be added?

[pshipton]

Created eclipse-openj9/openj9-docs#970

[Sreekala-Gopakumar]

file and label appear to be quite similar. I'm not sure if label is an old published option that was deprecated and removed from the docs, or just an internal option that escaped to customers. Most platforms use file, z/OS uses dsn for the system core, but label can be used in their place.

#563 was reverted because it was only supposed to fix the help output to match the defaults, but it changed the behaviour. There is no reason given for the revert, I'm not sure if it was a service issue or just something Keith noticed.

@pshipton - Does this require any change in the document? If yes, then could you please let me know how I can differentiate between the 'file' and 'label' options? Thanks!

[keithc-ca]

The document shows label in the defaults: I think those should instead say file. I don't think we need to document use of label (even though it is still accepted).

[yathamravali]

Document update has been done (eclipse-openj9/openj9-docs#970 - closed).

This item can be closed.