[CI] MlDistributedFailureIT testClusterWithTwoMlNodes_StopsDatafeed_GivenJobFailsOnReassign failing

[mark-vieira]

There have been two instances of this today (both in PR builds) that blew up due to an OOME.

Build scan:
https://gradle-enterprise.elastic.co/s/ryfsljqpbyjlm/tests/:x-pack:plugin:ml:internalClusterTest/org.elasticsearch.xpack.ml.integration.MlDistributedFailureIT/testClusterWithTwoMlNodes_StopsDatafeed_GivenJobFailsOnReassign

Reproduction line:
./gradlew ':x-pack:plugin:ml:internalClusterTest' --tests "org.elasticsearch.xpack.ml.integration.MlDistributedFailureIT.testClusterWithTwoMlNodes_StopsDatafeed_GivenJobFailsOnReassign" -Dtests.seed=A3E13461A939BDF6 -Dtests.locale=ar-IQ -Dtests.timezone=America/Porto_Velho -Druntime.java=17

Applicable branches:
master

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.ml.integration.MlDistributedFailureIT&tests.test=testClusterWithTwoMlNodes_StopsDatafeed_GivenJobFailsOnReassign

Failure excerpt:

java.lang.Exception: Test abandoned because suite timeout was reached.

  at __randomizedtesting.SeedInfo.seed([A3E13461A939BDF6]:0)

[elasticmachine]

Pinging @elastic/ml-core (Team:ML)

[benwtrent]

So, looking at the heapdump:

A new object added in: #88097

is taking 250MB of a ~500MB of heap.

x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/ILMImmutableStateHandlerProvider.java

@grcevski what does this provider do?
You can get the heap dump from the test results (or I can provide it directly, its much too big to add here).

[grcevski]

Oh that's unexpected. I guess I have a bug there, it should be only one object at the moment. It appears that the ILM plugin would reload many times and we add that immutable state handler many many times. I'll take this over to fix.

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[davidkyle]

Different test in the same suite and failure is a ResourceNotFoundException but looking at the logs the JVM is spending nearly all the time garbage collecting so I'm guessing the root cause is the same.

https://gradle-enterprise.elastic.co/s/i4badycr5nt5k/tests/:x-pack:plugin:ml:internalClusterTest/org.elasticsearch.xpack.ml.integration.MlDistributedFailureIT/testClusterWithTwoMlNodes_StopsDatafeed_GivenJobFailsOnReassign