It is possible to disable NodeAndClusterIdConverter plugin.

[lukasz-soszynski-eliatra]

Description

New system property org.opensearch.common.logging.NodeAndClusterIdConverter.enabled added to disable custom Log4j2 plugin NodeAndClusterIdConverter.

Probably this change needs to be merged to another or additional branch.

Issues Resolved

[List any issues this PR will resolve]

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/2196/
• CommitID: 407bb65

[dblock]

Would you mind explaining why you need this?

Is there precedent where we disable other logging plugins this way elsewhere in the code?

Either way will need tests.

[lukasz-soszynski-eliatra]

This PR is related to the integration tests for the OpenSearch security plugin. For more details, please see the conversation in another PR: opensearch-project/security#1967 (comment)

[lukasz-soszynski-eliatra]

I have pushed unit tests related to the modified code fragments.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/2645/
• CommitID: d78b562

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/2646/
• CommitID: 225df5f

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/2706/
• CommitID: 98de5e7

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/2708/
• CommitID: 7719501

[lukasz-soszynski-eliatra]

I noticed that the build fails with the following error message:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':distribution:bwc:bugfix:buildBwcLinuxTar'.
> Building 2.2.1 didn't generate expected file /var/jenkins/workspace/gradle-check/search/distribution/bwc/bugfix/build/bwc/checkout-2.2/distribution/archives/linux-tar/build/distributions/opensearch-min-2.2.1-SNAPSHOT-linux-x64.tar.gz

I found another error message which seems to be related to the problem:

> Task :distribution:bwc:bugfix:buildBwcLinuxTar FAILED
 [2.2.1] > Task :distribution:archives:buildLinuxTar
 [2.2.1] > Task :distribution:archives:linux-tar:assemble
 [2.2.1] build complete, generating: /var/jenkins/workspace/gradle-check/search/distribution/bwc/bugfix/build/bwc/checkout-2.2/build/[2708.tar.bz](http://2708.tar.bz/)2
 [2.2.1] 
 [2.2.1] BUILD SUCCESSFUL in 3m 38s
 [2.2.1] 169 actionable tasks: 169 executed

Unfortunately, both error messages are not very descriptive. @peternied could you help to figure out what is wrong, please?

[peternied]

The following warning ApplierService#updateTask][T#1] WARN ClusterApplierService:628 - failed to notify ClusterStateListener occurs when two or more OpenSearch nodes are run in a single JVM. The root cause of the problem is related to the custom Log4j2 plugin NodeAndClusterIdConverter. The plugin stores node and cluster id in the static field

@lukasz-soszynski-eliatra I'm not sure this is the right change, from your description I think the issue is that we cannot run more than one OpenSearch node inside a single JVM without this issue coming up. It seems like the static field should be modified so it can handle this scenario - that would address the root cause.

If you don't see a way to address the root cause, then I'd recommend creating can create an issue on OpenSearch so it can be triaged / reviewed, then we could work around it by overloading the logger with like was done here for JarHell - see where it was discussed #3905

[lukasz-soszynski-eliatra]

@peternied I created an issue: #4466

[peternied]

@lukasz-soszynski-eliatra I'm going to close this PR out as I don't think this will be moved forward with. Thanks for the issue!