Jar-hell error while executing integTestRemote task

[DarshitChanpura]

Current Behavior:

I keep running into jar hell error everytime I run integTestRemote task, but gradle build seems to be running fine with regular tests.

This issue is blocking development of #1761

Error

java.lang.RuntimeException: found jar hell in test classpath
	at org.opensearch.bootstrap.BootstrapForTesting.<clinit>(BootstrapForTesting.java:118)
	at org.opensearch.test.OpenSearchTestCase.<clinit>(OpenSearchTestCase.java:257)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:398)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:623)
Caused by: java.lang.IllegalStateException: jar hell!
class: org.apache.kafka.common.message.TxnOffsetCommitRequestDataJsonConverter$TxnOffsetCommitRequestTopicJsonConverter
jar1: /Users/dchanp/.gradle/caches/modules-2/files-2.1/org.apache.kafka/kafka-clients/3.0.1/8f931e45e96e952728d540829e5bde9d79fab172/kafka-clients-3.0.1.jar
jar2: /Users/dchanp/.gradle/caches/modules-2/files-2.1/org.apache.kafka/kafka-clients/3.0.1/da500d6dfd3447f56df88dad8ab7ba801150fa4c/kafka-clients-3.0.1-test.jar
	at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:314)
	at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:213)
	at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:100)
	at org.opensearch.bootstrap.BootstrapForTesting.<clinit>(BootstrapForTesting.java:116)
	... 4 more

Repro

Checking out https://github.com/DarshitChanpura/security/tree/initial-test-distr-suite and run ./gradlew integTestRemote to see the

Additional details

build.gradle changes:

import org.opensearch.gradle.test.RestIntegTestTask

apply plugin: 'opensearch.rest-test'
apply plugin: 'opensearch.testclusters'

task integTestRemote(type: RestIntegTestTask) {
    filter {
        setIncludePatterns("org.opensearch.security.sanity.tests.SingleClusterSanityIT")
    }
    nonInputProperties.systemProperty("tests.clustername", "follower")
}

The test class:

package org.opensearch.security.sanity.tests;

import org.junit.Assert;
import org.junit.Test;
import org.opensearch.test.rest.OpenSearchRestTestCase;

import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.stream.Collectors;

public class SingleClusterSanityIT extends OpenSearchRestTestCase {

    private static final String CLUSTER_NAME = "follower";
    private static final String SECURITY_PLUGIN_NAME = "opensearch-security";

    @Test
    public void testSecurityPluginInstallation() throws Exception {
        String uri = "_nodes/" + CLUSTER_NAME + "/plugins";
        verifyPluginInstallation(uri);
    }

    private void verifyPluginInstallation(String uri) throws Exception {
        Map<String, Map<String, Object>> responseMap = (Map<String, Map<String, Object>>) getAsMap(uri).get("nodes");
        for (Map<String, Object> response : responseMap.values()) {
            List<Map<String, Object>> plugins = (List<Map<String, Object>>) response.get("plugins");
            Set<Object> pluginNames = plugins.stream().map(map -> map.get("name")).collect(Collectors.toSet());

            Assert.assertTrue(pluginNames.contains(SECURITY_PLUGIN_NAME));

        }
    }
}

Expected Behavior:

Should not fail with this error.

[peternied]

I was able to work around this issue with DarshitChanpura#4 This is not the correct solution but you can use this to build up the test while we figure out how to resolve the conflicting jars

[peternied]

Following up from the issue in OpenSearch (Above), we should keep the 'disable JarHell' logic in place for the time being, it will keep us unblocked until we arrival at a final state. Please do not resolve this issue until that code has been reverted.

[peternied]

Good news, kafka 3.4.0 will include the build fix for this issue, fix pull request, when this version is released we can remove this workaround

[davidlago]

@peternied are we good to remove the workaround?

[peternied]

@davidlago Thanks for mentioning this issue - yes we can re-enable this check. I've created a pull request to resolve this.