Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Plugin installation on OpenSearch 2.0.0 runs into jar hell #2590

Closed
cliu123 opened this issue Mar 24, 2022 · 9 comments · Fixed by opensearch-project/security#1709
Closed

[BUG] Plugin installation on OpenSearch 2.0.0 runs into jar hell #2590

cliu123 opened this issue Mar 24, 2022 · 9 comments · Fixed by opensearch-project/security#1709
Assignees
@mch2
Labels
bug Something isn't working

Comments

@cliu123
Copy link
Member

cliu123 commented Mar 24, 2022 •
edited
Loading

Describe the bug
Installing security plugin on OpenSearch 2.0.0 fails on jar hell. The following is the error:

-> Installing file:///Users/cgliu/git/cliu123_OpenSearch_Security/build/distributions/opensearch-security-2.0.0.0.zip
-> Downloading file:///Users/cgliu/git/cliu123_OpenSearch_Security/build/distributions/opensearch-security-2.0.0.0.zip
[=================================================] 100%
-> Failed installing file:///Users/cgliu/git/cliu123_OpenSearch_Security/build/distributions/opensearch-security-2.0.0.0.zip
-> Rolling back file:///Users/cgliu/git/cliu123_OpenSearch_Security/build/distributions/opensearch-security-2.0.0.0.zip
-> Rolled back file:///Users/cgliu/git/cliu123_OpenSearch_Security/build/distributions/opensearch-security-2.0.0.0.zip
Exception in thread "main" java.lang.IllegalStateException: failed to load plugin opensearch-security due to jar hell
	at org.opensearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:675)
	at org.opensearch.plugins.InstallPluginCommand.jarHellCheck(InstallPluginCommand.java:857)
	at org.opensearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:825)
	at org.opensearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:870)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:275)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:249)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)
Caused by: java.lang.IllegalStateException: jar hell!
class: com.fasterxml.jackson.dataformat.yaml.JacksonYAMLParseException
jar1: /Users/cgliu/git/opensearch-2.0.0-SNAPSHOT/lib/jackson-dataformat-yaml-2.13.2.jar
jar2: /Users/cgliu/git/opensearch-2.0.0-SNAPSHOT/plugins/.installing-15161370711895821889/jackson-dataformat-yaml-2.13.2.jar
	at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:317)
	at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:212)
	at org.opensearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:673)

Expected behavior
Plugin is installed smoothly.
@cliu123 cliu123 added bug Something isn't working untriaged labels Mar 24, 2022
@Rishikesh1159 Rishikesh1159 self-assigned this Mar 24, 2022
@peterzhuamazon peterzhuamazon mentioned this issue Mar 24, 2022
Closed
51 tasks
@peternied
Copy link
Member

[Work around] Can we remove the library from the security plugin since its included transitively in OpenSearch?

@peternied peternied mentioned this issue Mar 24, 2022
Merged
3 tasks
@cliu123
Copy link
Member Author

cliu123 commented Mar 25, 2022

The above workaround doesn't look working. @Rishikesh1159 Would you please help out?
Cc: @CEHENKLE

@Rishikesh1159
Copy link
Member

Sure @cliu123 I am working on it. I will provide you an update once I find anything.

@Rishikesh1159
Copy link
Member

Rishikesh1159 commented Mar 28, 2022 •
edited
Loading

Due to some breaking changes in opensearch, security plugin build is failing. So, I cannot reproduce the issue. @cliu123 is working on fixeing build failures in security plugin. Once it is fixed I can resume working on this.

@peternied
Copy link
Member

@Rishikesh1159

I believe the repro includes using a build where @cliu123 has fixed the breaking changes you can get a copy from here https://github.com/cliu123/security/tree/bump_opensearch_dependency_to_2.0.0

The issue itself can be seen in the plugin install workflow link, here is the PR

Inside the workflow logs there is an error, indicating the service did not start, and there is another task called Get Docker Logs that includes debug information such as:

docker logs ops
  shell: /usr/bin/bash -e {0}
  env:
    JAVA_HOME_11.0.14_x64: /opt/hostedtoolcache/jdk/11.0.14/x64
    JAVA_HOME: /opt/hostedtoolcache/jdk/11.0.14/x64
    JAVA_HOME_11_0_14_X64: /opt/hostedtoolcache/jdk/11.0.14/x64
Error: Exception in thread "main" java.lang.IllegalStateException: failed to load plugin opensearch-security due to jar hell
-> Installing file:///docker-host/security-plugin.zip
	at org.opensearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:678)
-> Downloading file:///docker-host/security-plugin.zip
	at org.opensearch.plugins.InstallPluginCommand.jarHellCheck(InstallPluginCommand.java:857)
	at org.opensearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:825)
	at org.opensearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:870)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:275)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:249)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:1[3](https://github.com/opensearch-project/security/runs/5726583125?check_suite_focus=true#step:13:3)8)
	at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:10[4](https://github.com/opensearch-project/security/runs/5726583125?check_suite_focus=true#step:13:4))
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)
Caused by: java.lang.IllegalStateException: jar hell!
class: com.fasterxml.jackson.dataformat.cbor.CBORConstants
jar1: /opensearch/plugins/.installing-777472399439[5](https://github.com/opensearch-project/security/runs/5726583125?check_suite_focus=true#step:13:5)884235/jackson-dataformat-cbor-2.13.2.jar
jar2: /opensearch/lib/jackson-dataformat-cbor-2.12.[6](https://github.com/opensearch-project/security/runs/5726583125?check_suite_focus=true#step:13:6).jar
	at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:31[7](https://github.com/opensearch-project/security/runs/5726583125?check_suite_focus=true#step:13:7))
	at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:212)
	at org.opensearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:676)
	... [11](https://github.com/opensearch-project/security/runs/5726583125?check_suite_focus=true#step:13:11) more

If you would like to reproduce locally you can by use the workflow file

This is blocking the 2.0.0 release from building, lets please focus attention on this issue and reach out if we are having trouble making progress

@Rishikesh1159
Copy link
Member

Rishikesh1159 commented Mar 29, 2022 •
edited
Loading

@mch2 will be working on this issue as top priority. Thanks @peternied @cliu123

@Rishikesh1159 Rishikesh1159 assigned mch2 and unassigned Rishikesh1159 Mar 29, 2022
@mch2
Copy link
Member

mch2 commented Mar 29, 2022

taking a look

@mch2
Copy link
Member

mch2 commented Mar 29, 2022 •
edited
Loading

Hi @cliu123,
I just built your fork with the jackson-databind dependency removed from your gradle scripts - this line and successfully installed & ran the tarball with the demo config with ./bin/opensearch.

I had to use the tarball pulled by the failed workflow, it looks like there are security build failures against the latest version of the 2.0 branch?

What is the desired dependency resolution here? I would suggest unless another version is required security inherits the jackson version from core vs declaring its own.

Also this is failing on the validate step after the server is running. I would expect an error during plugin installation for jarHell vs at runtime, is that ignored/obfuscated in the workflow?

I don't think there is any change required on OpenSearch here. Plugins should either exclude the inherited jar or inherit the jar and not declare its own version. OS also bumped to version 2.13.2.2 today if there was a requirement for 2.13.2 in security.

@mch2 mch2 removed the untriaged label Mar 29, 2022
@peternied
Copy link
Member

I think we should merge opensearch-project/security#1709 to mitigate this issue (and prevent it from coming up again)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mch2 mch2

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove explict dependency on jackson-databind peternied/security
4 participants
@mch2 @peternied @cliu123 @Rishikesh1159