[backport] 8221351: Crash in KlassFactory::check_shared_class_file_load_hook

[navyxliu]

Thank you for taking the time to help improve OpenJDK and Corretto 11.

If your pull request concerns a security vulnerability then please do not file it here.
Instead, report the problem by email to aws-security@amazon.com.
(You can find more information regarding security issues at https://aws.amazon.com/security/vulnerability-reporting/.)

Otherwise, if your pull request concerns OpenJDK 11
and is not specific to Corretto 11,
then we ask you to redirect your contribution to the OpenJDK project.
See http://openjdk.java.net/contribute/ for details on how to do that.

If your issue is specific to Corretto 11,
then you are in the right place.
Please fill in the following information about your pull request.

Description

Current hotspot can't start with both CDS and JVMTI enabled. If users launch Java with java agents, HotSpot will enable jvmti automatically. It is a regression introduced in JDK-8218751

This PR contains 2 backports:

1. JDK-8221351
2. JDK-8221621

Related issues

Motivation and context

How has this been tested?

Platform information

Works on OS: [e.g. Amazon Linux 2 only]
Applies to version [e.g. "11.0.1+13-1" (see output from "java -version")]

Additional context

[caojoshua]

I'm looking through the merge conflicts, and most seem OK. However, the following hunk is not in the PR:

<<<<<<< HEAD
        ProcessBuilder pb = ProcessTools.createJavaProcessBuilder(true, cmd);
=======
        if (RUN_WITH_JFR) {
            boolean usesJFR = false;
            for (String s : cmd) {
                if (s.startsWith("-XX:StartFlightRecording=") || s.startsWith("-XX:FlightRecorderOptions")) {
                    System.out.println("JFR option might have been specified. Don't interfere: " + s);
                    usesJFR = true;
                    break;
                }
            }
            if (!usesJFR) {
                System.out.println("JFR option not specified. Enabling JFR ...");
                cmd.add(0, "-XX:StartFlightRecording=dumponexit=true");
                System.out.println(cmd);
            }
        }

        String[] cmdLine = cmd.toArray(new String[cmd.size()]);
        ProcessBuilder pb = ProcessTools.createJavaProcessBuilder(true, cmdLine);
>>>>>>> e2ffa157620 (8221351: Crash in KlassFactory::check_shared_class_file_load_hook)
        return executeAndLog(pb, "exec");
    }

Seems you just took HEAD, but I think we should take the changes from the patch. What do you think?

[navyxliu]

hi, @caojoshua
Thank you for checking the merge conflicts. I think I drop that hunk by mistake. I will bring it back.