After running the plugin I can't compile or run anything in IDEA

[maor11]

Right clicking the project in the project explorer, selecting SpotBugs -> Analyze Project Files Not Including Test Sources.
The plugin runs successfully but after it's done I can't compile or run anything (including the plugin).
The error I'm getting is:
Error:Cannot run program "C:\Program Files\Java\jdk11.0.4_10\bin\java.exe" (in directory "C:\Users\user.IntelliJIdea2019.3\system\compile-server"): Malformed argument has embedded quote: -Dkotlin.daemon.client.alive.path="C:\Users\user\AppData\Local\Temp\kotlin-idea-10242761005087101084-is-running"

Restarting IDEA solves the issue.

IDEA version:
IntelliJ IDEA 2019.3.4 (Ultimate Edition)
Build #IU-193.6911.18, built on March 17, 2020
Runtime version: 11.0.6+8-b520.43 amd64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
Windows 10 10.0
GC: ParNew, ConcurrentMarkSweep
Memory: 953M
Cores: 4
Registry: debugger.watches.in.variables=false
Non-Bundled Plugins: org.jetbrains.plugins.spotbugs

[MCMicS]

still happen with version 1.1.0
seem related to issue in Youtrack: https://youtrack.jetbrains.com/issue/IDEA-230268

After check files with spotbugs (directly or from commit dialog) results in this error

[Beakster]

I believe I've found a work around to this problem. Before doing this I could run SpotBugs twice before it failed and refused to run again without a restart of IntelliJ:
Help -> Edit Custom VM Options
Add the following line at the bottom of the file:
-Djdk.lang.Process.allowAmbiguousCommands=true

Save the file and exit IntelliJ. Reload IntelliJ and SpotBugs should run every time without failure now.

[amaembo]

First, this affects Windows only.

Second, it looks like the reason is inside the SpotBugs itself:

            if (!SystemProperties.RUNNING_IN_ECLIPSE) {
                /**   don't do this if running in Eclipse; check below is the quick
                fix for bug 3599258 (Random obscure Eclipse failures during
                 analysis) */

                try {
                    Global.getAnalysisCache().getClassAnalysis(ClassData.class, checkerName);

                    // found it.
                    SecurityManager m = System.getSecurityManager();
                    if (m == null) {
                        if (DEBUG_CLASSLOADING) {
                            System.out.println("Setting ValidationSecurityManager");
                        }
                        System.setSecurityManager(ValidationSecurityManager.INSTANCE);
                    }
   ...

So it installs the SecurityManager for the whole IntelliJ IDEA! This obviously affects the behavior of ProcessImpl:

        final SecurityManager security = System.getSecurityManager();
        final String value = GetPropertyAction.
                privilegedGetProperty("jdk.lang.Process.allowAmbiguousCommands",
                        (security == null ? "true" : "false"));
        final boolean allowAmbiguousCommands = !"false".equalsIgnoreCase(value);

        if (allowAmbiguousCommands && security == null) {
            // Legacy mode.
            ... // go here before first SpotBugs invocation
        } else {
            ... // go here after first SpotBugs invocation
            cmdstr = createCommandLine(
                    // We need the extended verification procedures
                    isShell ? VERIFICATION_CMD_BAT
                            : (allowAmbiguousCommands ? VERIFICATION_WIN32 : VERIFICATION_WIN32_SAFE),
                    quoteString(executablePath),
                    cmd);
        }

Installing security manager inside the plugin looks like a very bad idea that may have unforeseen consequences. Unfortunately, it cannot be fixed on the SpotBugs plugin side, the patch should go to the SpotBugs itself. It already has RUNNING_IN_ECLIPSE switch (likely due to similar problems in Eclipse), but we cannot easily modify it as it's final and declared this way:

    public final static boolean RUNNING_IN_ECLIPSE = SystemProperties.class.getClassLoader().getClass().getCanonicalName()
            .startsWith("org.eclipse.osgi");

[amaembo]

Filed SpotBugs PR: spotbugs/spotbugs#1129

[amaembo]

Fixed in SpotBugs, updated in SpotBugs IntelliJ plugin, should work correctly since v.1.1.1.