Logged Errors do not fail tests with IntelliJ 2024.3

[jrlogsdon]

What happened?

As part of https://youtrack.jetbrains.com/issue/IJPL-453/Make-Logger.error-in-tests-work-the-same-as-in-production logged errors no longer fail tests when run as part of Gradle.

It seems there's an aggregator which runs for JPS to collect logged errors and report new test failures related to them, but this does not work for Gradle.

Note: ideally the solution should also work for Bazel, since we use this for tests when developing the Android plugin.

In case anyone else is running into this, for a temporary workaround I'm able to supply the system property intellij.testFramework.rethrow.logged.errors to get these errors thrown instead of silently failing in tests.

Relevant log output or stack trace

No response

Steps to reproduce

Create a gradle intellij plugin with a test extending BasePlatformTestCase which reports something to log.error targeting IntelliJ 2024.3.
Run test

Result:
Test passes
Expected:
test fails (or have at least some indication of a failure)

Gradle IntelliJ Plugin version

2.1.0

Gradle version

8.10.2

Operating System

None

Link to build, i.e. failing GitHub Action job

No response

[hsz]

Thanks for reporting!

To reproduce the issue, it is enough to adjust the Plugin Template with:

Index: src/main/kotlin/org/jetbrains/plugins/template/services/MyProjectService.kt
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/main/kotlin/org/jetbrains/plugins/template/services/MyProjectService.kt b/src/main/kotlin/org/jetbrains/plugins/template/services/MyProjectService.kt
--- a/src/main/kotlin/org/jetbrains/plugins/template/services/MyProjectService.kt	(revision f63486377949e245453da82720cfb8c67ec0a33f)
+++ b/src/main/kotlin/org/jetbrains/plugins/template/services/MyProjectService.kt	(date 1732131885593)
@@ -10,6 +10,7 @@
 
     init {
         thisLogger().info(MyBundle.message("projectService", project.name))
+        thisLogger().error("My error message")
         thisLogger().warn("Don't forget to remove all non-needed sample code files with their corresponding registration entries in `plugin.xml`.")
     }

and then run tests – they'll pass with ##teamcity[...] entries in logs, containing logged errors.
I need to rethink how to approach this story, but for now, I'll enable intellij.testFramework.rethrow.logged.errors via IntelliJ Platform Gradle Plugin.

YouTrack ticket related to this change: https://youtrack.jetbrains.com/issue/IJPL-453#focus=Comments-27-11093365.0-0