Support Gradle configuration cache on multiple JVMs

[nedtwigg]

Gradle configuration cache requires task state to be round-trip serializable. As outlined at #987, this is a signficant constraint on plugin developers and it slows down the end-user experience of change-test-change-test on a single JVM daemon. We built a workaround (JvmLocalCache) that complies with Gradle's requirement by making our tasks round-trip serializable without requiring much work from plugin developers or the end user's machine, but it only works within a single JVM.

To work on multiple JVMs, we need to

• make every step round-trip serializable
• remove JvmLocalCache

[eskatos]

FYI Gradle 8 will do a full roundtrip on the first invocation with CC enabled.
See work in progress here gradle/gradle#21985 (comment)
The goal is for this to be the default in 8.0.

[nedtwigg]

Full roundtrip will work great on the same JVM, so Spotless will still work and not work under the same conditions it does currently. EDIT: Compatibility with configuration cache within a single JVM on Gradle 8 is confirmed.

[nedtwigg]

I think we now have a way to incrementally build this until we can eventually remove our workaround. See:

• Start making each FormatterStep roundtrip serializable. #1945

In this comment, I'm going to maintain a list of remaining TODOs before we will fully support Gradle's configuration cache model without any workarounds. If you'd like to help: comment in this issue when you start on a given step, follow-up with a PR (draft is fine) and I will update the checklist below

• remove LineEnding.GIT_ATTRIBUTES (at least for Gradle) because very difficult to make it serializable Support Gradle configuration cache on multiple JVMs  #1274
  • we can keep GIT_ATTRIBUTES_FAST_ALLSAME which is the default anyway
• lib(generic.PipeStepPair) Serializable refactor pipe/fence #1954
• lib(generic.EndWithNewlineStep)
• lib(generic.IndentStep) Start making each FormatterStep roundtrip serializable. #1945
• lib(generic.Jsr223Step) Make all steps with JarState serializable #2039
• lib(generic.LicenseHeaderStep) Port all stragglers except npm #2132
• lib(generic.NativeCmdStep) Port all stragglers except npm #2132
• lib(generic.ReplaceRegexStep)
• lib(generic.ReplaceStep)
• lib(generic.TrimTrailingWhitespaceStep)
• lib(groovy.RemoveSemicolonsStep)
• lib(antlr4.Antlr4FormatterStep) Serializable Groovy, Java, Kotlin, and Scala steps #2033
• lib(cpp.ClangFormatStep) Serializable foreign exe #1955
• extra(cpp.EclipseFormatterStep) Make the Equo-based steps round-trip serializable #1950
• lib(gherkin.GherkinUtilsStep) Make all steps with JarState serializable #2039
• extra(groovy.GrEclipseFormatterStep) Make the Equo-based steps round-trip serializable #1950
• lib(java.GoogleJavaFormatStep) Serializable Groovy, Java, Kotlin, and Scala steps #2033
• lib(java.ImportOrderStep) Serializable Groovy, Java, Kotlin, and Scala steps #2033
• lib(java.PalantirJavaFormatStep) Serializable Groovy, Java, Kotlin, and Scala steps #2033
• lib(java.RemoveUnusedImportsStep) Serializable Groovy, Java, Kotlin, and Scala steps #2033
• extra(java.EclipseJdtFormatterStep) Make the Equo-based steps round-trip serializable #1950
• lib(java.FormatAnnotationsStep) Serializable Groovy, Java, Kotlin, and Scala steps #2033
• lib(java.CleanthatJavaStep) Serializable Groovy, Java, Kotlin, and Scala steps #2033
• lib(json.gson.GsonStep) Make all steps with JarState serializable #2039
• lib(json.JacksonJsonStep) Make all steps with JarState serializable #2039
• lib(json.JsonSimpleStep) Make all steps with JarState serializable #2039
• lib(json.JsonPatchStep) Make all steps with JarState serializable #2039
• lib(kotlin.KtLintStep) Serializable Groovy, Java, Kotlin, and Scala steps #2033
• lib(kotlin.KtfmtStep) Serializable Groovy, Java, Kotlin, and Scala steps #2033
• lib(kotlin.DiktatStep) Start making each FormatterStep roundtrip serializable. #1945
• lib(markdown.FreshMarkStep) Make all steps with JarState serializable #2039
• lib(markdown.FlexmarkStep) Make all steps with JarState serializable #2039
• lib(npm.EslintFormatterStep) Make the npm steps roundtrip serializable #2135
• lib(npm.PrettierFormatterStep) Make the npm steps roundtrip serializable #2135
• lib(npm.TsFmtFormatterStep) Make the npm steps roundtrip serializable #2135
• lib(pom.SortPomStepStep) Make all steps with JarState serializable #2039
• lib(protobuf.BufStep) Serializable foreign exe #1955
• lib(python.BlackStep) Serializable foreign exe #1955
• lib(biome.BiomeStep) Make the npm steps roundtrip serializable #2135
• lib(scala.ScalaFmtStep) Serializable Groovy, Java, Kotlin, and Scala steps #2033
• lib(sql.DBeaverSQLFormatterStep) Port all stragglers except npm #2132
• extra(wtp.EclipseWtpFormatterStep) Serializable refactor legacy eclipse steps #1952
• lib(yaml.JacksonYamlStep) Make all steps with JarState serializable #2039

[lehmanju]

I checked npm.EslintFormatterStep, npm.PrettierFormatterStep, npm.TsFmtFormatterStep and sql.DBeaverSQLFormatterStep. They all use StepHarness for testing which has roundtrip serialization testing enabled by default. Since their tests do not fail, it should mean they are already serializable, am I correct?

For NativeCmdStep I created a PR #2108 which adds a test using StepHarness.

[nedtwigg]

Thanks mostly to @Goooler's tireless dedication, we have finally shipped:

• plugin-gradle 3.0.0.BETA1
• plugin-maven 2.44.0.BETA1

These contain a bazillion fixes. Please test and let us know if you encounter any issues!

[marcphilipp]

JUnit build updated ✅

[bddckr]

All working fine for me as well, except I used

removeUnusedImports("cleanthat-javaparser-unnecessaryimport")

as seen in the docs. That one seems to not support the config cache yet:

Could not load the value of field `steps` of task `:spotlessInternalRegisterDependencies` of type `com.diffplug.gradle.spotless.RegisterDependenciesTask`.
> Could not load the value of field `excluded` of `com.diffplug.spotless.java.CleanthatJavaStep` bean found in field `roundtripStateInternal` of `com.diffplug.spotless.FormatterStepSerializationRoundtrip` bean found in field `steps` of task `:spotlessInternalRegisterDependencies` of type `com.diffplug.gradle.spotless.RegisterDependenciesTask`.

I just removed it as it looks like googleJavaFormat already takes care of that.

[Magotchi]

I get a similar error to @bddckr if I use any prettier steps, and only while using the Gradle Configuration Cache, like:

Could not load the value of field `steps` of task `:spotlessJava` of type `com.diffplug.gradle.spotless.SpotlessTaskImpl`.
> Could not load the value of field `additionalNpmrcLocations` of `com.diffplug.spotless.npm.NpmPathResolver` bean found in field `resolver` of `com.diffplug.spotless.npm.NpmFormatterStepLocations` bean found in field `locations` of `com.diffplug.spotless.npm.PrettierFormatterStep$State` bean found in field `roundtripStateInternal` of `com.diffplug.spotless.FormatterStepSerializationRoundtrip` bean found in field `steps` of task `:spotlessJava` of type `com.diffplug.gradle.spotless.SpotlessTaskImpl`.

or:

Could not load the value of field `steps` of task `:spotlessPrettierGeneric` of type `com.diffplug.gradle.spotless.SpotlessTaskImpl`.
> Could not load the value of field `additionalNpmrcLocations` of `com.diffplug.spotless.npm.NpmPathResolver` bean found in field `resolver` of `com.diffplug.spotless.npm.NpmFormatterStepLocations` bean found in field `locations` of `com.diffplug.spotless.npm.PrettierFormatterStep$State` bean found in field `roundtripStateInternal` of `com.diffplug.spotless.FormatterStepSerializationRoundtrip` bean found in field `steps` of task `:spotlessPrettierGeneric` of type `com.diffplug.gradle.spotless.SpotlessTaskImpl`.

I've worked around it by declaring Spotless incompatible with the configuration cache, like:

tasks.withType<SpotlessTask>().configureEach {
  notCompatibleWithConfigurationCache(
      "Spotless 7.0.0.BETA1 with prettier steps does not support the configuration cache.")
}

[davidburstrom]

FWIW, I've upgraded to 7.0.0.BETA2 in all my repositories and everything seems to be in order! Great job!

[nedtwigg]

With 7.0.0.BETA4 I think this is now just about fully solved. Last chance for feedback before these APIs all get baked!

• Feedback wanted! Ending beta period for 7.0 #2311

[ZacSweers]

Just filed a regression on #2318