Add support for build caching of in-place processed files

[mvysny]

Expected Behavior

Certain plugins edit files in-place: for example file formatters formatting java files in-place, or newline character plugins, converting CRLF to just CR or just LF. Gradle should allow a task parameter to be both input and output, to add support for build-caching plugins employing this kind of in-place editing. The cache key can be calculated before the plugin runs; the task outputs can be cached after the task runs.

Current Behavior (optional)

No response

Context

The Gradle build caching mechanism supports for caching outputs for given inputs; if the input files hashes match the cache then the outputs are reused from the build cache and the task is skipped and marked as FROM-CACHE.

My use-case calls for an in-place processing of a file. I'm writing a Vaadin Gradle plugin which reads the package.json file and fills in particular missing keys+values and writes the changes back to the package.json file. If those missing keys+values are already present in the json file then nothing gets updated. That means that the output eventually reaches a stable state where the file is not changed - namely, second and all subsequent runs of the plugin no longer needs to modify the file.

This means that the package.json file is both an @InputFile and an @OutputFile at the same time. Unfortunately, Gradle doesn't allow a parameter to be marked with both @InputFile and @OutputFile at the same time. We tried to only use @OutputFile annotation but quickly reached the problem of "overlapping outputs" as described at https://mvysny.github.io/gradle-overlapping-outputs/ .

The problem is described in more detail as this StackOverflow question: https://stackoverflow.com/questions/77807234/gradle-cache-task-which-processes-a-file-in-place . A solution is suggested, however the solution relies on writing 'synthetic' tasks that do nothing other than copy the file into a subdirectory and then back again, to 'simulate' a parameter that's both input and output. Such solution would be hard to understand and maintain, and therefore I believe it would be great to have a proper support for input+output properties directly by Gradle.

[steve-todorov]

This problem has been a blocker for us for a few months now. We invested quite a lot of time to debug and reproduce it. I don't know if this is an "expected" behavior, but it is certainly an undesired behavior that is very hard to track.

I hope there is a reasonable way to implement this feature instead of relying on synthetic tasks.

[cobexer]

Thank you for your interest in Gradle!

This issue needs a decision from the team responsible for that area. They have been informed. Response time may vary.

[oliveryasuna]

@cobexer Thank you for looking into this. This is a problem for my team.

[steve-todorov]

Any news on this? :)

[steve-todorov]

@cobexer It has been a month since this ticket was opened. Can we get some update? This problem is causing a lot of wasted build time for us (vaadin/flow#17941).

[lptr]

Adding support for lint-like tasks that modify their own inputs has been on our radar for some time, but there is no plan to implement this feature in the foreseeable future. I'll keep the issue open because it's a good description of the problem (thanks for that).

As an alternative, consider generating the output package.json in a separate location from the input file, and use it from there. That setup would be more in line with how Gradle thinks about builds, where tasks are supposed to be like pure functions. Doing so allows us to employ a number of modifications safely. Allowing in-place modification as part of a CI build can also result in weird situations like leaving the VCS state dirty.

We might still support in-place input modification, but if we do that, it is likely going to be targeted at the linting use case, to be used by developers in the IDE, not as part of the build. Not sure how the mechanics of this would work yet, though.