-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP: Make QuarkusBuild
not pollute Gradle's build cache
#30894
Conversation
/cc @geoand this is a working draft. No integration tests or the like yet. I've chosen 2.16 as the base branch, because we have no choice to move to Jakarta EE 10 quickly :( |
c415c80
to
856031d
Compare
Remaining TODOs for this PR:
|
13890d9
to
352a978
Compare
Thanks a lot for this!
I am pretty certain we won't be merging changes of this magniture to |
352a978
to
63294d1
Compare
Also worth taking into account that we also support application.yaml, config profiles, env vars, etc. |
Good point. I actually thought quite a lot about the way the "effective configuration" is calculated in "all the places". One thing that I think is a bug in the current implementation is that configuration in
WDYT? |
7431fc9
to
c092355
Compare
Sounds good @snazy |
Also adds a shutdown hook for launched processes to kill those when tests are aborted w/ ctrl-c. A process, like the ones spawned by for example `FastJarFormatWorksTest`, can otherwise survive, still "hold" port 8080 and cause subsequent test failures.
Currently the `QuarkusBuild` task implementation adds even large build artifacts and unmodified dependency jars to Gradle's build cache. This pollutes the Gradle build cache quite a lot and therefore lets archived build caches become unnecessary huge. This change updates the build logic to fix this behavior by not adding dependencies and large build artifacts, like uber-jar and native binary, to the Gradle build cache. The `QuarkusBuild` task has been split into three tasks: 1. a new `QuarkusBuildDependencies` task that only collects the contents for `build/quarkus-app/lib` 2. a new `QuarkusBuildApp` task that to collect everything else from a Quarkus build (everything except the `build/quarkus-app/lib`) 3. the `QuarkusBuild` task now combines the outputs of the above two tasks `QuarkusBuildDependencies` (named 'quarkusDependenciesBuild`) is not cacheable, because it only collects dependencies, which come either from a repository (and are already available locally elsewhere) or are built by other Gradle tasks. This task is only executed if the configured Quarkus package type requires the "quarkus-app" directory (`fast-jar` + `native`). It's "build working directory" is `build/quarkus-build/dep`. `QuarkusBuildApp` (named `quarkusAppBuild`) collects the contents of the "quarkus-app" directory _excluding_ the `lib/` directory are cacheable, which is the default for CI environments. Non-CI environments still cache all outputs, even uber-jars and native binaries to retain the existing behavior for developers and keep build turn-around times low. CI environments can opt-in to add even huge artifacts to Gradle's build cache by explicitly setting the `cacheUberAndNativeRunners` property in the Quarkus extension to `true`. It's "build working directory" is `build/quarkus-build/app`. Since `QuarkusBuild` only combines the outputs of the above two tasks, the same "CI vs local" caching behavior as for the `QuarkusBuildApp` task applies. To make "up to date" checks (kind of) more reliable, all outputs are removed first. This means, that for example an existing uber-jar in `build/` will disappear, when the build's package type is "fast-jar". This behavior can be disabled by setting the `cleanupBuildOutput` property on the Quarkus extension to `false`. Both `QuarkusBuildDependencies` and `QuarkusBuildApp` can trigger an actual Quarkus application build. That Quarkus app build will only be triggered when needed and only once per Gradle build. The task names `quarkusDependenciesBuild` and `quarkusAppBuild` are intentionally "that way around". Letting the names of these tasks begin with `quarkusBuild...` could confuse users, who use abbreviated task names on the command line (for example `./gradlew qB` is automagically expanded to `./gradlew quarkusBuild`, which would become ambiguous with `quarkusBuildDependencies` and `quarkusBuildApp`). Unless the `cacheLargeArtifacts` property on the `quarkus` extension is set to `true`, the output of/for the package type `fast-jar` minus the dependency jars is cached by Gradle's build cache (similar for `legacy-jar`). Basically everything is cacheable to allow fast(er) local development turn-around cycles. To be able to investigate which configuration settings were effectively used, there's another new task called `quarkusShowEffectiveConfig` that shows all the `quarkus.*` properties and some more information, including the loaded `application.(properties|yaml|yml)`. This task is intended to debug build issues. The task can optionally save the effecitve config properties as a properties file in the `build/` directory, if run with the command line option `./gradlew quarkusShowEffectiveConfig --save-config-properties`. Relates to: quarkusio#30852
c092355
to
5c1160a
Compare
Hum - changing the priority of the configurations as described above is much trickier than I expected. Looks like an existing Gradle integration tests are passing locally. Opened #31165 to replace this one, because this PR is based on 2.16 |
Currently the
QuarkusBuild
task implementation adds even large build artifacts and unmodified dependency jars to Gradle's build cache. This pollutes the Gradle build cache quite a lot and therefore lets archived build caches become unnecessary huge.This change updates the build logic to fix this behavior by not adding dependencies and large build artifacts, like uber-jar and native binary, to the Gradle build cache.
The
QuarkusBuild
task has been split into three tasks:QuarkusBuildDependencies
task that only collects the contents forbuild/quarkus-app/lib
QuarkusBuildApp
task that to collect everything else from a Quarkus build (everything except thebuild/quarkus-app/lib
)QuarkusBuild
task now combines the outputs of the above two tasksQuarkusBuildDependencies
(named 'quarkusDependenciesBuild) is not cacheable, because it only collects dependencies, which come either from a repository (and are already available locally elsewhere) or are built by other Gradle tasks. This task is only executed if the configured Quarkus package type requires the "quarkus-app" directory (
fast-jar+
native). It's "build working directory" is
build/quarkus-build/dep`.QuarkusBuildApp
(namedquarkusAppBuild
) collects the contents of the "quarkus-app" directory excluding thelib/
directory are cacheable, which is the default for CI environments. Non-CI environments still cache all outputs, even uber-jars and native binaries to retain the existing behavior for developers and keep build turn-around times low. CI environments can opt-in to add even huge artifacts to Gradle's build cache by explicitly setting thecacheUberAndNativeRunners
property in the Quarkus extension totrue
. It's "build working directory" isbuild/quarkus-build/app
.Since
QuarkusBuild
only combines the outputs of the above two tasks, the same "CI vs local" caching behavior as for theQuarkusBuildApp
task applies. To make "up to date" checks (kind of) more reliable, all outputs are removed first. This means, that for example an existing uber-jar inbuild/
will disappear, when the build's package type is "fast-jar". This behavior can be disabled by setting thecleanupBuildOutput
property on the Quarkus extension tofalse
.Both
QuarkusBuildDependencies
andQuarkusBuildApp
can trigger an actual Quarkus application build. That Quarkus app build will only be triggered when needed and only once per Gradle build.The task names
quarkusDependenciesBuild
andquarkusAppBuild
are intentionally "that way around". Letting the names of these tasks begin withquarkusBuild...
could confuse users, who use abbreviated task names on the command line (for example./gradlew qB
is automagically expanded to./gradlew quarkusBuild
, which would become ambiguous withquarkusBuildDependencies
andquarkusBuildApp
).Relates to: #30852
Gradle caching in CI
Unless the
cacheLargeArtifacts
property on thequarkus
extension is set totrue
, the output of/for the package typefast-jar
minus the dependency jars is cached by Gradle's build cache (similar forlegacy-jar
).Gradle caching (non CI)
Basically everything is cacheable to allow fast(er) local development turn-around cycles.