Skip to content

Commit

Permalink
WIP: Make QuarkusBuild not pollute Gradle's build cache
Browse files Browse the repository at this point in the history
Currently the `QuarkusBuild` task implementation adds even large build artifacts and unmodified dependency jars to Gradle's build cache. This pollutes the Gradle build cache quite a lot and therefore lets archived build caches become unnecessary huge.

This change updates the build logic to fix this behavior by not adding dependencies and large build artifacts, like uber-jar and native binary, to the Gradle build cache.

The `QuarkusBuild` task has been split into three tasks:
1. a new `QuarkusBuildDependencies` task that only collects the contents for `build/quarkus-app/lib`
2. a new `QuarkusBuildApp` task that to collect everything else from a Quarkus build (everything except the `build/quarkus-app/lib`)
3. the `QuarkusBuild` task now combines the outputs of the above two tasks

`QuarkusBuildDependencies` (named 'quarkusDependenciesBuild`) is not cacheable, because it only collects dependencies, which come either from a repository (and are already available locally elsewhere) or are built by other Gradle tasks. This task is only executed if the configured Quarkus package type requires the "quarkus-app" directory (`fast-jar` + `native`). It's "build working directory" is `build/quarkus-build/dep`.

`QuarkusBuildApp` (named `quarkusAppBuild`) collects the contents of the "quarkus-app" directory _excluding_ the `lib/` directory are cacheable, which is the default for CI environments. Non-CI environments still cache all outputs, even uber-jars and native binaries to retain the existing behavior for developers and keep build turn-around times low. CI environments can opt-in to add even huge artifacts to Gradle's build cache by explicitly setting the `cacheUberAndNativeRunners` property in the Quarkus extension to `true`. It's "build working directory" is `build/quarkus-build/app`.

Since `QuarkusBuild` only combines the outputs of the above two tasks, the same "CI vs local" caching behavior as for the `QuarkusBuildApp` task applies. To make "up to date" checks (kind of) more reliable, all outputs are removed first. This means, that for example an existing uber-jar in `build/` will disappear, when the build's package type is "fast-jar". This behavior can be disabled by setting the `cleanupBuildOutput` property on the Quarkus extension to `false`.

Both `QuarkusBuildDependencies` and `QuarkusBuildApp` can trigger an actual Quarkus application build. That Quarkus app build will only be triggered when needed and only once per Gradle build.

The task names `quarkusDependenciesBuild` and `quarkusAppBuild` are intentionally "that way around". Letting the names of these tasks begin with `quarkusBuild...` could confuse users, who use abbreviated task names on the command line (for example `./gradlew qB` is automagically expanded to `./gradlew quarkusBuild`, which would become ambiguous with `quarkusBuildDependencies` and `quarkusBuildApp`).

Unless the `cacheLargeArtifacts` property on the `quarkus` extension is set to `true`, the output of/for the package type `fast-jar` minus the dependency jars is cached by Gradle's build cache (similar for `legacy-jar`).

Basically everything is cacheable to allow fast(er) local development turn-around cycles.

To be able to investigate which configuration settings were effectively used, there's another new task called `quarkusShowEffectiveConfig` that shows all the `quarkus.*` properties and some more information, including the loaded `application.(properties|yaml|yml)`. This task is intended to debug build issues. The task can optionally save the effecitve config properties as a properties file in the `build/` directory, if run with the command line option `./gradlew quarkusShowEffectiveConfig --save-config-properties`.

Relates to: #30852
  • Loading branch information
snazy committed Feb 14, 2023
1 parent 07fd942 commit 5c1160a
Show file tree
Hide file tree
Showing 39 changed files with 1,535 additions and 280 deletions.
1 change: 1 addition & 0 deletions devtools/gradle/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ subprojects {

javadoc {
options.addStringOption('encoding', 'UTF-8')
options.addStringOption("Xdoclint:-reference", "-quiet")
}
}

Expand Down
1 change: 1 addition & 0 deletions devtools/gradle/gradle-application-plugin/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ dependencies {
implementation "io.quarkus:quarkus-devtools-common:${version}"
implementation "io.quarkus:quarkus-core-deployment:${version}"
implementation "io.quarkus:quarkus-bootstrap-gradle-resolver:${version}"
implementation "io.smallrye.config:smallrye-config-source-yaml:${smallrye_config_version}"

implementation project(":gradle-model")

Expand Down
4 changes: 4 additions & 0 deletions devtools/gradle/gradle-application-plugin/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,10 @@
<artifactId>quarkus-devmode-test-utils</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.smallrye.config</groupId>
<artifactId>smallrye-config-source-yaml</artifactId>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-gradle-plugin</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,9 @@
import io.quarkus.gradle.tasks.ImagePush;
import io.quarkus.gradle.tasks.QuarkusAddExtension;
import io.quarkus.gradle.tasks.QuarkusBuild;
import io.quarkus.gradle.tasks.QuarkusBuildApp;
import io.quarkus.gradle.tasks.QuarkusBuildConfiguration;
import io.quarkus.gradle.tasks.QuarkusBuildDependencies;
import io.quarkus.gradle.tasks.QuarkusDev;
import io.quarkus.gradle.tasks.QuarkusGenerateCode;
import io.quarkus.gradle.tasks.QuarkusGoOffline;
Expand All @@ -46,6 +48,7 @@
import io.quarkus.gradle.tasks.QuarkusListPlatforms;
import io.quarkus.gradle.tasks.QuarkusRemoteDev;
import io.quarkus.gradle.tasks.QuarkusRemoveExtension;
import io.quarkus.gradle.tasks.QuarkusShowEffectiveConfig;
import io.quarkus.gradle.tasks.QuarkusTest;
import io.quarkus.gradle.tasks.QuarkusTestConfig;
import io.quarkus.gradle.tasks.QuarkusUpdate;
Expand All @@ -55,7 +58,17 @@
public class QuarkusPlugin implements Plugin<Project> {

public static final String ID = "io.quarkus";
public static final String QUARKUS_PROFILE = "quarkus.profile";
public static final String DEFAULT_PROFILE = "prod";
public static final String QUARKUS_PACKAGE_OUTPUT_NAME = "quarkus.package.output-name";
public static final String QUARKUS_PACKAGE_ADD_RUNNER_SUFFIX = "quarkus.package.add-runner-suffix";
public static final String QUARKUS_PACKAGE_TYPE = "quarkus.package.type";
public static final String DEFAULT_PACKAGE_TYPE = "jar";
public static final String OUTPUT_DIRECTORY = "quarkus.package.output-directory";
public static final String DEFAULT_OUTPUT_DIRECTORY = "quarkus-app";
public static final String CLASS_LOADING_REMOVED_ARTIFACTS = "quarkus.class-loading.removed-artifacts";
public static final String CLASS_LOADING_PARENT_FIRST_ARTIFACTS = "quarkus.class-loading.parent-first-artifacts";
public static final String QUARKUS_ARTIFACT_PROPERTIES = "quarkus-artifact.properties";

public static final String EXTENSION_NAME = "quarkus";
public static final String LIST_EXTENSIONS_TASK_NAME = "listExtensions";
Expand All @@ -66,6 +79,9 @@ public class QuarkusPlugin implements Plugin<Project> {
public static final String QUARKUS_GENERATE_CODE_TASK_NAME = "quarkusGenerateCode";
public static final String QUARKUS_GENERATE_CODE_DEV_TASK_NAME = "quarkusGenerateCodeDev";
public static final String QUARKUS_GENERATE_CODE_TESTS_TASK_NAME = "quarkusGenerateCodeTests";
public static final String QUARKUS_BUILD_DEP_TASK_NAME = "quarkusDependenciesBuild";
public static final String QUARKUS_BUILD_APP_TASK_NAME = "quarkusAppBuild";
public static final String QUARKUS_SHOW_EFFECTIVE_CONFIG_TASK_NAME = "quarkusShowEffectiveConfig";
public static final String QUARKUS_BUILD_TASK_NAME = "quarkusBuild";
public static final String QUARKUS_DEV_TASK_NAME = "quarkusDev";
public static final String QUARKUS_REMOTE_DEV_TASK_NAME = "quarkusRemoteDev";
Expand Down Expand Up @@ -96,6 +112,7 @@ public class QuarkusPlugin implements Plugin<Project> {

private final ToolingModelBuilderRegistry registry;

@SuppressWarnings("CdiInjectionPointsInspection")
@Inject
public QuarkusPlugin(ToolingModelBuilderRegistry registry) {
this.registry = registry;
Expand Down Expand Up @@ -148,11 +165,38 @@ private void registerTasks(Project project, QuarkusPluginExtension quarkusExt) {
TaskProvider<QuarkusGenerateCode> quarkusGenerateCodeTests = tasks.register(QUARKUS_GENERATE_CODE_TESTS_TASK_NAME,
QuarkusGenerateCode.class, task -> task.setTest(true));

QuarkusBuildConfiguration buildConfig = new QuarkusBuildConfiguration(project);
QuarkusBuildConfiguration buildConfig = new QuarkusBuildConfiguration(project, quarkusExt);

TaskProvider<QuarkusBuild> quarkusBuild = tasks.register(QUARKUS_BUILD_TASK_NAME, QuarkusBuild.class, build -> {
build.dependsOn(quarkusGenerateCode);
build.getForcedProperties().set(buildConfig.getForcedProperties());
TaskProvider<QuarkusBuildDependencies> quarkusBuildDep = tasks.register(QUARKUS_BUILD_DEP_TASK_NAME,
QuarkusBuildDependencies.class, buildConfig);
quarkusBuildDep.configure(task -> {
task.getOutputs().doNotCacheIf("Dependencies are never cached", t -> true);
});

TaskProvider<QuarkusShowEffectiveConfig> quarkusShowConfig = tasks.register(QUARKUS_SHOW_EFFECTIVE_CONFIG_TASK_NAME,
QuarkusShowEffectiveConfig.class, buildConfig);
quarkusShowConfig.configure(task -> {
task.setDescription("Show effective Quarkus build configuration.");
});

TaskProvider<QuarkusBuildApp> quarkusBuildApp = tasks.register(QUARKUS_BUILD_APP_TASK_NAME,
QuarkusBuildApp.class, buildConfig);
quarkusBuildApp.configure(task -> {
task.dependsOn(quarkusGenerateCode);
task.getOutputs().doNotCacheIf(
"Not adding uber-jars, native binaries and mutable-jar package type to Gradle " +
"build cache by default. To allow caching of uber-jars, native binaries and mutable-jar " +
"package type, set 'cacheUberAndNativeRunners' in the 'quarkus' Gradle extension to 'true'.",
t -> !task.isCachedByDefault() && !quarkusExt.getCacheLargeArtifacts().get());
});

TaskProvider<QuarkusBuild> quarkusBuild = tasks.register(QUARKUS_BUILD_TASK_NAME, QuarkusBuild.class, buildConfig);
quarkusBuild.configure(build -> {
build.dependsOn(quarkusBuildDep, quarkusBuildApp);
build.getOutputs().doNotCacheIf(
"Collects and combines the outputs of " + QUARKUS_BUILD_APP_TASK_NAME + " and "
+ QUARKUS_BUILD_DEP_TASK_NAME,
t -> !quarkusExt.getCacheLargeArtifacts().get());
});

TaskProvider<ImageBuild> imageBuild = tasks.register(IMAGE_BUILD_TASK_NAME, ImageBuild.class, buildConfig);
Expand Down Expand Up @@ -242,7 +286,7 @@ private void registerTasks(Project project, QuarkusPluginExtension quarkusExt) {
quarkusGenerateCode,
quarkusGenerateCodeTests);
});
quarkusBuild.configure(
quarkusBuildApp.configure(
task -> task.dependsOn(classesTask, resourcesTask, tasks.named(JavaPlugin.JAR_TASK_NAME)));

SourceSetContainer sourceSets = project.getExtensions().getByType(SourceSetContainer.class);
Expand Down Expand Up @@ -310,7 +354,7 @@ public void execute(Task task) {
t.useJUnitPlatform();
});
// quarkusBuild is expected to run after the project has passed the tests
quarkusBuild.configure(task -> task.shouldRunAfter(tasks.withType(Test.class)));
quarkusBuildApp.configure(task -> task.shouldRunAfter(tasks.withType(Test.class)));

SourceSet generatedSourceSet = sourceSets.create(QuarkusGenerateCode.QUARKUS_GENERATED_SOURCES);
SourceSet generatedTestSourceSet = sourceSets.create(QuarkusGenerateCode.QUARKUS_TEST_GENERATED_SOURCES);
Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
package io.quarkus.gradle.extension;

import static io.quarkus.gradle.QuarkusPlugin.QUARKUS_PACKAGE_ADD_RUNNER_SUFFIX;
import static io.quarkus.gradle.QuarkusPlugin.QUARKUS_PACKAGE_OUTPUT_NAME;

import java.io.File;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.LinkedHashSet;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.Properties;
import java.util.Set;
import java.util.StringJoiner;
import java.util.stream.Collectors;
Expand Down Expand Up @@ -42,12 +45,20 @@ public class QuarkusPluginExtension {
private final MapProperty<String, String> quarkusBuildProperties;
private final SourceSetExtension sourceSetExtension;

private final Property<Boolean> cacheLargeArtifacts;
private final Property<Boolean> cleanupBuildOutput;

public QuarkusPluginExtension(Project project) {
this.project = project;

finalName = project.getObjects().property(String.class);
finalName.convention(project.provider(() -> String.format("%s-%s", project.getName(), project.getVersion())));

this.cleanupBuildOutput = project.getObjects().property(Boolean.class)
.convention(true);
this.cacheLargeArtifacts = project.getObjects().property(Boolean.class)
.convention(!System.getenv().containsKey("CI"));

this.sourceSetExtension = new SourceSetExtension();
this.quarkusBuildProperties = project.getObjects().mapProperty(String.class, String.class);
}
Expand Down Expand Up @@ -96,34 +107,66 @@ public void beforeTest(Test task) {
}
}

public String buildNativeRunnerName(final Map<String, Object> taskSystemProps) {
Properties properties = new Properties(taskSystemProps.size());
properties.putAll(taskSystemProps);
quarkusBuildProperties.get().entrySet()
.forEach(buildEntry -> properties.putIfAbsent(buildEntry.getKey(), buildEntry.getValue()));
System.getProperties().entrySet()
.forEach(propEntry -> properties.putIfAbsent(propEntry.getKey(), propEntry.getValue()));
System.getenv().entrySet().forEach(
envEntry -> properties.putIfAbsent(envEntry.getKey(), envEntry.getValue()));
StringBuilder nativeRunnerName = new StringBuilder();

if (properties.containsKey("quarkus.package.output-name")) {
nativeRunnerName.append(properties.get("quarkus.package.output-name"));
} else {
nativeRunnerName.append(finalName());
public String resolveBuildProperty(String propertyKey, Map<String, Object> taskSystemProps, String defaultValue) {
Object v = taskSystemProps.get(propertyKey);
if (v instanceof String) {
return v.toString();
}
if (!properties.containsKey("quarkus.package.add-runner-suffix")
|| (properties.containsKey("quarkus.package.add-runner-suffix")
&& Boolean.parseBoolean((String) properties.get("quarkus.package.add-runner-suffix")))) {
nativeRunnerName.append("-runner");
String s = quarkusBuildProperties.get().get(propertyKey);
if (s != null) {
return s;
}
return nativeRunnerName.toString();
s = System.getProperty(propertyKey);
if (s != null) {
return s;
}
s = System.getenv(propertyKey.toUpperCase(Locale.ROOT).replace('.', '_'));
if (s != null) {
return s;
}
return defaultValue;
}

public String buildNativeRunnerBaseName(Map<String, Object> taskSystemProps) {
return resolveBuildProperty(QUARKUS_PACKAGE_OUTPUT_NAME, taskSystemProps, finalName());
}

public String buildNativeRunnerName(Map<String, Object> taskSystemProps) {
String outputName = buildNativeRunnerBaseName(taskSystemProps);
if (Boolean.parseBoolean(resolveBuildProperty(QUARKUS_PACKAGE_ADD_RUNNER_SUFFIX, taskSystemProps, "true"))) {
return outputName + "-runner";
}
return outputName;
}

public Property<String> getFinalName() {
return finalName;
}

/**
* Whether the build output, build/*-runner[.jar] and build/quarkus-app, for other package types than the
* currently configured one are removed, default is 'true'.
*/
public Property<Boolean> getCleanupBuildOutput() {
return cleanupBuildOutput;
}

public void setCleanupBuildOutput(boolean cleanupBuildOutput) {
this.cleanupBuildOutput.set(cleanupBuildOutput);
}

/**
* Whether large build artifacts, like uber-jar and native runners, are cached. Defaults to 'false' if the 'CI' environment
* variable is set, otherwise defaults to 'true'.
*/
public Property<Boolean> getCacheLargeArtifacts() {
return cacheLargeArtifacts;
}

public void setCacheLargeArtifacts(boolean cacheLargeArtifacts) {
this.cacheLargeArtifacts.set(cacheLargeArtifacts);
}

public String finalName() {
return getFinalName().get();
}
Expand Down
Loading

0 comments on commit 5c1160a

Please sign in to comment.