Skip to content

Commit

Permalink
Make QuarkusBuild not pollute Gradle's build cache
Browse files Browse the repository at this point in the history
Improve Quarkus build in Gradle, proper up-to-date mechanisms, right config-source priorities.

Currently the `QuarkusBuild` task implementation adds even large build artifacts and unmodified dependency jars to Gradle's build cache. This pollutes the Gradle build cache quite a lot and therefore lets archived build caches become unnecessary huge.
On top of that, "relicts" from previous runs of `QuarkusBuild` using a different configuration, like a different package-type, causes multiple permutations of the cached artifacts and leads to restoring unexpected artifacts from the build cache.

This large change updates the Gradle logic to have "proper" up-to-date checks and in turn a reasonable setting of what is cached and what is not cached.

Background on how I discovered the "caching and up-to-date issue" with `QuarkusBuild`: Quarkus 2.16 added the `@CacheableTask` annotation to `QuarkusBuild`. This means that _everything_ that is generated by `QuarkusBuild` (and "declared" as an output: fast-jar directory, uber-jar, native-runner) is cached. After having Quarkus 2.16.x in our "main" branch for about 1/2 day, nearly the whole "10 GB budget" for cache artifacts in GitHub was occupied by Gradle build cache artifacts - the build cache artifacts grew from build to build - this was not the case without the `@CacheableTask` annotation.

The current inputs of the `QuarkusBuild` task are also incomplete. Those inputs do not include for example the configured `finalName`, the settings on the Gradle project and the settings on the Quarkus extension. This leads to situations that the wrong Gradle cache artifacts could be restored (think: you ask for a fast-jar, but get an uber-jar).

Another issue is that repeated runs of `quarkusBuild` against the exact same code base and dependencies leads to different outputs, the contents of the generated jars and e.g. quarkus-application.dat differ. This causes different outputs of the `QuarkusBuild` task, which cause depending tasks and projects to unnecessarily run/work again.

The initial goal of this effort was to make caching of `fast-jar` builds work nice in CI environments, I assume that's the most common package type used in CI environments for example for testing purposes. The idea here is to _not_ cache all the dependency jars, but leverage the mechanisms available in Gradle to "just collect" the dependencies. In other words (and simplified): for `fast-jar`, cache everthing "in `quarkus-app` except `lib/`.

There is no change for users running a `./gradlew quarkusBuild`. `QuarkusBuild` is still the task that users should execute (or depend on in their builds).

To achive the goal to make especially `fast-jar` builds "nicely cacheable" in CI, two new tasks had to be introduced:

* `QuarkusBuildDependencies` uses the dependency information from the Quarkus `ApplicationModel` to tell Gradle to collect those. This task works for `fast-jar` and `legacy-jar` and is a "no op" for other package types. This task is _never_ cacheable, but has _up-to-date_ checks.
* `QuarkusBuildCacheableAppParts` performs a Quarkus application build and collects the _cacheable_ parts of that build. This task works for `fast-jar` and `legacy-jar` and is a "no op" for other package types. This task is cacheable.
* `QuarkusBuild`, for `fast-jar` and `legacy-jar`, assembles the outputs of the above two tasks to produce the expected output. Performs a Quarkus build for other package types. See below on when this task is cacheable.

The `build/quarkus-build/` directory is used to properly separate the outputs of the above three tasks:

* `build/quarkus-build/gen/` receives the full output of a Quarkus build
* `build/quarkus-build/app/` receives the _cacheable_ parts from `build/quarkus-build/gen/`
* `build/quarkus-build/dep/` receives the dependency jars

The output of `QuarkusBuild` is, by default, cacheable in non-CI environments and _not_ cacheable in CI environments. The behavior can be explicitly overridden using the new property `cacheLargeArtifacts` property. As outlined above, caching huge artifacts is not beneficial in CI, either due to space limitations (GitHub's 10GB limit for example) or just the cost of network traffic/duration. On a developer's machine however, the cost of storing/retrieving even bigger artifacts is rather neglectible and lower than rebuilding a Quarkus application.

Before this change, the "priority" of configuration settings was (in most cases...) effectively: System properties, environment variables, `application.properties`, configurations in Gradle build scripts & Gradle project properties. This order is unintuitive and was changed to: system properties, environment variables, Gradle project properties, Quarkus build configuration, application.properties.  The previous code had several places in which Quarkus related configuration settings were retrieved with sometimes different priorities of the "config sources".

The whole "machinery" to get the configuration has been rewritten and encapsulated in new classes `EffectiveConfig` (effective for a Quarkus build) and `BaseConfig` (available during Gradle task configuration phase). `EffectiveConfig` is not really different from `BaseConfig, but contains the "special" settings from e.g. `ImagePush` task.

The new implementation also uses SmallRye Config and especially the existing Quarkus mechanisms to pull information out of a `PackageConfig` _object_ produced from a configuration. It turned out to be easier to reason about `PackageConfig` than "raw property values".

Support for `application.yaml/yml` has also been added.

All calls into the "inner workings" of a Quarkus build and Quarkus code generation have been moved to separate worker processes. The code in this change to support this is pretty dead simple.

The primary reason to move that work into isolated worker processes is that all the `QuarkusBootstrap` and derived pieces know nothing about Gradle, so there is no "property like" mechanism to override settings from `application.properties/yaml/yml` with those from e.g. the `QuarkusPluginExtension`. The only option would have been to modify the system properties of the Gradle build process - but that's a no-go, especially considering other tasks running in parallel (think: two `QuarkusBuild` of different projects running at the same time). It would have been relatively easy to serialize all `QuarkusBuild` actions across a Gradle build, but why - and it would prevent using "beefy machines" to run many `QuarkusBuild`s in parallel. Another option would have been to implement a rather complex (and likely very racy) mechanism to track modifications to system properties.

As a conclusion, it wasn't just very simple to leverage the process isolation using the Gradle worker API, but it's also not bad. Gradle does reuse already spawned and compatible worker instances.

The "trick" implemented to "prioritize" configs from Gradle project properties and Quarkus extension settings is to pass the whole config as system properties to the worker.

A bunch of hopefully useful logging has been implemented and added.

With info-level loogging, the tasks emit at least some basic information about what they do and at least the package type being used.

To be able to investigate which configuration settings were effectively used, there's another new task called `quarkusShowEffectiveConfig` that shows all the `quarkus.*` properties and some more information, including the loaded `application.(properties|yaml|yml)`. This task is intended to debug build issues. The task can optionally save the effecitve config properties as a properties file in the `build/` directory, if run with the command line option `./gradlew quarkusShowEffectiveConfig --save-config-properties`.

None of the existing tests has changed, except the `BuildCOnfigurationTest` had to be adopted to reflect the updated order of config sources.

Relates to: quarkusio#30852
  • Loading branch information
snazy authored and holly-cummins committed Apr 20, 2023
1 parent 7554229 commit 95423ec
Show file tree
Hide file tree
Showing 81 changed files with 2,524 additions and 581 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,15 @@ private BuiltInType(final String name) {
public String toString() {
return name;
}

public static BuiltInType fromString(String name) {
for (PackageConfig.BuiltInType type : values()) {
if (type.toString().equals(name)) {
return type;
}
}
throw new IllegalArgumentException("Unknown Quarkus package type '" + name + "'");
}
}

/**
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
package io.quarkus.deployment.pkg;

import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatIllegalArgumentException;

import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.EnumSource;

public class BuiltInTypeTest {

@ParameterizedTest
@EnumSource(PackageConfig.BuiltInType.class)
void packageTypeConversion(PackageConfig.BuiltInType packageType) {
assertThat(PackageConfig.BuiltInType.fromString(packageType.toString())).isSameAs(packageType);
}

@Test
void invalidPackageType() {
assertThatIllegalArgumentException().isThrownBy(() -> PackageConfig.BuiltInType.fromString("not-a-package-type"))
.withMessage("Unknown Quarkus package type 'not-a-package-type'");
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,10 @@ public static <T> T handleObject(Supplier<T> supplier) {

public static void handleObject(Object o) {
final SmallRyeConfig config = (SmallRyeConfig) ConfigProvider.getConfig();
handleObject(o, config);
}

public static void handleObject(Object o, SmallRyeConfig config) {
final String clsNameSuffix = getClassNameSuffix(o);
if (clsNameSuffix == null) {
// unsupported object type
Expand Down Expand Up @@ -190,7 +194,11 @@ private static Converter<?> getConverterFor(Type type, SmallRyeConfig config) {
} else if (rawType == Optional.class) {
return Converters.newOptionalConverter(getConverterFor(typeOfParameter(type, 0), config));
} else if (rawType == List.class) {
return Converters.newCollectionConverter(getConverterFor(typeOfParameter(type, 0), config), ArrayList::new);
return Converters.newCollectionConverter(getConverterFor(typeOfParameter(type, 0), config),
ConfigUtils.listFactory());
} else if (rawType == Set.class) {
return Converters.newCollectionConverter(getConverterFor(typeOfParameter(type, 0), config),
ConfigUtils.setFactory());
} else {
return config.requireConverter(rawTypeOf(type));
}
Expand Down
1 change: 1 addition & 0 deletions devtools/gradle/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ subprojects {

javadoc {
options.addStringOption('encoding', 'UTF-8')
options.addStringOption("Xdoclint:-reference", "-quiet")
}
}

Expand Down
1 change: 1 addition & 0 deletions devtools/gradle/gradle-application-plugin/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ dependencies {
implementation "io.quarkus:quarkus-devtools-common:${version}"
implementation "io.quarkus:quarkus-core-deployment:${version}"
implementation "io.quarkus:quarkus-bootstrap-gradle-resolver:${version}"
implementation "io.smallrye.config:smallrye-config-source-yaml:${smallrye_config_version}"

implementation project(":gradle-model")

Expand Down
4 changes: 4 additions & 0 deletions devtools/gradle/gradle-application-plugin/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,10 @@
<artifactId>quarkus-devmode-test-utils</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.smallrye.config</groupId>
<artifactId>smallrye-config-source-yaml</artifactId>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-gradle-plugin</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,8 @@
import io.quarkus.gradle.tasks.ImagePush;
import io.quarkus.gradle.tasks.QuarkusAddExtension;
import io.quarkus.gradle.tasks.QuarkusBuild;
import io.quarkus.gradle.tasks.QuarkusBuildConfiguration;
import io.quarkus.gradle.tasks.QuarkusBuildCacheableAppParts;
import io.quarkus.gradle.tasks.QuarkusBuildDependencies;
import io.quarkus.gradle.tasks.QuarkusDev;
import io.quarkus.gradle.tasks.QuarkusGenerateCode;
import io.quarkus.gradle.tasks.QuarkusGoOffline;
Expand All @@ -47,6 +48,7 @@
import io.quarkus.gradle.tasks.QuarkusListPlatforms;
import io.quarkus.gradle.tasks.QuarkusRemoteDev;
import io.quarkus.gradle.tasks.QuarkusRemoveExtension;
import io.quarkus.gradle.tasks.QuarkusShowEffectiveConfig;
import io.quarkus.gradle.tasks.QuarkusTest;
import io.quarkus.gradle.tasks.QuarkusTestConfig;
import io.quarkus.gradle.tasks.QuarkusUpdate;
Expand All @@ -56,7 +58,8 @@
public class QuarkusPlugin implements Plugin<Project> {

public static final String ID = "io.quarkus";
public static final String QUARKUS_PACKAGE_TYPE = "quarkus.package.type";
public static final String DEFAULT_OUTPUT_DIRECTORY = "quarkus-app";
public static final String QUARKUS_PACKAGE_TYPE = "quarkus.package.type"; // constant left here to not break potential users

public static final String EXTENSION_NAME = "quarkus";
public static final String LIST_EXTENSIONS_TASK_NAME = "listExtensions";
Expand All @@ -67,6 +70,9 @@ public class QuarkusPlugin implements Plugin<Project> {
public static final String QUARKUS_GENERATE_CODE_TASK_NAME = "quarkusGenerateCode";
public static final String QUARKUS_GENERATE_CODE_DEV_TASK_NAME = "quarkusGenerateCodeDev";
public static final String QUARKUS_GENERATE_CODE_TESTS_TASK_NAME = "quarkusGenerateCodeTests";
public static final String QUARKUS_BUILD_DEP_TASK_NAME = "quarkusDependenciesBuild";
public static final String QUARKUS_BUILD_APP_PARTS_TASK_NAME = "quarkusAppPartsBuild";
public static final String QUARKUS_SHOW_EFFECTIVE_CONFIG_TASK_NAME = "quarkusShowEffectiveConfig";
public static final String QUARKUS_BUILD_TASK_NAME = "quarkusBuild";
public static final String QUARKUS_DEV_TASK_NAME = "quarkusDev";
public static final String QUARKUS_REMOTE_DEV_TASK_NAME = "quarkusRemoteDev";
Expand All @@ -80,6 +86,7 @@ public class QuarkusPlugin implements Plugin<Project> {

@Deprecated
public static final String BUILD_NATIVE_TASK_NAME = "buildNative";
@Deprecated
public static final String TEST_NATIVE_TASK_NAME = "testNative";

@Deprecated
Expand All @@ -98,6 +105,7 @@ public class QuarkusPlugin implements Plugin<Project> {

private final ToolingModelBuilderRegistry registry;

@SuppressWarnings("CdiInjectionPointsInspection")
@Inject
public QuarkusPlugin(ToolingModelBuilderRegistry registry) {
this.registry = registry;
Expand Down Expand Up @@ -150,27 +158,40 @@ private void registerTasks(Project project, QuarkusPluginExtension quarkusExt) {
TaskProvider<QuarkusGenerateCode> quarkusGenerateCodeTests = tasks.register(QUARKUS_GENERATE_CODE_TESTS_TASK_NAME,
QuarkusGenerateCode.class, task -> task.setTest(true));

QuarkusBuildConfiguration buildConfig = new QuarkusBuildConfiguration(project);
tasks.register(QUARKUS_SHOW_EFFECTIVE_CONFIG_TASK_NAME,
QuarkusShowEffectiveConfig.class, task -> {
task.setDescription("Show effective Quarkus build configuration.");
});

TaskProvider<QuarkusBuildDependencies> quarkusBuildDependencies = tasks.register(QUARKUS_BUILD_DEP_TASK_NAME,
QuarkusBuildDependencies.class,
task -> task.getOutputs().doNotCacheIf("Dependencies are never cached", t -> true));

TaskProvider<QuarkusBuildCacheableAppParts> quarkusBuildCacheableAppParts = tasks.register(
QUARKUS_BUILD_APP_PARTS_TASK_NAME,
QuarkusBuildCacheableAppParts.class, task -> {
task.dependsOn(quarkusGenerateCode);
task.getOutputs().doNotCacheIf(
"Not adding uber-jars, native binaries and mutable-jar package type to Gradle " +
"build cache by default. To allow caching of uber-jars, native binaries and mutable-jar " +
"package type, set 'cacheUberAndNativeRunners' in the 'quarkus' Gradle extension to 'true'.",
t -> !task.isCachedByDefault() && !quarkusExt.getCacheLargeArtifacts().get());
});

TaskProvider<QuarkusBuild> quarkusBuild = tasks.register(QUARKUS_BUILD_TASK_NAME, QuarkusBuild.class, build -> {
build.dependsOn(quarkusGenerateCode);
build.getForcedProperties().set(buildConfig.getForcedProperties());
build.dependsOn(quarkusBuildDependencies, quarkusBuildCacheableAppParts);
build.getOutputs().doNotCacheIf(
"Only collects and combines the outputs of " + QUARKUS_BUILD_APP_PARTS_TASK_NAME + " and "
+ QUARKUS_BUILD_DEP_TASK_NAME + ", see 'cacheLargeArtifacts' in the 'quarkus' Gradle extension " +
"for details.",
t -> !quarkusExt.getCacheLargeArtifacts().get());
});

TaskProvider<ImageBuild> imageBuild = tasks.register(IMAGE_BUILD_TASK_NAME, ImageBuild.class, buildConfig);
imageBuild.configure(task -> {
task.finalizedBy(quarkusBuild);
});
tasks.register(IMAGE_BUILD_TASK_NAME, ImageBuild.class, task -> task.finalizedBy(quarkusBuild));

TaskProvider<ImagePush> imagePush = tasks.register(IMAGE_PUSH_TASK_NAME, ImagePush.class, buildConfig);
imagePush.configure(task -> {
task.finalizedBy(quarkusBuild);
});
tasks.register(IMAGE_PUSH_TASK_NAME, ImagePush.class, task -> task.finalizedBy(quarkusBuild));

TaskProvider<Deploy> deploy = tasks.register(DEPLOY_TASK_NAME, Deploy.class, buildConfig);
deploy.configure(task -> {
task.finalizedBy(quarkusBuild);
});
tasks.register(DEPLOY_TASK_NAME, Deploy.class, task -> task.finalizedBy(quarkusBuild));

TaskProvider<QuarkusDev> quarkusDev = tasks.register(QUARKUS_DEV_TASK_NAME, QuarkusDev.class, devRuntimeDependencies,
quarkusExt);
Expand Down Expand Up @@ -249,7 +270,7 @@ private void registerTasks(Project project, QuarkusPluginExtension quarkusExt) {
quarkusGenerateCode,
quarkusGenerateCodeTests);
});
quarkusBuild.configure(
quarkusBuildCacheableAppParts.configure(
task -> task.dependsOn(classesTask, resourcesTask, tasks.named(JavaPlugin.JAR_TASK_NAME)));

SourceSetContainer sourceSets = project.getExtensions().getByType(SourceSetContainer.class);
Expand Down Expand Up @@ -317,7 +338,7 @@ public void execute(Task task) {
t.useJUnitPlatform();
});
// quarkusBuild is expected to run after the project has passed the tests
quarkusBuild.configure(task -> task.shouldRunAfter(tasks.withType(Test.class)));
quarkusBuildCacheableAppParts.configure(task -> task.shouldRunAfter(tasks.withType(Test.class)));

SourceSet generatedSourceSet = sourceSets.create(QuarkusGenerateCode.QUARKUS_GENERATED_SOURCES);
SourceSet generatedTestSourceSet = sourceSets.create(QuarkusGenerateCode.QUARKUS_TEST_GENERATED_SOURCES);
Expand Down Expand Up @@ -389,6 +410,11 @@ private void configureBuildNativeTask(Project project) {
project.getGradle().getTaskGraph().whenReady(taskGraph -> {
if (taskGraph.hasTask(project.getPath() + BUILD_NATIVE_TASK_NAME)
|| taskGraph.hasTask(project.getPath() + TEST_NATIVE_TASK_NAME)) {
// Nag user
project.getLogger().warn("The Quarkus tasks {} and {} are deprecated and subject to removal. " +
"Please migrate your build to use 'test -Dquarkus.package.type=native' and " +
"'quarkusBuild -Dquarkus.package.type=native'.",
TEST_NATIVE_TASK_NAME, BUILD_NATIVE_TASK_NAME);
project.getExtensions().getExtraProperties()
.set(QUARKUS_PACKAGE_TYPE, "native");
}
Expand Down
Loading

0 comments on commit 95423ec

Please sign in to comment.