Skip to content
This repository has been archived by the owner on Aug 5, 2022. It is now read-only.

Commit

Permalink
Merge pull request apache#4 in ABFX/maven-forked from incremental-mav…
Browse files Browse the repository at this point in the history 
…en-3.6.3 to MNG-7129/maven-3.6.3-incremental

Squashed commit of the following:

commit 21c620a34ddbbf84b3c44014b069a7b222cf91a7
Author: Maximilian Novikov <maximilian.novikov@db.com>
Date:   Tue Aug 10 19:41:41 2021 +0200

    Merge pull request apache#3 in ABFX/maven-forked from code-cleanup-2 to incremental-maven-3.6.3

    Squashed commit of the following:

    commit 3401739ec9eb957fb03887f6a3c24477c9c3327b
    Author: Alexander Ashitkin <alexander.ashitkin@db.com>
    Date:   Tue Aug 10 13:37:13 2021 -0400

        cleanup db specifics

    commit 5d32f00559951445ecaf7d5ec4d847541c10e7e9
    Author: Alexander Ashitkin <alexander.ashitkin@db.com>
    Date:   Tue Aug 10 11:05:28 2021 -0400

        cleanup db specifics

commit fcec28f82c0d447de64032c610b39fa75bb344b0
Merge: 6fd401af2 5b0dc1858
Author: Alexander Ashitkin <alexander.ashitkin@db.com>
Date:   Mon Aug 9 13:22:16 2021 +0100

    Pull request apache#9: Remote cache setup

    Merge in BND/maven-forked from feature/remote-cache-setup to incremental-maven-3.6.3

    * commit '5b0dc1858cf4ca499502f0741b2f3bf2ac55530c':
      remote cache setup - fail fast, baseline build, diff reporting, documentation.

commit 5b0dc1858cf4ca499502f0741b2f3bf2ac55530c
Author: Alexander Ashitkin <alexander.ashitkin@db.com>
Date:   Mon Mar 22 03:30:03 2021 +0100

    remote cache setup - fail fast, baseline build, diff reporting, documentation.

commit 6fd401af285692df568312ee66ee00edfe21f1cc
Author: Maximilian Novikov <maximilian.novikov@db.com>
Date:   Tue Apr 13 17:36:49 2021 +0100

    Pull request apache#8: Incremental Maven - added non-overrideable cache entries

    Merge in BND/maven-forked from non-overritable-build-info to incremental-maven-3.6.3

    Squashed commit of the following:

    commit 405bbef8a59cabc0b83e9a64717f65fbd6fb9e3d
    Author: maximilian.novikov@db.com <maximilian.novikov@db.com>
    Date:   Mon Apr 12 20:18:27 2021 +0200

        Incremental Maven - added non-overrideable cache entries - review fixes

    commit 197704672756e558fb3521e16859d0be324044b8
    Author: maximilian.novikov@db.com <maximilian.novikov@db.com>
    Date:   Fri Apr 9 11:35:48 2021 +0200

        Incremental Maven - added non-overrideable cache entries

commit 64c4b1f5cbdf920ad43560f34cc5be56d4557acf
Author: Alexander Ashitkin <alexander.ashitkin@db.com>
Date:   Tue Mar 9 04:10:20 2021 +0100

    injected version

commit 727e507ae2fd90dce7011f3f1c76eb4969577d5e
Author: maximilian.novikov@db.com <maximilian.novikov@db.com>
Date:   Wed Jan 13 16:17:25 2021 +0100

    Incremental Maven - Java 11 fix

commit b2f297566d5afcb0a285985cb5540685a28cafec
Author: maximilian.novikov@db.com <maximilian.novikov@db.com>
Date:   Tue Jan 12 14:10:08 2021 +0100

    Incremental Maven - initial commit
  • Loading branch information
@maximilian-novikov-db
maximilian-novikov-db committed Aug 10, 2021
1 parent cecedd3 commit 7d1b8c0
Show file tree
Hide file tree
Showing 83 changed files with 11,490 additions and 90 deletions.
1 change: 1 addition & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ target/
.classpath
.settings/
.svn/
.cache/
bin/
# Intellij
*.ipr
Expand Down
204 changes: 204 additions & 0 deletions Documentation/CACHE-HOWTO.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,204 @@
<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

### Overview

Cache configuration provides you additional control over incremental maven behavior. Follow it step by step to
understand how it works and figure out your optimal config

### Minimal config

Absolutely minimal config which enables incremental maven with local cache

```xml
<?xml version="1.0" encoding="UTF-8" ?>
<cache xmlns="org:apache:maven:cache:config:v1"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="org:apache:maven:cache:config:v1 cache-config.xsd">

<configuration>
<enabled>true</enabled>
<hashAlgorithm>XX</hashAlgorithm>
</configuration>

<input>
<global>
<glob>{*.java,*.xml,*.properties}</glob>
</global>
</input>
</cache>
```

### Enabling remote cache

Just add `<remote>` section under `<configuration>`

```xml

<configuration>
<enabled>true</enabled>
<hashAlgorithm>XX</hashAlgorithm>
<remote>
<url>https://yourserver:port</url>
</remote>
</configuration>
```

### Adding more file types to input

Add all the project specific source code files in `<glob>`. Scala in this case:

```xml

<input>
<global>
<glob>{*.java,*.xml,*.properties,*.scala}</glob>
</global>
</input>
```

### Adding source directory for bespoke project layouts

In most of the cases incremental maven will recognize directories automatically by build introspection. If not, you can
add additional directories with `<include>`. Also you can filter out undesirable dirs and files by using exclude tag

```xml

<input>
<global>
<glob>{*.java,*.xml,*.properties,*.scala}</glob>
<include>importantdir/</include>
<exclude>tempfile.out</exclude>
</global>
</input>
```

### Plugin property is env specific (breaks checksum and caching)

Consider to exclude env specific properties:

```xml

<input>
<global>
...
</global>
<plugin artifactId="maven-surefire-plugin">
<effectivePom>
<excludeProperty>argLine</excludeProperty>
</effectivePom>
</plugin>
</input>
```

Implications - builds with different `argLine` will have identical checksum. Validate that is semantically valid.

### Plugin property points to directory where only subset of files is relevant

If plugin configuration property points to `somedir` it will be scanned with default glob. You can tweak it with custom
processing rule

```xml

<input>
<global>
...
</global>
<plugin artifactId="protoc-maven-plugin">
<dirScan mode="auto">
<!--<protoBaseDirectory>${basedir}/..</protoBaseDirectory>-->
<tagScanConfig tagName="protoBaseDirectory" recursive="false" glob="{*.proto}"/>
</dirScan>
</plugin>
</input>
```

### Local repository is not updated because `install` is cached

Add `executionControl/runAlways` section

```xml
<?xml version="1.0" encoding="UTF-8" ?>
<cache xmlns="org:apache:maven:cache:config:v1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="org:apache:maven:cache:config:v1 cache-config.xsd">
<configuration>
...
</configuration>
<input>
...
</input>
<executionControl>
<runAlways>
<plugin artifactId="maven-failsafe-plugin"/>
<execution artifactId="maven-dependency-plugin">
<execId>unpack-autoupdate</execId>
</execution>
<goals artifactId="maven-install-plugin">
<goal>install</goal>
</goals>
</runAlways>
</executionControl>
</cache>
```

### I occasionally cached build with `-DskipTests=true` and tests do not run now

If you add command line flags to your build, they do not participate in effective pom - maven defers final value
resolution to plugin runtime. To invalidate build if filed value is different in runtime, add reconciliation section
to `executionControl`:

```xml

<executionControl>
<runAlways>
...
</runAlways>
<reconcile>
<plugin artifactId="maven-surefire-plugin" goal="test">
<reconcile propertyName="skip" skipValue="true"/>
<reconcile propertyName="skipExec" skipValue="true"/>
<reconcile propertyName="skipTests" skipValue="true"/>
<reconcile propertyName="testFailureIgnore" skipValue="true"/>
</plugin>
</reconcile>
</executionControl>
```

Please notice `skipValue` attribute. It denotes value which forces skipped execution.
Read `propertyName="skipTests" skipValue="true"` as if property skipTests has value true, plugin will skip execution If
you declare such value incremental maven will reuse appropriate full-build though technically they are different, but
because full-build is better it is safe to reuse

### How to renormalize line endings in working copy after committing .gitattributes (git 2.16+)

Ensure you've committed (and ideally pushed everything) - no changes in working copy. After that:

```shell
# Rewrite objects and update index
git add --renormalize .
# Commit changes
git commit -m "Normalizing line endings"
# Remove working copy paths from git cache
git rm --cached -r .
# Refresh with new line endings
git reset --hard
```

### I want to cache interim build and override it later with final version

Solution: set `-Dremote.cache.save.final=true` to nodes which produce final builds. Such builds will not be overridden
and eventually will replace all interim builds
52 changes: 52 additions & 0 deletions Documentation/CACHE-PARAMETERS.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

# Overview

This documents contains various configuration parameters supported by cache engine

## Command line flags

| Parameter | Description | Usage Scenario |
| ----------- | ----------- | ----------- |
| `-Dremote.cache.configPath=true/false` | Location of cache configuration file | Cache config is not in default location |
| `-Dremote.cache.enabled=true/false` | Remote cache and associated features disabled/enabled | To remove noise from logs then remote cache is not available |
| `-Dremote.cache.save.enabled=true/false` | Remote cache save allowed or not | To designate nodes which allowed to push in remote shared cache |
| `-Dremote.cache.save.final=true/false` | Is it allowed or not to override produced cache | To ensure that reference build is not overriden by interim build |
| `-Dremote.cache.failFast=true/false` | Fail on the first module whcih cannot be restored from cache | Remote cache setup/tuning/troubleshooting |
| `-Dremote.cache.baselineUrl=<http url>` | Location of baseline build for comparison | Remote cache setup/tuning/troubleshooting |

## Project level properties

Project level parameters allow overriding global parameters on project level Must be specified as project properties:

```xml

<pom>
...
<properties>
<remote.cache.input.glob>{*.css}</remote.cache.input.glob>
</properties>
</pom>
```

| Parameter | Description |
| ----------------------------- | ----------- |
| `remote.cache.input.glob` | Project specific glob to select sources. Overrides global glob. |
| `remote.cache.input` | Property prefix to mark paths which must be additionally scanned for source code. Value of property starting with this prefix will be treated as path relatively to current project/module |
| `remote.cache.exclude` | Property prefix to mark paths which must be excluded from source code search. Value of property starting with this prefix will be treated as path to current project/module |
| `remote.cache.processPlugins` | Introspect plugins to find inputs or not. Default is true. |
Loading

0 comments on commit 7d1b8c0

Please sign in to comment.