Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(gradle): Support for gradle consistent versions plugin #19182

Merged
merged 54 commits into from
Dec 8, 2022
Merged
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
33b0ca9
Support for plugin gradle-consistent-versions by Palantir
janhoy Nov 29, 2022
02ff915
Convert const function to proper function
janhoy Nov 30, 2022
dc5d84c
Add test for missing lock file and not activated plugin
janhoy Nov 30, 2022
86c7b3b
Fix kebab rules, more robust glob-to-regex
janhoy Nov 30, 2022
b923c34
Update lib/modules/manager/gradle/extract.spec.ts
janhoy Dec 1, 2022
9733032
Review comment isLockFile
janhoy Dec 1, 2022
090f068
Review feedback - use 'codeBlock' in tests
janhoy Dec 1, 2022
6842549
Review feedback - Match the props and lock on sub folder, not only re…
janhoy Dec 1, 2022
2a4db1c
Move GCV check above for-loop
janhoy Dec 1, 2022
3dfdc48
Revert "Update lib/modules/manager/gradle/extract.spec.ts"
janhoy Dec 1, 2022
da2599e
Review feedback
janhoy Dec 1, 2022
1758cc7
Fix logick in isLockFile to also support both root and sub folders
janhoy Dec 1, 2022
e73b384
Fix name of build.gradle.kts support
janhoy Dec 1, 2022
49cef0a
Support both top level and sub folder in index.ts
janhoy Dec 1, 2022
75d410f
Remove comment block
janhoy Dec 1, 2022
6275b48
VersionWithPositions fields cannot be null
janhoy Dec 1, 2022
381cb1f
Rework the glob'ing/regex/group logic:
janhoy Dec 1, 2022
b51b71b
Parsing of props file positions in CRLF and LF
janhoy Dec 1, 2022
8b9df5f
More validation of parsing regex, including a test
janhoy Dec 2, 2022
f4ecf56
Support for depType=test
janhoy Dec 2, 2022
d871f79
Added test for Test Dependencies parsing
janhoy Dec 2, 2022
4a0e81b
Check header line of lockfile instead of loooking for plugin activati…
janhoy Dec 2, 2022
e8a5ac6
Register currentValue, not currentVersion
janhoy Dec 2, 2022
ae2f093
Merge branch 'main' into palantir-consistent-versions
janhoy Dec 2, 2022
22689fd
Fix bug for monorepo - report correct packageFile for sub folders
janhoy Dec 2, 2022
ed3d522
Review comment - use 'depName' instead of 'packageName'
janhoy Dec 3, 2022
c630c16
Move unit tests to separate spec file
janhoy Dec 3, 2022
1bcd43e
Merge branch 'main' into palantir-consistent-versions
janhoy Dec 3, 2022
f4829d7
Combine if checks into same if clause
janhoy Dec 3, 2022
5924824
Remove unnecessary comments
janhoy Dec 3, 2022
5f0259c
Fix comments
janhoy Dec 4, 2022
54346dc
No need to export mockFs(), since we don't even need it in gcv tests …
janhoy Dec 4, 2022
2c3794c
Tweak to test to make it more readable
janhoy Dec 4, 2022
673cb14
Review feedback regarding use of jest assertions
janhoy Dec 4, 2022
4f8893d
Merge branch 'main' into palantir-consistent-versions
janhoy Dec 4, 2022
4359a36
Merge branch 'main' into palantir-consistent-versions
janhoy Dec 4, 2022
dcdac86
Update yarn.lock
janhoy Dec 5, 2022
451afe3
Merge branch 'main' into palantir-consistent-versions
janhoy Dec 5, 2022
7b2f3b8
Add debug printing to PR
janhoy Dec 5, 2022
f509a28
More debug print to PR
janhoy Dec 6, 2022
abaffc1
More debugging for CI
janhoy Dec 6, 2022
d9a7f7a
Yet more logging
janhoy Dec 6, 2022
3138d8e
Log parsed objects
janhoy Dec 6, 2022
19a3db5
Log also lockfile map
janhoy Dec 6, 2022
f6dbe8a
Do not use replaceAll()
janhoy Dec 6, 2022
1cf989b
Remove debug logging
janhoy Dec 6, 2022
1edb143
Remove misplaced comment
janhoy Dec 6, 2022
1782d75
Merge branch 'main' into palantir-consistent-versions
janhoy Dec 6, 2022
b38b48e
Merge branch 'main' into palantir-consistent-versions
janhoy Dec 6, 2022
a7af9d7
Revert "Update yarn.lock"
janhoy Dec 7, 2022
226ac43
Merge branch 'main' into palantir-consistent-versions
janhoy Dec 7, 2022
35529c3
Merge remote-tracking branch 'origin/palantir-consistent-versions' in…
janhoy Dec 7, 2022
054a756
Merge branch 'main' into palantir-consistent-versions
rarkins Dec 8, 2022
bc767d8
Merge branch 'main' into palantir-consistent-versions
rarkins Dec 8, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/usage/java.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ Renovate can update:
- dependencies whose version is defined in a `*.properties` file
- `*.versions.toml` files in any directory or `*.toml` files inside the `gradle`
directory ([Gradle Version Catalogs docs](https://docs.gradle.org/current/userguide/platforms.html))
- `versions.props` from [gradle-consistent-versions](https://github.com/palantir/gradle-consistent-versions) plugin

Renovate does not support:

Expand Down
15 changes: 12 additions & 3 deletions lib/modules/manager/gradle/artifacts.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,17 @@ import {
prepareGradleCommand,
} from '../gradle-wrapper/utils';
import type { UpdateArtifact, UpdateArtifactsResult } from '../types';
import {
isGcvLockFile,
isGcvPropsFile,
} from './extract/consistent-versions-plugin';
import { isGradleBuildFile } from './utils';

// .lockfile is gradle default lockfile, /versions.lock is gradle-consistent-versions plugin lockfile
function isLockFile(fileName: string): boolean {
return fileName.endsWith('.lockfile') || isGcvLockFile(fileName);
}

async function getUpdatedLockfiles(
oldLockFileContentMap: Record<string, string | null>
): Promise<UpdateArtifactsResult[]> {
Expand All @@ -31,7 +40,7 @@ async function getUpdatedLockfiles(
const status = await getRepoStatus();

for (const modifiedFile of status.modified) {
if (modifiedFile.endsWith('.lockfile')) {
if (isLockFile(modifiedFile)) {
const newContent = await readLocalFile(modifiedFile, 'utf8');
if (oldLockFileContentMap[modifiedFile] !== newContent) {
res.push({
Expand Down Expand Up @@ -90,7 +99,7 @@ export async function updateArtifacts({
logger.debug(`gradle.updateArtifacts(${packageFileName})`);

const fileList = await getFileList();
const lockFiles = fileList.filter((file) => file.endsWith('.lockfile'));
const lockFiles = fileList.filter((file) => isLockFile(file));
if (!lockFiles.length) {
logger.debug('No Gradle dependency lockfiles found - skipping update');
return null;
Expand Down Expand Up @@ -147,7 +156,7 @@ export async function updateArtifacts({
.map(quote)
.join(' ')}`;

if (config.isLockFileMaintenance) {
if (config.isLockFileMaintenance || isGcvPropsFile(packageFileName)) {
janhoy marked this conversation as resolved.
Show resolved Hide resolved
cmd += ' --write-locks';
} else {
const updatedDepNames = updatedDeps
Expand Down
259 changes: 259 additions & 0 deletions lib/modules/manager/gradle/extract.spec.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
import { stripIndent } from 'common-tags';
import { Fixtures } from '../../../../test/fixtures';
import { fs, logger } from '../../../../test/util';
import type { ExtractConfig } from '../types';
import {
parseLockFile,
parsePropsFile,
usesGcv,
} from './extract/consistent-versions-plugin';
import * as parser from './parser';
import { extractAllPackageFiles } from '.';

Expand Down Expand Up @@ -841,4 +847,257 @@ describe('modules/manager/gradle/extract', () => {
},
]);
});

// Tests for gradle-consistent-version plugin
it('gradle-consistent-versions parse versions files', async () => {
const fsMock = {
'build.gradle': '(this file contains) com.palantir.consistent-versions',
'versions.props': `org.apache.lucene:* = 1.2.3`,
'versions.lock': stripIndent`
org.apache.lucene:lucene-core:1.2.3 (10 constraints: 95be0c15)
org.apache.lucene:lucene-codecs:1.2.3 (5 constraints: 1231231)`,
};

mockFs(fsMock);

const res = await extractAllPackageFiles(
{} as ExtractConfig,
Object.keys(fsMock)
);

expect(res).toMatchObject([
{
packageFile: 'versions.lock',
},
{
packageFile: 'versions.props',
deps: [
{
depName: 'org.apache.lucene:lucene-core',
depType: 'dependencies',
fileReplacePosition: 22,
groupName: 'org.apache.lucene:*',
lockedVersion: '1.2.3',
managerData: {
fileReplacePosition: 22,
packageFile: 'versions.props',
},
registryUrls: [],
},
{
depName: 'org.apache.lucene:lucene-codecs',
depType: 'dependencies',
fileReplacePosition: 22,
groupName: 'org.apache.lucene:*',
lockedVersion: '1.2.3',
managerData: {
fileReplacePosition: 22,
packageFile: 'versions.props',
},
registryUrls: [],
},
],
},
{
packageFile: 'build.gradle',
},
]);
});

it('gradle-consistent-versions plugin not used due to plugin not defined', async () => {
const fsMock = {
'build.gradle': 'no plugin defined here',
'versions.props': `org.apache.lucene:* = 1.2.3`,
'versions.lock': `org.apache.lucene:lucene-core:1.2.3`,
};
mockFs(fsMock);

const res = await extractAllPackageFiles(
{} as ExtractConfig,
Object.keys(fsMock)
);
expect(res).toBeNull();
});

it('gradle-consistent-versions plugin not used due to lockfile missing', async () => {
const fsMock = {
'build.gradle': '(this file contains) com.palantir.consistent-versions',
'versions.props': `org.apache.lucene:* = 1.2.3`,
};
mockFs(fsMock);

const res = await extractAllPackageFiles(
{} as ExtractConfig,
Object.keys(fsMock)
);
expect(res).toBeNull();
});

it('gradle-consistent-versions plugin works for sub folders', () => {
const fsMock = {
'mysub/build.gradle.kts': `(this file contains) 'com.palantir.consistent-versions'`,
'mysub/versions.props': `org.apache.lucene:* = 1.2.3`,
'mysub/versions.lock': `org.apache.lucene:lucene-core:1.2.3`,
'othersub/build.gradle.kts': `nothing here`,
};
mockFs(fsMock);

expect(usesGcv('mysub/versions.props', fsMock)).toBeTrue();
expect(usesGcv('othersub/versions.props', fsMock)).toBeFalse();
});

it('gradle-consistent-versions multi levels of glob', async () => {
const fsMock = {
'build.gradle': '(this file contains) com.palantir.consistent-versions',
'versions.props': stripIndent`
org.apache.* = 4
org.apache.lucene:* = 3
org.apache.lucene:a.* = 2
org.apache.lucene:a.b = 1
`,
'versions.lock': stripIndent`
org.apache.solr:x.y:1 (10 constraints: 95be0c15)
org.apache.lucene:a.b:1 (10 constraints: 95be0c15)
org.apache.lucene:a.c:1 (10 constraints: 95be0c15)
org.apache.lucene:a.d:1 (10 constraints: 95be0c15)
org.apache.lucene:d:1 (10 constraints: 95be0c15)
org.apache.lucene:e.f:1 (10 constraints: 95be0c15)
`,
};
mockFs(fsMock);

const res = await extractAllPackageFiles(
{} as ExtractConfig,
Object.keys(fsMock)
);

// Each lock dep is only present once, with highest prio for exact prop match, then globs from longest to shortest
expect(res).toMatchObject([
{
packageFile: 'versions.lock',
datasource: 'maven',
deps: [],
},
{
packageFile: 'versions.props',
datasource: 'maven',
deps: [
{
managerData: {
packageFile: 'versions.props',
fileReplacePosition: 91,
},
packageName: 'org.apache.lucene:a.b',
currentValue: '1',
currentVersion: '1',
lockedVersion: '1',
fileReplacePosition: 91,
registryUrls: [],
depType: 'dependencies',
},
{
managerData: {
packageFile: 'versions.props',
fileReplacePosition: 65,
},
depName: 'org.apache.lucene:a.c',
currentValue: '2',
currentVersion: '2',
lockedVersion: '1',
groupName: 'org.apache.lucene:a.*',
fileReplacePosition: 65,
registryUrls: [],
depType: 'dependencies',
},
{
managerData: {
packageFile: 'versions.props',
fileReplacePosition: 65,
},
depName: 'org.apache.lucene:a.d',
currentValue: '2',
currentVersion: '2',
lockedVersion: '1',
groupName: 'org.apache.lucene:a.*',
fileReplacePosition: 65,
registryUrls: [],
depType: 'dependencies',
},
{
managerData: {
packageFile: 'versions.props',
fileReplacePosition: 39,
},
depName: 'org.apache.lucene:d',
currentValue: '3',
currentVersion: '3',
lockedVersion: '1',
groupName: 'org.apache.lucene:*',
fileReplacePosition: 39,
registryUrls: [],
depType: 'dependencies',
},
{
managerData: {
packageFile: 'versions.props',
fileReplacePosition: 39,
},
depName: 'org.apache.lucene:e.f',
currentValue: '3',
currentVersion: '3',
lockedVersion: '1',
groupName: 'org.apache.lucene:*',
fileReplacePosition: 39,
registryUrls: [],
depType: 'dependencies',
},
],
},
{
packageFile: 'build.gradle',
datasource: 'maven',
deps: [],
},
]);
});

it('gradle-consistent-versions plugin correct position for CRLF and LF', () => {
const crlfProps2ndLine = parsePropsFile(`a.b:c.d=1\r\na.b:c.e=2`)[0].get(
'a.b:c.e'
);
const lfProps2ndLine =
parsePropsFile(`a.b:c.d=1\na.b:c.e=2`)[0].get('a.b:c.e');

expect(crlfProps2ndLine?.filePos).toBe(19);
expect(lfProps2ndLine?.filePos).toBe(18);
});

it('gradle-consistent-versions plugin test bogus input lines', () => {
const parsedProps = parsePropsFile(stripIndent`
# comment:foo.bar = 1
123.foo:bar = 2
this has:spaces = 3
starts.with:space = 4
contains(special):chars = 5
a* = 6
this.is:valid.dep = 7
valid.glob:* = 8
`);

expect(parsedProps[0]?.size).toBe(1); // no 7 is valid exact dep
expect(parsedProps[1]?.size).toBe(1); // no 8 is valid glob dep

// lockfile
const parsedLock = parseLockFile(stripIndent`
# comment:foo.bar:1 (10 constraints: 95be0c15)
123.foo:bar:2 (10 constraints: 95be0c15)
this has:spaces:3 (10 constraints: 95be0c15)
starts.with:space:4 (10 constraints: 95be0c15)
contains(special):chars:5 (10 constraints: 95be0c15)
no.colon:6 (10 constraints: 95be0c15)
this.is:valid.dep:7 (10 constraints: 95be0c15)
`);

expect(parsedLock.size).toBe(1); // no 7 is valid exact dep
});
});
13 changes: 13 additions & 0 deletions lib/modules/manager/gradle/extract.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ import { getFileContentMap } from '../../../util/fs';
import { MavenDatasource } from '../../datasource/maven';
import type { ExtractConfig, PackageDependency, PackageFile } from '../types';
import { parseCatalog } from './extract/catalog';
import {
isGcvLockFile,
isGcvPropsFile,
parseGcv,
usesGcv,
} from './extract/consistent-versions-plugin';
import { parseGradle, parseProps } from './parser';
import type {
GradleManagerData,
Expand Down Expand Up @@ -70,6 +76,13 @@ export async function extractAllPackageFiles(
} else if (isTOMLFile(packageFile)) {
const updatesFromCatalog = parseCatalog(packageFile, content);
extractedDeps.push(...updatesFromCatalog);
} else if (isGcvLockFile(packageFile)) {
// Skip and do not trigger parsing, wait for VERSIONS_PROPS
} else if (isGcvPropsFile(packageFile)) {
if (usesGcv(packageFile, fileContents)) {
const updatesFromGcv = parseGcv(packageFile, fileContents);
extractedDeps.push(...updatesFromGcv);
} // else skip silently since this is just a name collision
janhoy marked this conversation as resolved.
Show resolved Hide resolved
} else {
const vars = getVars(registry, dir);
const {
Expand Down
Loading