Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify logic surrounding building of Java & Kotlin parsers. #665

Merged
merged 1 commit into from
Nov 28, 2023
Merged

Simplify logic surrounding building of Java & Kotlin parsers. #665

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 29 additions & 37 deletions src/main/java/org/openrewrite/maven/MavenMojoProjectParser.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -323,36 +323,34 @@ private Stream<SourceFile> processMainSources(
javaParserBuilder.classpath(dependencies).typeCache(typeCache);
kotlinParserBuilder.classpath(dependencies).typeCache(new JavaTypeCache());

Stream<? extends SourceFile> cus = mainJavaSources.isEmpty() ? Stream.empty()
: Stream.of(javaParserBuilder).map(JavaParser.Builder::build)
.flatMap(parser -> parser.parse(mainJavaSources, baseDir, ctx));
Stream<SourceFile> parsedJava = Stream.empty();
if (!mainJavaSources.isEmpty()) {
parsedJava = javaParserBuilder.build().parse(mainJavaSources, baseDir, ctx);
logDebug(mavenProject, "Scanned " + mainJavaSources.size() + " java source files in main scope.");
}

Stream<? extends SourceFile> kcus = mainKotlinSources.isEmpty() ? Stream.empty()
: Stream.of(kotlinParserBuilder).map(KotlinParser.Builder::build)
.flatMap(parser -> parser.parse(mainKotlinSources, baseDir, ctx));
Stream<SourceFile> parsedKotlin = Stream.empty();
if (!mainKotlinSources.isEmpty()) {
parsedKotlin = kotlinParserBuilder.build().parse(mainKotlinSources, baseDir, ctx);
logDebug(mavenProject, "Scanned " + mainKotlinSources.size() + " kotlin source files in main scope.");
}

List<Marker> mainProjectProvenance = new ArrayList<>(projectProvenance);
mainProjectProvenance.add(sourceSet("main", dependencies, typeCache));

Stream<SourceFile> parsedJava = cus.map(addProvenance(baseDir, mainProjectProvenance, generatedSourcePaths));
logDebug(mavenProject, "Scanned " + mainJavaSources.size() + " java source files in main scope.");

Stream<SourceFile> parsedKotlin = kcus.map(addProvenance(baseDir, mainProjectProvenance, generatedSourcePaths));
logDebug(mavenProject, "Scanned " + mainKotlinSources.size() + " kotlin source files in main scope.");

//Filter out any generated source files from the returned list, as we do not want to apply the recipe to the
//generated files.
Path buildDirectory = baseDir.relativize(Paths.get(mavenProject.getBuild().getDirectory()));
Stream<SourceFile> sourceFiles = Stream.concat(parsedJava, parsedKotlin).filter(s -> !s.getSourcePath().startsWith(buildDirectory));
Stream<SourceFile> sourceFiles = Stream.concat(parsedJava, parsedKotlin)
.filter(s -> !s.getSourcePath().startsWith(buildDirectory))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like it would be even better if we could filter out these generated sources before parsing them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had thought we parse them such that we have their types available for attribution, as indicated on line 302 above. What improvement would you make here then?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose we have to test this, but these sources should still not need to be parsed, because the type attribution in other sources is via class files.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it boils down to the question if getCompileClasspathElements() returns the classes compiled fom generated sources as well, which could depend on a number of different factors.

rewrite-maven-plugin/src/main/java/org/openrewrite/maven/MavenMojoProjectParser.java

Lines 317 to 324 in 0b8fc33

logInfo(mavenProject, "Parsing source files");
List<Path> dependencies = mavenProject.getCompileClasspathElements().stream()
.distinct()
.map(Paths::get)
.collect(toList());
JavaTypeCache typeCache = new JavaTypeCache();
javaParserBuilder.classpath(dependencies).typeCache(typeCache);
kotlinParserBuilder.classpath(dependencies).typeCache(new JavaTypeCache());

Right now we define generated sources as anything we find in the target/ directory, and parse those as source rather than compiled classes.

rewrite-maven-plugin/src/main/java/org/openrewrite/maven/MavenMojoProjectParser.java

Lines 302 to 308 in 0b8fc33

// Some annotation processors output generated sources to the /target directory. These are added for parsing but
// should be filtered out of the final SourceFile list.
List<Path> generatedSourcePaths = listJavaSources(mavenProject.getBasedir().toPath().resolve(mavenProject.getBuild().getDirectory()));
List<Path> mainJavaSources = Stream.concat(
generatedSourcePaths.stream(),
listJavaSources(mavenProject.getBasedir().toPath().resolve(mavenProject.getBuild().getSourceDirectory())).stream()
).collect(toList());

While I agree it might be interesting to not parse those generated sources as source files but rather only as classpath entries, I'm not entirely sure that would work in all cases, and verifying that would take more time than I currently have available. Are you ok merging this PR as-is to at least simplify the logic?

.map(addProvenance(baseDir, mainProjectProvenance, generatedSourcePaths));

// Any resources parsed from "main/resources" should also have the main source set added to them.
int sourcesParsedBefore = alreadyParsed.size();
Stream<SourceFile> parsedResourceFiles = resourceParser.parse(mavenProject.getBasedir().toPath().resolve("src/main/resources"), alreadyParsed)
.map(addProvenance(baseDir, mainProjectProvenance, null));

logDebug(mavenProject, "Scanned " + (alreadyParsed.size() - sourcesParsedBefore) + " resource files in main scope.");
// Any resources parsed from "main/resources" should also have the main source set added to them.
sourceFiles = Stream.concat(sourceFiles, parsedResourceFiles);
return sourceFiles;
return Stream.concat(sourceFiles, parsedResourceFiles);
}

private Stream<SourceFile> processTestSources(
Expand All @@ -379,35 +377,29 @@ private Stream<SourceFile> processTestSources(
// scan Kotlin files
String kotlinSourceDir = getKotlinDirectory(mavenProject.getBuild().getSourceDirectory());
List<Path> testKotlinSources = (kotlinSourceDir != null) ? listKotlinSources(mavenProject.getBasedir().toPath().resolve(kotlinSourceDir)) : Collections.emptyList();

alreadyParsed.addAll(testKotlinSources);

Stream<? extends SourceFile> cus = testJavaSources.isEmpty() ? Stream.empty()
: Stream.of(javaParserBuilder).map(JavaParser.Builder::build)
.flatMap(parser -> parser.parse(testJavaSources, baseDir, ctx));
Stream<SourceFile> parsedJava = Stream.empty();
if (!testJavaSources.isEmpty()) {
parsedJava = javaParserBuilder.build().parse(testJavaSources, baseDir, ctx);
logDebug(mavenProject, "Scanned " + testJavaSources.size() + " java source files in test scope.");
}

Stream<? extends SourceFile> kcus = testKotlinSources.isEmpty() ? Stream.empty()
: Stream.of(kotlinParserBuilder).map(KotlinParser.Builder::build)
.flatMap(parser -> parser.parse(testKotlinSources, baseDir, ctx));
Stream<SourceFile> parsedKotlin = Stream.empty();
if (!testKotlinSources.isEmpty()) {
parsedKotlin = kotlinParserBuilder.build().parse(testKotlinSources, baseDir, ctx);
logDebug(mavenProject, "Scanned " + testKotlinSources.size() + " kotlin source files in main scope.");
}

List<Marker> markers = new ArrayList<>(projectProvenance);
markers.add(sourceSet("test", testDependencies, typeCache));

Stream<SourceFile> parsedJava = cus.map(addProvenance(baseDir, markers, null));
logDebug(mavenProject, "Scanned " + testJavaSources.size() + " java source files in test scope.");

Stream<SourceFile> parsedKotlin = kcus.map(addProvenance(baseDir, markers, null));
logDebug(mavenProject, "Scanned " + testKotlinSources.size() + " kotlin source files in main scope.");

Stream<SourceFile> sourceFiles = Stream.concat(parsedJava, parsedKotlin);

// Any resources parsed from "test/resources" should also have the test source set added to them.
int sourcesParsedBefore = alreadyParsed.size();
Stream<SourceFile> parsedResourceFiles = resourceParser.parse(mavenProject.getBasedir().toPath().resolve("src/test/resources"), alreadyParsed)
.map(addProvenance(baseDir, markers, null));
Stream<SourceFile> parsedResourceFiles = resourceParser.parse(mavenProject.getBasedir().toPath().resolve("src/test/resources"), alreadyParsed);
logDebug(mavenProject, "Scanned " + (alreadyParsed.size() - sourcesParsedBefore) + " resource files in test scope.");
sourceFiles = Stream.concat(sourceFiles, parsedResourceFiles);
return sourceFiles;
return Stream.concat(Stream.concat(parsedJava, parsedKotlin), parsedResourceFiles)
.map(addProvenance(baseDir, markers, null));
}

@Nullable
Expand Down Expand Up @@ -504,7 +496,7 @@ public Map<MavenProject, Xml.Document> parseMaven(List<MavenProject> mavenProjec
if (document instanceof Xml.Document) {
projectMap.put(mavenProject, (Xml.Document) document);
} else if (document instanceof ParseError) {
logError(mavenProject, "Parse error in Maven Project File '" + path + "': "+ document);
logError(mavenProject, "Parse error in Maven Project File '" + path + "': " + document);
}
}
}
Expand Down