Add the concept of roots

In larger projects, a lot of noise is produced by unused classes in
dependencies. To deal with some of that noise, we add the concept of
"roots," which is a subset of application classes. Instead of validating
all application classes, we start with roots and recursively traverse
application classes reachable from those roots. The resulting closure is
the set of application classes that we will actually validate. By
default, the roots are all classes in the current project.

README.md

@@ -19,7 +19,7 @@ to be using. In the ideal world, this would never happen, because you are enforc
 is using semver correctly. But in the real world, you have a runtime error on your hands and you won't catch it until 
 you exercise a specific code path in production or - hopefully - tests.
 
-This tool can detect _some_ binary incompatibilities at build time. Specifically, it can report
+This tool can detect certain binary incompatibilities at build time. Specifically, it will report
 
 * Missing classes
 * Duplicate classes
@@ -35,7 +35,9 @@ Conceptually, the inputs for running the tool are _application classes_ and _pla
 classes are the classes that need to be validated, and the platform APIs are the APIs provided by the runtime
 environment of the application, e.g. the JVM or the Android SDK.
 
-Typically, the application classes are the classes compiled directly by the project and its runtime dependencies.
+Typically, the application classes are the classes compiled directly by the project and the classes from the project's 
+runtime dependencies that they reference.
+
 The platform APIs can be specified by the JVM, a set of dependencies, or serialized type descriptors.
 
 ## Basic setup
@@ -50,7 +52,12 @@ plugins {
 
 ## Application classes
 
-By default, the application classes are the classes from the main source set and the runtime dependencies of the project.
+By default, the application classes are selected from the main source set and the runtime dependencies of the project.
+The classes compiled from the main source set and subproject dependencies are treated as _roots_. The roots and all
+classes from the external runtime dependencies reachable from the roots then form the set of application classes.
+
+The concept of roots makes it easier to filter out unused classes from third-party dependencies, which would otherwise
+produce noise.
 
 You can customize this behavior, e.g. change the Gradle configuration that describes the dependencies.
 
@@ -65,6 +72,33 @@ expediter {
 
 For example, in an Android project, you will want to use a different configuration, such as `productionReleaseRuntime`.
 
+You can also customize how the roots are chosen. This is the implicit default setup, where the roots are the classes
+compiled from the current project and other subprojects of the same projects.
+
+```kotlin
+expediter {
+    application {
+        roots {
+            project()
+        }
+    }
+}
+```
+
+This is a different setup, where the roots are all classes compiled from the current project and all classes in its
+runtime dependencies. Beware that with this setup, there will likely be a lot of noise from unused classes providig
+optional functionality.
+
+```kotlin
+expediter {
+    application {
+        roots {
+            all()
+        }
+    }
+}
+```
+
 ## Platform APIs
 
 Platform APIs can be provided by the JVM or specified as a set of dependencies or published type descriptors in the

core/src/main/kotlin/com/toasttab/expediter/Expediter.kt

@@ -20,6 +20,7 @@ import com.toasttab.expediter.ignore.Ignore
 import com.toasttab.expediter.issue.Issue
 import com.toasttab.expediter.provider.ApplicationTypesProvider
 import com.toasttab.expediter.provider.PlatformTypeProvider
+import com.toasttab.expediter.roots.RootSelector
 import com.toasttab.expediter.types.ApplicationType
 import com.toasttab.expediter.types.ApplicationTypeContainer
 import com.toasttab.expediter.types.InspectedTypes
@@ -40,13 +41,15 @@ import protokt.v1.toasttab.expediter.v1.TypeFlavor
 class Expediter(
     private val ignore: Ignore,
     private val appTypes: ApplicationTypeContainer,
-    private val platformTypeProvider: PlatformTypeProvider
+    private val platformTypeProvider: PlatformTypeProvider,
+    private val rootSelector: RootSelector
 ) {
     constructor(
         ignore: Ignore,
         appTypes: ApplicationTypesProvider,
-        platformTypeProvider: PlatformTypeProvider
-    ) : this(ignore, ApplicationTypeContainer.create(appTypes.types()), platformTypeProvider)
+        platformTypeProvider: PlatformTypeProvider,
+        rootSelector: RootSelector = RootSelector.All
+    ) : this(ignore, ApplicationTypeContainer.create(appTypes.types()), platformTypeProvider, rootSelector)
 
     private val inspectedTypes: InspectedTypes by lazy {
         InspectedTypes(appTypes, platformTypeProvider)
@@ -119,7 +122,7 @@ class Expediter(
 
     fun findIssues(): Set<Issue> {
         return (
-            inspectedTypes.classes.flatMap { appType ->
+            inspectedTypes.reachableTypes(rootSelector).flatMap { appType ->
                 findIssues(appType)
             } + inspectedTypes.duplicateTypes
             ).filter { !ignore.ignore(it) }

core/src/main/kotlin/com/toasttab/expediter/parser/TypeParsers.kt

@@ -16,6 +16,7 @@
 package com.toasttab.expediter.parser
 
 import com.toasttab.expediter.types.ApplicationType
+import com.toasttab.expediter.types.ClassfileSource
 import com.toasttab.expediter.types.FieldAccessType
 import com.toasttab.expediter.types.MemberAccess
 import com.toasttab.expediter.types.MemberSymbolicReference
@@ -35,7 +36,7 @@ import protokt.v1.toasttab.expediter.v1.TypeDescriptor
 import java.io.InputStream
 
 object TypeParsers {
-    fun applicationType(stream: InputStream, source: String) = ApplicationTypeParser(source).apply {
+    fun applicationType(stream: InputStream, source: ClassfileSource) = ApplicationTypeParser(source).apply {
         ClassReader(stream).accept(this, SKIP_DEBUG)
     }.get()
 
@@ -44,7 +45,7 @@ object TypeParsers {
     }.get()
 }
 
-private class ApplicationTypeParser(private val source: String) : ClassVisitor(ASM9, TypeDescriptorParser()) {
+private class ApplicationTypeParser(private val source: ClassfileSource) : ClassVisitor(ASM9, TypeDescriptorParser()) {
     private val refs: MutableSet<MemberAccess<*>> = hashSetOf()
     private val referencedTypes: MutableSet<String> = hashSetOf()
 
@@ -99,6 +100,7 @@ private class ApplicationTypeParser(private val source: String) : ClassVisitor(A
                 )
 
                 referencedTypes.addAll(SignatureParser.parseMethod(descriptor).referencedTypes())
+                referencedTypes.add(owner)
             }
 
             override fun visitFieldInsn(opcode: Int, owner: String, name: String, descriptor: String) {
@@ -118,6 +120,7 @@ private class ApplicationTypeParser(private val source: String) : ClassVisitor(A
                 )
 
                 referencedTypes.addAll(SignatureParser.parseType(descriptor).referencedTypes())
+                referencedTypes.add(owner)
             }
 
             override fun visitInvokeDynamicInsn(

core/src/main/kotlin/com/toasttab/expediter/provider/ClasspathApplicationTypesProvider.kt

@@ -17,10 +17,10 @@ package com.toasttab.expediter.provider
 
 import com.toasttab.expediter.parser.TypeParsers
 import com.toasttab.expediter.scanner.ClasspathScanner
-import java.io.File
+import com.toasttab.expediter.types.ClassfileSource
 
 class ClasspathApplicationTypesProvider(
-    elements: Iterable<File>
+    elements: Iterable<ClassfileSource>
 ) : ApplicationTypesProvider {
     private val scanner = ClasspathScanner(elements)
 

core/src/main/kotlin/com/toasttab/expediter/roots/RootSelector.kt

@@ -0,0 +1,11 @@
+package com.toasttab.expediter.roots
+
+import com.toasttab.expediter.types.ApplicationType
+
+interface RootSelector {
+    fun isRoot(type: ApplicationType): Boolean
+
+    object All : RootSelector {
+        override fun isRoot(type: ApplicationType) = true
+    }
+}

core/src/main/kotlin/com/toasttab/expediter/scanner/ClasspathScanner.kt

@@ -15,24 +15,24 @@
 
 package com.toasttab.expediter.scanner
 
-import java.io.File
+import com.toasttab.expediter.types.ClassfileSource
 import java.io.InputStream
 import java.lang.Exception
 import java.lang.RuntimeException
 import java.util.jar.JarInputStream
 import java.util.zip.ZipFile
 
 class ClasspathScanner(
-    private val elements: Iterable<File>
+    private val elements: Iterable<ClassfileSource>
 ) {
-    fun <T> scan(parse: (stream: InputStream, source: String) -> T): List<T> = elements.flatMap { types(it, parse) }
+    fun <T> scan(parse: (stream: InputStream, source: ClassfileSource) -> T): List<T> = elements.flatMap { types(it, parse) }
 
     private fun isClassFile(name: String) = name.endsWith(".class") &&
         !name.startsWith("META-INF/versions") && // mrjars not supported yet
         !name.endsWith("package-info.class") &&
         !name.endsWith("module-info.class")
 
-    private fun <T> scanJarStream(stream: JarInputStream, source: String, parse: (stream: InputStream, source: String) -> T) = generateSequence { stream.nextJarEntry }
+    private fun <T> scanJarStream(stream: JarInputStream, source: ClassfileSource, parse: (stream: InputStream, source: ClassfileSource) -> T) = generateSequence { stream.nextJarEntry }
         .filter { isClassFile(it.name) }
         .map {
             try { parse(stream, source) } catch (e: Exception) {
@@ -41,34 +41,34 @@ class ClasspathScanner(
         }
         .toList()
 
-    fun <T> scanJar(path: File, parse: (stream: InputStream, source: String) -> T): List<T> = path.inputStream().use {
-        scanJarStream(JarInputStream(it), path.name, parse)
+    fun <T> scanJar(source: ClassfileSource, parse: (stream: InputStream, source: ClassfileSource) -> T): List<T> = source.file.inputStream().use {
+        scanJarStream(JarInputStream(it), source, parse)
     }
 
-    fun <T> scanAar(path: File, parse: (stream: InputStream, source: String) -> T): List<T> =
-        ZipFile(path).use { aar ->
+    fun <T> scanAar(source: ClassfileSource, parse: (stream: InputStream, source: ClassfileSource) -> T): List<T> =
+        ZipFile(source.file).use { aar ->
             when (val classesEntry = aar.getEntry("classes.jar")) {
                 null -> emptyList()
                 else -> {
                     JarInputStream(aar.getInputStream(classesEntry)).use {
-                        scanJarStream(it, path.name, parse)
+                        scanJarStream(it, source, parse)
                     }
                 }
             }
         }
 
-    fun <T> scanClassDir(path: File, parse: (stream: InputStream, source: String) -> T): List<T> =
-        path.walkTopDown().filter { isClassFile(it.name) }.map {
+    fun <T> scanClassDir(source: ClassfileSource, parse: (stream: InputStream, source: ClassfileSource) -> T): List<T> =
+        source.file.walkTopDown().filter { isClassFile(it.name) }.map {
             it.inputStream().use { classStream ->
-                parse(classStream, path.name)
+                parse(classStream, source)
             }
         }.toList()
 
-    private fun <T> types(path: File, parse: (stream: InputStream, source: String) -> T) =
+    private fun <T> types(source: ClassfileSource, parse: (stream: InputStream, source: ClassfileSource) -> T) =
         when {
-            path.isDirectory -> scanClassDir(path, parse)
-            path.name.endsWith(".jar") -> scanJar(path, parse)
-            path.name.endsWith(".aar") -> scanAar(path, parse)
+            source.file.isDirectory -> scanClassDir(source, parse)
+            source.file.name.endsWith(".jar") -> scanJar(source, parse)
+            source.file.name.endsWith(".aar") -> scanAar(source, parse)
             else -> emptyList()
         }
 }

core/src/main/kotlin/com/toasttab/expediter/types/InspectedTypes.kt

@@ -19,6 +19,8 @@ import com.toasttab.expediter.issue.Issue
 import com.toasttab.expediter.parser.SignatureParser
 import com.toasttab.expediter.parser.TypeSignature
 import com.toasttab.expediter.provider.PlatformTypeProvider
+import com.toasttab.expediter.roots.RootSelector
+import java.util.LinkedList
 import java.util.concurrent.ConcurrentHashMap
 import java.util.concurrent.ConcurrentMap
 
@@ -28,7 +30,7 @@ class ApplicationTypeContainer(
 ) {
     companion object {
         fun create(all: List<ApplicationType>): ApplicationTypeContainer {
-            val duplicates = HashMap<String, MutableList<String>>()
+            val duplicates = HashMap<String, MutableList<ClassfileSource>>()
             val types = HashMap<String, ApplicationType>()
 
             for (type in all) {
@@ -48,7 +50,7 @@ class ApplicationTypeContainer(
                 }
             }
 
-            return ApplicationTypeContainer(types, duplicates.map { (k, v) -> Issue.DuplicateType(k, v) })
+            return ApplicationTypeContainer(types, duplicates.map { (k, v) -> Issue.DuplicateType(k, v.map { it.file.name }) })
         }
     }
 }
@@ -75,7 +77,7 @@ class InspectedTypes(
         }
     }
 
-    private fun traverse(type: Type): TypeHierarchy {
+    private fun hierarchy(type: Type): TypeHierarchy {
         val cached = hierarchyCache[type.name]
 
         if (cached == null) {
@@ -92,7 +94,7 @@ class InspectedTypes(
                 if (superType == null) {
                     superTypes.add(OptionalType.MissingType(signature.name))
                 } else {
-                    val hierarchy = traverse(superType)
+                    val hierarchy = hierarchy(superType)
                     superTypes.add(OptionalType.PresentType(superType))
                     superTypes.addAll(hierarchy.superTypes)
                 }
@@ -112,9 +114,30 @@ class InspectedTypes(
     }
 
     fun resolveHierarchy(type: Type): ResolvedTypeHierarchy {
-        return traverse(type).resolve()
+        return hierarchy(type).resolve()
     }
 
     val classes: Collection<ApplicationType> get() = appTypes.appTypes.values
+
+    fun reachableTypes(rootSelector: RootSelector): Collection<ApplicationType> {
+        if (rootSelector == RootSelector.All) {
+            return appTypes.appTypes.values
+        } else {
+            val reachable = hashMapOf<String, ApplicationType>()
+            val todo = LinkedList(appTypes.appTypes.values.filter(rootSelector::isRoot))
+
+            while (todo.isNotEmpty()) {
+                val next = todo.remove()
+
+                if (reachable.put(next.name, next) == null) {
+                    todo.addAll(next.referencedTypes.mapNotNull(appTypes.appTypes::get))
+                    todo.addAll(hierarchy(next).presentSuperTypes().filterIsInstance<ApplicationType>())
+                }
+            }
+
+            return reachable.values
+        }
+    }
+
     val duplicateTypes: Collection<Issue.DuplicateType> get() = appTypes.duplicates
 }

core/src/main/kotlin/com/toasttab/expediter/types/TypeHierarchy.kt

@@ -20,16 +20,20 @@ class TypeHierarchy(
     val superTypes: Set<OptionalType>
 ) {
     fun resolve(): ResolvedTypeHierarchy {
-        val missing = superTypes.filterIsInstance<OptionalType.MissingType>()
+        val missing = missingSuperTypes()
         return if (missing.isNotEmpty()) {
             ResolvedTypeHierarchy.IncompleteTypeHierarchy(type, missing.toSet())
         } else {
             ResolvedTypeHierarchy.CompleteTypeHierarchy(
                 type,
-                superTypes.asSequence().filterIsInstance<OptionalType.PresentType>().map { it.type }
+                presentSuperTypes()
             )
         }
     }
+
+    fun missingSuperTypes() = superTypes.filterIsInstance<OptionalType.MissingType>()
+
+    fun presentSuperTypes() = superTypes.filterIsInstance<OptionalType.PresentType>().map { it.type }
 }
 
 /**
@@ -67,7 +71,7 @@ sealed interface ResolvedTypeHierarchy : OptionalResolvedTypeHierarchy, Identifi
     override val name get() = type.name
 
     class IncompleteTypeHierarchy(override val type: Type, val missingType: Set<OptionalType.MissingType>) : ResolvedTypeHierarchy
-    class CompleteTypeHierarchy(override val type: Type, val superTypes: Sequence<Type>) : ResolvedTypeHierarchy {
+    class CompleteTypeHierarchy(override val type: Type, val superTypes: Iterable<Type>) : ResolvedTypeHierarchy {
         val allTypes: Sequence<Type> get() = sequenceOf(type) + superTypes
     }
 }

model/src/main/kotlin/com/toasttab/expediter/types/ClassfileSource.kt

@@ -0,0 +1,26 @@
+package com.toasttab.expediter.types
+
+import java.io.File
+
+/**
+ * Reference to a class file with additional metadata describing where the class file comes from.
+ */
+data class ClassfileSource(
+    val file: File,
+    val type: ClassfileSourceType,
+    val name: String
+)
+
+enum class ClassfileSourceType {
+    /** unmanaged by the build system, e.g. a raw jar file */
+    UNKNOWN,
+
+    /** compiled from source in the current project/subproject */
+    SOURCE_SET,
+
+    /** a different subproject within the same project */
+    SUBPROJECT_DEPENDENCY,
+
+    /** an external dependency managed by the build system */
+    EXTERNAL_DEPENDENCY
+}

model/src/main/kotlin/com/toasttab/expediter/types/Type.kt

@@ -35,7 +35,7 @@ class ApplicationType(
     override val descriptor: TypeDescriptor,
     val memberAccess: Set<MemberAccess<*>>,
     val referencedTypes: Set<String>,
-    val source: String
+    val source: ClassfileSource
 ) : Type {
     override fun toString() = "ApplicationType[$name]"
 }