Support running KSP via KotlinCompilerRunner

[ting-yuan]

which is based on KGP's GradleCompilerRunnerWithWorker.

Also refactored the code to allow selection between the old and new
ways to create KSP tasks and call the compiler.

[ting-yuan]

@gavra0 here is a bit more explanation.

KotlinCompilerRunner.kt is what I hope to contribute to kotlin-gradle-plugin-api and KotlinCompilerRunnerImpls.kt to kotlin-gradle-plugin.

[gavra0]

Overall, it seems ok. The API surface is not too big:

• given a task you'll get a compiler runner
• compiler runner should allow you to specify compiler args and input/output/common sources

[gavra0]

This requires moving arguments to -api module, we need to check if that's possible and if they match the API guarantees.

[ting-yuan]

Good catch. I'll create a simplified / public version of those.

[ting-yuan]

Addressed with a simplified version of the arguments in 8782e89.

[gavra0]

I'm not sure if this is the right way to solve this. Forking of args has some downsides, as they are generated directly from compiler flags.

@Tapchicoma what do you think? Is compiler args something you'd like to expose in the API?

[Tapchicoma]

these particular flags are the part of compiler and will never be in KGP-api artifact. On the other side - we are exposing currently kotlinOptions and from 1.8.0 - compilerOptions generated DSL. And we have plans to add more missing options to compilerOptions in followup releases.

Probably we should think about another approach that will not require to use Kotlin compiler arguments from KSP side. Also this arguments are not an official compiler api and could have breaking changes between minor releases.

[gavra0]

I'm positively surprised that GradleCompileTaskProvider does not require Kotlin-specific task type, only Gradle Task is enough :)

[Tapchicoma]

Because it is declared on AbstractKotlinCompile and is used to provide generic compilation configuration that is same for all targets 🙂

[ting-yuan]

The proposed KotlinCompilerArguments mostly comes from kotlinOptions, compilerOptions (compiler args annotated with GradleOptions) and fields in BaseKotlinCompile and KotlinCompileTool:

    // From compilerOptions / kotlinOptions:
    var verbose: Boolean = false
    var allWarningsAsErrors: Boolean = false
    var languageVersion: String? = null
    var apiVersion: String? = null
    var useK2: Boolean = false
    var multiPlatform: Boolean = false
    var moduleName: String? = null
    var jvmTarget: String? = null // JVM
    var noJdk: Boolean = false // JVM
    var noStdlib: Boolean = false // JS
    var target: String = "v5"  // JS
    var target: String? = null // K/N

    // From BaseKotlinCompile / KotlinCompileTool / KotlinTaskConfigs.kt:
    var friendPaths: List<File> = emptyList()
    var libraries: List<File> = emptyList()
    var destination: File? = null
    var pluginOptions: List<String> = emptyList()
    var pluginClasspaths: List<File> = emptyList()

With a few exceptions. They are unlikely to change and may be stable enough to get into the KGP-api artifact.

    // JVM
    var noStdlib: Boolean = false
    var noReflect: Boolean = false
    var allowNoSourceFiles: Boolean = false

Proposal

1. Add the above JVM options (noStdlib, noReflect, allowNoSourceFiles) into CompilerJvmOptions. And replace KotlinCompilerArguments (defined in this patch) with compilerOptions. Because CompilerCommonOptions / CompilerJvmOptions are configurable in Kotlin DSL, it should make sense to use it to configure / setup the custom compilation.
2. Require the entry point in the proposed (e.g., createKotlinJvmCompilerRunner) API to accept a subtype of BaseKotlinCompile or KotlinJvmCompile, so that runJvmCompilerAsync can read the options defined in KotlinJvmCompile / BaseKotlinCompile.
3. Add freeArgs to runJvmCompilerAsync. KSP needs this to pass modified / removed files in its plugin options at execution time. Also it can cover some of the unstable compiler arguments (at users' discretion). Please see comments below.

Admittedly, there are still a few options that may be subject to change in the future. It might be better to handle those in freeArgs, at users' risk / discretion, until they become stable.

    // Used to select between the new and old expect actual linker
    var expectActualLinker: Boolean = false

    // These are used to select among JS backends.
    // In fact, in KGP, they are never set up; The free format args, e.g. -Xir-only are used instead.
    var irOnly: Boolean = false
    var irProduceJs: Boolean = false
    var irProduceKlibDir: Boolean = false
    var irProduceKlibFile: Boolean = false
    var irBuildCache: Boolean = false
    var wasm: Boolean = false

To summarize, with the newly proposed KotlinCompilerRunner:

1. It accepts subtypes of BaseKotlinCompile and returns an instance of KotlinCompilerRunner, which has a member function run<target>CompilerAsync.
2. Users (e.g., KSP) fill CompilerCommonOptions / CompilerJvmOptions / etc and configure BaseKotlinCompile, and call run<target>CompilerAsync with the options, freeArgs and sources / destination.

@Tapchicoma what do you think? The only new interface introduced is KotlinCompilerRunner. Nothing new around compiler args will need to be maintained.

edit: A factory / builder of the Compiler<target>Options will also be needed. As well as a way to provide default values of fields in BaseKotlinCompile.

[ting-yuan]

@Tapchicoma I've uploaded an implementation that makes use of compilerOptions instead of CommonCompilerArguments. The use of freeArgs is limited and most of them can be added to the API.

"-Xallow-no-source-files", "-no-stdlib", "-Xmulti-platform", "-Xexpect-actual-linker",
"-Xplugin=...", "-P plugin:..."

Currently I have to copy some free args from JS compilation task to select the backend and filter the libraries. Because IR_LEGACY is being deprecated, these will go away soon:

"-Xir-only", "-Xir-produce-js", "-Xir-produce-klib-file", "-Xir-produce-klib-dir", "-Xir-build-cache", "-Xwasm"

Paths are moved into the parameters of run*CompilerAsync. The API now becomes, for example,

interface KotlinJvmCompilerRunner : KotlinCompilerRunner {
    fun runJvmCompilerAsync(
        options: CompilerJvmOptions,
        freeArgs: List<String>,
        sources: List<File>,
        commonSources: List<File>,
        friendPaths: List<File>,
        libraries: List<File>,
        destination: File
    )
}

// Can be moved to somewhere more appropriate, with objectFactory removed.
fun createKotlinJvmCompilerRunner(
    task: Task,
    objectFactory: ObjectFactory,
): KotlinJvmCompilerRunner {
    return objectFactory.newInstance<KotlinJvmCompilerRunnerImpl>(task)
}

[Tapchicoma]

KGP has compileOnly dependency on this library as all classes should come from kotlin-compiler-embeddable on runtime. Probably you should do the same

[Tapchicoma]

If you are targeting Kotlin 1.8.0 - you could already use KotlinJvmFactory to create CompilerJvmOptions

[Tapchicoma]

For KotlinJsFactory and KotlinCommonFactory please open new issues, possibly it is still ok to add them into 1.8.0

[Tapchicoma]

If you need same way to create for soon-to-added CompilerNativeOptionsinterface - pleases create an issue as well

[Tapchicoma]

Add the above JVM options (noStdlib, noReflect, allowNoSourceFiles) into CompilerJvmOptions. And replace KotlinCompilerArguments (defined in this patch) with compilerOptions. Because CompilerCommonOptions / CompilerJvmOptions are configurable in Kotlin DSL, it should make sense to use it to configure / setup the custom compilation.

FYI: KGP adds these noStdlib and noReflect options when final Kotlin compiler arguments are calculated.

[Tapchicoma]

I think KGP or some Kotlin library should provide method to create compiler runners for specific platforms. It could be useful for other build tools as well. Would be nice to open an issue.

[Tapchicoma]

Because it is declared on AbstractKotlinCompile and is used to provide generic compilation configuration that is same for all targets 🙂

[Tapchicoma]

Add freeArgs to runJvmCompilerAsync. KSP needs this to pass modified / removed files in its plugin options at execution time. Also it can cover some of the unstable compiler arguments (at users' discretion). Please see comments below.

Generally KGP will try to move away from freeCompilerArguments by adding all required options into Kotlin*CompilerOptions. If you need some specific options open a new issues, so we can prioritize them.

[Tapchicoma]

Add freeArgs to runJvmCompilerAsync. KSP needs this to pass modified / removed files in its plugin options at execution time. Also it can cover some of the unstable compiler arguments (at users' discretion). Please see comments below.

FYI: KGP will go away from freeCompilerArgs and try to provide all user-required via compilerOptions.