Can't compile JavaFX projects (regression?)

[s-bernard]

More precisely, since commit a124774, a simple JavaFX project cannot compile and log the following errors:

[info] Compiling 1 Java source to ./out/javafx/compile/dest/classes ...
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by sbt.internal.inc.javac.DiagnosticsReporter$PositionImpl$ (file:/~/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-sbt/zinc-compile-core_2.12/1.2.5/zinc-compile-core_2.12-1.2.5.jar) to field com.sun.tools.javac.api.ClientCodeWrapper$DiagnosticSourceUnwrapper.d
WARNING: Please consider reporting this to the maintainers of sbt.internal.inc.javac.DiagnosticsReporter$PositionImpl$
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[error] ./javafx/src/ButtonApp.java:1:1: package javafx.application does not exist
[error] import javafx.application.Application;
[error] ./javafx/src/ButtonApp.java:2:1: package javafx.stage does not exist
[error] import javafx.stage.Stage;
…

Basically it cannot find the JavaFX packages.

Latest commit tested: 80eb7e0
Tested with java-11-graalvm and java-11-openjdk.
Was working as expected before a124774.

Sample code

cat build.sc

import mill._, scalalib._

val javaFXVersion = "13.0.2"

object javafx extends JavaModule {
  def mainClass = Some("Main")
  def ivyDeps = Agg(ivy"org.openjfx:javafx-controls:$javaFXVersion")
}

cat javafx/src/ButtonApp.java

import javafx.application.Application;
import javafx.stage.Stage;
import javafx.scene.Scene;
import javafx.scene.layout.VBox;
import javafx.scene.control.Button;

public class ButtonApp extends Application {
  public static void main(String[] args) {
    launch(args);
  }

  @Override public void start(Stage stage) {
    var button = new Button("Please Click Me!");
    button.setOnAction(e -> stage.close());
    var box = new VBox();
    box.getChildren().add(button);
    var scene = new Scene(box);
    stage.setScene(scene);
    stage.setTitle("Just a button");
    stage.show();
  }
}

cat javafx/src/Main.java

public class Main {

  public static void main(String[] args) {
    ButtonApp.main(args);
  }
}

[lefou]

It is probably an change in coursier, which might got a version bump as a transitive dependency.
Looks like you now need to depend on ivy"org.openjfx:javafx-controls:13;classifier=linux or whatever your platform is. javafx-controls with classifier is a transitive dependency of javafx-controls without classifier.

Might be related to #759

Maybe @alexarchambault sees more and can comment?

Here is the pom.xml of javafx-controls:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.openjfx</groupId>
  <artifactId>javafx-controls</artifactId>
  <version>13</version>
  <parent>
    <groupId>org.openjfx</groupId>
    <artifactId>javafx</artifactId>
    <version>13</version>
  </parent>
  <dependencies>
    <dependency>
      <groupId>org.openjfx</groupId>
      <artifactId>javafx-controls</artifactId>
      <version>13</version>
      <classifier>${javafx.platform}</classifier>
    </dependency>
    <dependency>
      <groupId>org.openjfx</groupId>
      <artifactId>javafx-graphics</artifactId>
      <version>13</version>
    </dependency>
  </dependencies>
</project>

(I posted this before: https://gitter.im/lihaoyi/mill?at=5e3175b5433e1d40397320f8)

[s-bernard]

I've just tried with classifier=linux with a124774: same behavior (I'm on linux).

Actually, I tried with some garbage classifiers like ivy"org.openjfx:javafx-controls:$javaFXVersion;classifier=aiueauieai" and I have the same results as without classifier: 4f4fab6 compiles and runs as there was no classifier and a124774 & 80eb7e0 fail to compile with the same package xxxx does not exist errors. No error on javafx.compileClasspath like when you change the artfactId.

[alexarchambault]

coursier does fetch the OS-dependent artifacts. It does from the CLI at least:

$ cs fetch org.openjfx:javafx-controls:13
~/.cache/coursier/v1/https/repo1.maven.org/maven2/org/openjfx/javafx-base/13/javafx-base-13-linux.jar
~/.cache/coursier/v1/https/repo1.maven.org/maven2/org/openjfx/javafx-graphics/13/javafx-graphics-13.jar
~/.cache/coursier/v1/https/repo1.maven.org/maven2/org/openjfx/javafx-graphics/13/javafx-graphics-13-linux.jar
~/.cache/coursier/v1/https/repo1.maven.org/maven2/org/openjfx/javafx-base/13/javafx-base-13.jar
~/.cache/coursier/v1/https/repo1.maven.org/maven2/org/openjfx/javafx-controls/13/javafx-controls-13-linux.jar
~/.cache/coursier/v1/https/repo1.maven.org/maven2/org/openjfx/javafx-controls/13/javafx-controls-13.jar

Same kind of result on macOS.

You might be missing OS and JDK-related infos in the way you're calling the coursier API. The high level API of coursier sets them up this way.

@lefou If that's a possibility for you, I would advise using the higher level API (coursier.Resolve, coursier.Fetch, coursier.Artifacts, etc.), described here, rather than the low-level one, to lower the odds of that kind of issue.

[lefou]

@alexarchambault Thanks for your comments. I'll have a look.

[rom1dep]

For the record, bumped into that as well when updating mill to > 0.5.3

[s-bernard]

Hello, any news on this issue? Thanks.

[lefou]

No, we discussed the implementation a bit (in #775), but nobody did the work yet.

[siddhartha-gadgil]

I may need to work around this issue, and wanted advice. I have done the following for now, and it seems to work. Is this likely to cause some trouble somewhere else?

override def unmanagedClasspath = 
      T{
        import coursier._
        val files = Fetch().addDependencies(dep"org.nd4j:nd4j-native-platform:1.0.0-beta7").run()
        val pathRefs = files.map(f => PathRef(Path(f)))
        Agg(pathRefs : _*)
      }

[lefou]

@siddhartha-gadgil unmannagedClasspath is what it is called, unmanaged. So there is a slighly chance, that dependencies provided there may collide with some provided by ivyDeps. But besides that, I think it's a perfectly reasonable work-around.

[siddhartha-gadgil]

Thanks. I will watch out for this, especially with transitive dependencies.

[lefou]

I updated PR #775 to gather some feedback.

[hmf]

Just a head's up for those who have not been following. Positive feedback at scalafx/scalafx#343 (comment). Discussion found here.

@lefou Nice work!
@rom1dep thanks for the test/example

8-)

[lefou]

I merged PR #775. Give it some time to build and publish. After that you should be able to pull the latest mill snapshot release or mill >= 0.9.7 when available.

You need to adapt the coursier resolution process by overriding def resolutionCustomizer: Task[Resolution => Resolution]. This could look like this:

import mill._
import mill.scalalib._

object javafx extends JavaModule {
  override def resolutionCustomizer = T.task {
    Some((_: coursier.core.Resolution).withOsInfo(coursier.core.Activation.Os.fromProperties(sys.props.toMap)))
  }
  def ivyDeps = Agg(ivy"org.openjfx:javafx-controls:13.0.2")
}

This example reads all OS properties from the current system properties, but of course you could also set specific props.

[s-bernard]

Isn’t it possible to have the Resolution default to withOsInfo(coursier.core.Activation.Os.fromProperties(sys.props.toMap))?
It should not hurt if not needed but will help all project that depends on OS specific artifact.

[lefou]

@s-bernard Sure, it is possible, but it also makes the build less reproducible, as those variables may differ on different machines.

[s-bernard]

OK but as you say, even with the current default, a build is already not completely reproducible between different machines. And this setting still makes build deterministic on a given system, which is the most important.
Moreover It may break the reproducibility between machines only for projects that actually use this osInfo so that need it. And if you can’t reproduce a bug in a build due to this, you can still override the resolutionCustomizer to use the other system sys.props instead of local ones.

Finally, wasn’t the goal of maven profiles to adapt the build to the current system? Specially to allow the use of system library like javafx does?

On the plus side, it will really make mill more accessible.

BTW, thank you very much for solving this issue.

[lefou]

Unless more users vote in, I'm not going to make it the default. My reasons are based on personal experience being a developer for more than 2 decades. I could imaging that we add some extra trait NonPortableCoursierModule/PlatformDependantCoursierModule (name just made up) that you can mix-in in any Java/Scala module and which enables and documents these settings by default. PRs welcome!

[s-bernard]

I like the idea of a trait. I prefer PlatformDependantCoursierModule between the two names. Or maybe CoursierWithOsInfoModule?

[charego]

Sorry to resurrect this issue, but @lefou's workaround isn't working for me.

import mill._
import mill.scalalib._

object testfx extends JavaModule {

  override def resolutionCustomizer = T.task {
    Some((_: coursier.core.Resolution).withOsInfo(coursier.core.Activation.Os.fromProperties(sys.props.toMap)))
  }

  def ivyDeps = Agg(
    ivy"org.springframework.boot:spring-boot-starter:2.6.2",
    ivy"com.fasterxml.jackson.core:jackson-databind:2.13.1",
    ivy"org.openjfx:javafx-controls:17.0.1",
    ivy"org.openjfx:javafx-fxml:17.0.1"
  )
}

$ mill -i testfx.run
[16/31] testfx.resolvedIvyDeps
1 targets failed
testfx.resolvedIvyDeps Failed to load source dependencies
  not found: https://repo1.maven.org/maven2/org/openjfx/javafx-controls/17.0.1/javafx-controls-17.0.1-${javafx.platform}.jar
  not found: https://repo1.maven.org/maven2/org/openjfx/javafx-fxml/17.0.1/javafx-fxml-17.0.1-${javafx.platform}.jar
  not found: https://repo1.maven.org/maven2/org/openjfx/javafx-graphics/17.0.1/javafx-graphics-17.0.1-${javafx.platform}.jar
  not found: https://repo1.maven.org/maven2/org/openjfx/javafx-base/17.0.1/javafx-base-17.0.1-${javafx.platform}.jar

Nor is @alexarchambault's CLI command.

$ cs fetch org.openjfx:javafx-controls:17.0.1
Error fetching artifacts:
https://repo1.maven.org/maven2/org/openjfx/javafx-base/17.0.1/javafx-base-17.0.1-${javafx.platform}.jar: not found: https://repo1.maven.org/maven2/org/openjfx/javafx-base/17.0.1/javafx-base-17.0.1-${javafx.platform}.jar
https://repo1.maven.org/maven2/org/openjfx/javafx-controls/17.0.1/javafx-controls-17.0.1-${javafx.platform}.jar: not found: https://repo1.maven.org/maven2/org/openjfx/javafx-controls/17.0.1/javafx-controls-17.0.1-${javafx.platform}.jar
https://repo1.maven.org/maven2/org/openjfx/javafx-graphics/17.0.1/javafx-graphics-17.0.1-${javafx.platform}.jar: not found: https://repo1.maven.org/maven2/org/openjfx/javafx-graphics/17.0.1/javafx-graphics-17.0.1-${javafx.platform}.jar

Did something change with coursier or with JavaFX 17?

[lefou]

Something rings a bell, but I don't know if it is related. A quick search revealed this sbt issue sbt/sbt#6564 and this ScalaFX issue scalafx/scalafx#362

[hmf]

@charego You can take a look at the javaFXMill project for a working example. Unfortunately I have not updated to the latest Mill version. But it may provide hints as to why your example is not working.

EDIT: working with 0.9.11 now (0.10.0-M5 also works but you need to use millw)

[charego]

Thanks! I can proceed by tweaking the resolution customizer with @s-bernard's suggestion:

override def resolutionCustomizer = T.task {
  Some((_: coursier.core.Resolution)
    .withOsInfo(coursier.core.Activation.Os.fromProperties(sys.props.toMap))
    .withExtraProperties(Seq(("javafx.monocle", "false")))
  )
}

This makes sense because mill 0.9.11 depends on an older coursier that doesn't have the fix in coursier/coursier#2176. It's unclear why my cs fetch command was failing, as I have the latest milestone version (2.1.0-M2-1) of coursier CLI tool.