generate implicit _java import targets for every scala_library as an alternative to scala_export_to_java

[ghost]

I have spent some time benchmarking the bazel build of our internal Java/Scala codebase and came to the conclusion that a big chunk of our build time is spent generating deploy JARs of Scala targets caused by scala_export_to_java rules. Since a deploy JAR includes all transitive dependencies of a Scala target, having lots of these nodes in our dependency graphs essentially makes our build quadratic instead of linear. Further, having a deploy JAR as a dependency of a java_library rule also means that instantiation's dependencies have to be way less strict, since chances are good they can be already found somewhere in the transitive dependencies, and bazel can't force people to add them.

As a consequence, I've implemented the attached hack that allows us to bypass generating these deploy JARs, and also has the nice side effect of forcing much more strict dependencies when Java depends on Scala. The gist of it is that I've made scala_library into a macro that doesn't just instantiate the original scala_library rule, but also implicitly generates a java_import target that can be used from other java_* rules while still exporting all necessary Scala exports required. Since the scala_library might depend on other java_library rules itself, however, I had to make bazel "forget" about the JAR outs of those rules, since otherwise the native Java rule implementation in bazel complains about the fact that a java_* rule has a dependency on the JAR out of a java_library rule instead of on that target itself. I implemented that by piping every export and runtime dependency of the scala_library through a no-op genrule which I lovingly dubbed "making a yolocopy". Note that for all of this to work out in the end, I also had to adjust the names of the JAR outs of the scala_* rules so they exactly match the ones of the java_* rules, since my macro has no way of knowing whether such a dependency is Java or Scala just from its target name string.

Using this hack improved the build time of our mixed Java/Scala codebase from 358s to 161s on my laptop, which is a more than 2x speedup. Note that I timed this from a cold cache and with the persistent scalac worker activated. To give you some idea of the codebase, we have 424 Scala source files in 148 scala_library targets and 276 Java source files in 109 java_library targets.

I'm putting this PR up mainly for reference and to give an idea on what is currently needed to get a linear complexity mixed Java/Scala build, and I certainly don't expect this to get merged in the presented hacky form. However, the speedup we got is so significant that we have started using this internally, so if somebody has ideas of how to do this in a less hacky way, please shout :)

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please let us know the company's name.

[bazel-io]

Can one of the admins verify this patch?

[dinowernli]

cc @johnynek @smparkes @petemounce

Based on the massive impact this has had, we've actually decided to merge this into our fork of the scala rules. However, there may very well be a nicer way of doing this. Does anybody have any thoughts?

[johnynek]

Interesting approach.

My hope is that the work that is mentioned here:
bazelbuild/bazel#970

specifically the API for making a java provider, ships in the next release of bazel. Then we can do this totally first class and have any native java rule depend on the scala rules. Since you have this work around, I'd rather not merge if we have that hope (though they don't commit to a timespan except to say they are coding now).

Note this PR also relates to #57

[ittaiz]

+1 for Oscar's suggestion. Another option is to just use only scala_library in the meantime both for compiling targets which contain only Scala, mixed targets and ones which contain only Java.

…

On Fri, 2 Dec 2016 at 18:58 P. Oscar Boykin ***@***.***> wrote: Interesting approach. My hope is that the work that is mentioned here: bazelbuild/bazel#970 <bazelbuild/bazel#970> specifically the API for making a java provider, ships in the next release of bazel. Then we can do this totally first class and have any native java rule depend on the scala rules. Since you have this work around, I'd rather not merge if we have that hope (though they don't commit to a timespan except to say they are coding now). Note this PR also relates to #57 <#57> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#115 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIF4Kn7rge-dGqo_h3OvoVjGeVWB3uks5rEE5SgaJpZM4LCo79> .

[johnynek]

With #227 you should be able to have standard java targets depend on scala. Please reopen if you find any issues with the latest.

[ghost]

Thanks a lot for pinging me here @johnynek - I will upgrade to 0.5.2 and try to migrate to the new rules - much looking forward to this working natively without hacks!

[ghost]

We successfully upgraded our build to bazel 0.5.3 and the new version of the Scala rules. Everything seems to work nicely so far and build times are the same compared to using the attached hack (i.e. not quadratic), so we no longer need it at all.

Thanks a lot for fixing this issue, it is highly appreciated!

[johnynek]

@improbafabian nice to hear! Seems like things are working well these days.

I think the main issue open is cross compilation (or at least easily targeting a particular scala version)