[WIP] use graal native-image to create native images of jvm tools for performance increases in certain workflows #6893

cosmicexplorer · 2018-12-10T04:37:19Z

WIP, as it depends on:

Problem

Graal is a virtual machine with a very large variety of frontends, including JVM- and LLVM-based languages, as well as interpreted languages such as Python, JavaScript, and R. One of the many things it can do is create "native images" which are able to begin executing immediately, without any VM startup time (although the performance is limited to the capabilities of AOT compilation). In some experimentation, I found I was able to match the speed of a warm nailgunned scalafmt task on a large target set in our monorepo with an extremely naive parallelism strategy (invoking one native image process per target). Not having to maintain a warm nailgun (and the associated memory) has potential benefits on developer laptops, as well as CI machines, depending on how leaky the tool is, and the runtime environment. In an ephemeral environment of any sort (apparently kubernetes is an example of this), zero startup time may make it more scalable to lazily download JVM tools instead of maintaining state in any specific ephemeral node.

Solution

Add the GraalCE binary tool, which can transform a classpath into an AOT-compiled image with zero startup time.
Create a NativeImageExecutor for executing JVM code which creates a native image on the fly, then executes it hermetically.
Integrate native-image into the NailgunTaskBase implementation so that it can be selected using --execution-strategy=native-image, e.g. in pants.ini.
Add support for hermetic execution via native-image to the scalafmt task.

Followup

Use the v2 parallel product_request() API to perform parallel hermetic execution of all native-image subprocesses (ideally with an API to do this from NailgunTask).
Implement GraalNativeImageCreate to create native images out of any jvm_bundles, probably in a separate PR.
Fix the scalafmt task to only run on invalidated files, not all files in all invalidated targets, probably in a separate PR.
Define more clearly what workflows a graal-produced native image is generally more effective on, and document it.
Consider other similar tools such as OpenJDK's jaotc.

Result

Scalafmt can be invoked as a native-image, and the path is clear for other tools to do the same!

cosmicexplorer · 2019-04-08T08:40:08Z

This should be ready for review after removing many different parts from it. There might be a smaller surface to cut at, but I really wanted to have the end-to-end testing via scalafmt there along with this large new feature. I've made a project at https://github.com/pantsbuild/pants/projects/13 covering the larger arc of native-image work.

illicitonion

This is super exciting! I have a couple of questions about the core abstractions here...

When we were adding the hermetic execution strategy, I started feeling kind of sketchy about it because in reality, each subclass needs to write specific code to handle the hermetic case. It looks like it's the same for native-image as a strategy, and that in a bunch of places we have fake implementations; if you specify --execution-strategy=native-image for a bunch of Tasks where you can, it will either do subprocess, or hermetic, or raise. Accordingly, I don't actually think --execution-strategy is a good flag to have on a common super-class. It feels like it should only be exposed by things that support it (which can use a common library class to implement it), rather than exposed by any JVM tool (where some things just don't implement some of the strategies)...

Though if we do keep these things in NailgunTaskBase, we should probably rename that to something like JvmToolTaskBase, because it's very not nailgun-specific any more.

Similarly, I'm not convinced that the JVM Executor and Runner abstractions are right to load this into; right now, for native image we simply ignore the arguments passed to the API methods; that feels like we're trying to force an implementation into an existing (but incorrect) abstraction; either we should be able to properly use the arguments to the methods, or we should not try to use the same API.

illicitonion · 2019-04-08T10:48:38Z

src/python/pants/backend/jvm/subsystems/graal.py

+  def generate_urls(self, version, host_platform):
+    system_id = self._SYSTEM_ID[host_platform.os_name]
+    archive_basename = self._ARCHIVE_BASE_FMT.format(version=version, system_id=system_id)
+    return [self._DIST_URL_FMT.format(version=version, base=archive_basename)]


How does this interact with networks which can't access the public internet? Will this automatically fall back to some mirror, or is that code that needs writing?

This uses the same resolution mechanism as other binary tools, in that if --no-allow-external-binary-tool-downloads is set, it will attempt to fetch it from --binaries-baseurls. The url generators are a convenience for open source users which allows someone to set e.g. --graal-version=1.0.0rc16 when that version is released in the future, without requiring any work on our part to allow that. We haven't yet upstreamed any script which syncs our pantsbuild s3 to another source, so people in other monorepos without public internet access will have to write that code for themselves.

src/python/pants/backend/jvm/subsystems/graal.py

illicitonion · 2019-04-08T10:59:53Z

src/python/pants/backend/jvm/subsystems/graal.py

+    ] + list(native_image_config.extra_args)
+
+    pprinted_argv = safe_shlex_join(argv)
+    with temporary_dir() as tmp_dir:


You may want to specify the actual output dir as the root_dir here because it guarantees that the move you do later will be on the same device (so quick); people who have tempdirs on different partitions from their cachedir would otherwise need to incur a second disk write to do the move

I've now modified native-image execution to occur hermetically, so we avoid this entirely, but noting this for the future!

src/python/pants/java/executor.py

src/python/pants/java/nailgun_client.py

tests/python/pants_test/backend/jvm/tasks/test_scalafmt.py

illicitonion · 2019-04-08T11:45:53Z

src/python/pants/backend/jvm/subsystems/graal.py

+
+  class NativeImageCreationError(Exception): pass
+
+  def produce_native_image(self, tool_classpath, main_class, native_image_config, context,


This feels like it could do with a unit test

illicitonion · 2019-04-08T11:46:42Z

src/python/pants/engine/fs.py

+    text_type(root))
+
+
+def digest_file_paths(paths, scheduler, root=None):


This should probably have a unit test, as it's now a publicly exposed function

cosmicexplorer · 2019-04-08T19:31:40Z

@illicitonion I agree with your high-level feedback and will remove native-image as an execution strategy and instead (I'm thinking) move this functionality into its own task mixin!

cosmicexplorer · 2019-04-08T19:32:18Z

Marking this as WIP again until the above comments are addressed.

cosmicexplorer · 2019-04-14T01:11:08Z

The large refactoring behind the current diff is on top of the commits from #7551 and #7552. This remains a WIP due to the remaining comments above.

./pants --no-process-execution-cleanup-local-dirs fmt.scalafmt --execution-strategy=native-image --files-per-process=2 --no-skip src/scala:: Also, remove the parallel execution stuff and make scalafmt work for real: - had to delay initialization of a few classes until runtime for the scalafmt cli to stop making paths relative to the native-image build dir - made a `.native_image_config` property to specify these delayed initializations - make native-image creation output to a workunit - add docstrings

Eric-Arellano · 2020-05-17T22:36:49Z

Closing due to being stale (pretty big merge conflict) - although GraalVM is awesome and it would be neat to see this be revived.

cosmicexplorer force-pushed the graal-is-cool branch from 2e553c4 to 6a6bb18 Compare December 15, 2018 20:39

cosmicexplorer force-pushed the graal-is-cool branch from 3eaaf4c to 64922d3 Compare January 21, 2019 22:45

cosmicexplorer force-pushed the graal-is-cool branch from 71ec854 to 3323c8a Compare February 10, 2019 02:01

cosmicexplorer force-pushed the graal-is-cool branch from 3323c8a to 6b0d54b Compare February 28, 2019 19:50

cosmicexplorer force-pushed the graal-is-cool branch from 6b0d54b to 8ffb598 Compare March 8, 2019 08:10

cosmicexplorer force-pushed the graal-is-cool branch from 785f90c to 7e5c1c6 Compare March 20, 2019 05:45

cosmicexplorer mentioned this pull request Apr 7, 2019

embed a graal native-image configuration inside the zinc wrapper #7506

Merged

cosmicexplorer force-pushed the graal-is-cool branch 3 times, most recently from 671fa8e to 767e2d7 Compare April 8, 2019 04:07

cosmicexplorer force-pushed the graal-is-cool branch from 1f6e6d4 to 334ac4b Compare April 8, 2019 08:10

cosmicexplorer changed the title ~~[WIP] use graal to create native images of jvm tools and jvm_binaries for performance increases in certain workflows~~ use graal native-image to create native images of jvm tools for performance increases in certain workflows Apr 8, 2019

cosmicexplorer force-pushed the graal-is-cool branch from c5831bc to e96e5f3 Compare April 8, 2019 08:36

cosmicexplorer requested review from illicitonion, stuhood and wisechengyi April 8, 2019 08:37

illicitonion requested changes Apr 8, 2019

View reviewed changes

cosmicexplorer changed the title ~~use graal native-image to create native images of jvm tools for performance increases in certain workflows~~ [WIP] use graal native-image to create native images of jvm tools for performance increases in certain workflows Apr 8, 2019

This was referenced Apr 13, 2019

[WIP] pretend files which can't be accessed just don't exist when glob matching #7551

Closed

allow executing scalafmt hermetically #7552

Closed

cosmicexplorer force-pushed the graal-is-cool branch from e96e5f3 to 24727b2 Compare April 14, 2019 00:45

cosmicexplorer mentioned this pull request May 16, 2019

Platform/Environment-specific subsystems #7735

Closed

This was referenced May 18, 2019

add hacky options to ZincCompile to use a provided native-image binary #7755

Closed

Hermetic zinc runs with a native-image #7796

Merged

This was referenced Jul 14, 2019

add script to generate zinc native-images, with example usage #8036

Merged

helper function / @rule to inject environment variables with absolute paths into hermetic process executions #8076

Closed

cosmicexplorer added 5 commits August 30, 2019 11:11

pretend files which can't be accessed just don't exist in the engine

d5d48da

almost allow executing scalafmt hermetically

b7d96e8

add hermetic execution

1547c17

add hermetic testing!

2376277

cosmicexplorer force-pushed the graal-is-cool branch from 24727b2 to 5d20d23 Compare August 30, 2019 18:30

cosmicexplorer mentioned this pull request Oct 12, 2019

Let plugins declare @rules and move the native backend into contrib #5970

Closed

Eric-Arellano closed this May 17, 2020

terpstra mentioned this pull request Nov 30, 2020

Zinc as a native-image? #11244

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] use graal native-image to create native images of jvm tools for performance increases in certain workflows #6893

[WIP] use graal native-image to create native images of jvm tools for performance increases in certain workflows #6893

cosmicexplorer commented Dec 10, 2018 •

edited

Loading

cosmicexplorer commented Apr 8, 2019

illicitonion left a comment

illicitonion Apr 8, 2019

cosmicexplorer Apr 8, 2019

illicitonion Apr 8, 2019

cosmicexplorer Apr 14, 2019

illicitonion Apr 8, 2019

illicitonion Apr 8, 2019

cosmicexplorer commented Apr 8, 2019

cosmicexplorer commented Apr 8, 2019

cosmicexplorer commented Apr 14, 2019

Eric-Arellano commented May 17, 2020


		class NativeImageCreationError(Exception): pass

		def produce_native_image(self, tool_classpath, main_class, native_image_config, context,

		text_type(root))


		def digest_file_paths(paths, scheduler, root=None):

[WIP] use graal native-image to create native images of jvm tools for performance increases in certain workflows #6893

[WIP] use graal native-image to create native images of jvm tools for performance increases in certain workflows #6893

Conversation

cosmicexplorer commented Dec 10, 2018 • edited Loading

Problem

Solution

Followup

Result

cosmicexplorer commented Apr 8, 2019

illicitonion left a comment

Choose a reason for hiding this comment

illicitonion Apr 8, 2019

Choose a reason for hiding this comment

cosmicexplorer Apr 8, 2019

Choose a reason for hiding this comment

illicitonion Apr 8, 2019

Choose a reason for hiding this comment

cosmicexplorer Apr 14, 2019

Choose a reason for hiding this comment

illicitonion Apr 8, 2019

Choose a reason for hiding this comment

illicitonion Apr 8, 2019

Choose a reason for hiding this comment

cosmicexplorer commented Apr 8, 2019

cosmicexplorer commented Apr 8, 2019

cosmicexplorer commented Apr 14, 2019

Eric-Arellano commented May 17, 2020

cosmicexplorer commented Dec 10, 2018 •

edited

Loading