Support "multi-context" libraries

[jchavarri]

Introduction

At Ahrefs, we are now able to share more code between Melange and OCaml, thanks largely to Dune.

One of the main drivers for increasing code sharing is the UI for our products. This UI is comprised of React components implemented with Melange and ReasonReact, the Melange bindings to ReactJS. Recently, we began using server-reason-react to migrate some of these components to the server, which is built in OCaml.

As more components are migrated to be compilable and executable in OCaml to become "shared components", it is useful to rewrite existing Melange libraries in OCaml, preserving their structural layout (modules, functions) but using the native platform in the implementation. This allows for these migrations to proceed with fewer changes in the components that are becoming shared.

One way to implement these structurally uniform layouts is by leveraging Dune contexts, and have the Melange implementations live in one context and the OCaml ones in another.

Unfortunately, Dune currently does not support defining two libraries with the same name in different contexts using enabled_if. This feature would be very useful to support multi-context libraries, particularly for the UI components we are currently migrating.

Motivation

When it comes to shared Melange + OCaml code, the current approach we use involves building all code under the same Dune context and using copy_files to place the shared modules in the right folders.

This approach has a few downsides. For example, it is impossible to use the same name for both implementations of the same library. When attempting to define two libraries with the same name in the same context, Dune will rightly complain.

The current workaround is to use different names for each implementation and make all these multi-context libraries unwrapped, with the disadvantages that unwrapped libraries entail at scale, such as module collisions.

In addition to the above, copy_files brings its own issues with build performance, editor breakages, and problems when using PPX, which sometimes lead to "Multiple rules generated" Dune errors.

Technical details

@andreypopp originally explained some of these issues in depth in this Melange discussion. As mentioned in that discussion, our initial idea was to introduce a new Dune stanza for Melange libraries, and also introduce a separate PPX pipeline, exclusively for Melange.

Over time, we realized this is precisely what Dune build contexts are for. Dune contexts allow us to define completely separate builds, each with its own PPX pipelines. Libraries can also be set conditionally on a given context using enabled_if and the %{context_name} variable. For example:

(library
 (name foo)
 (enabled_if (= %{context_name} "alt-context")))

So it seems like the perfect solution for our use case.

Currently, support in Dune for this use case is limited. Dune makes a lot of assumptions that only one context will be used on a given build. Things like library duplication checks or source duplication checks don’t take the enabled_if field into account.

Our plan is to fix this with the help of the Dune maintainers team. We have started writing tests and prototyping some solutions in ocaml/dune#9839 and ocaml/dune#10179, and a few others have been extracted from these PRs.

After making Dune capable of handling multi-context libraries, we should fix editor integration too, so that users can choose in which build context they want to work. We also realized that the Dune-Merlin integration doesn’t support multiple contexts (see ocaml/dune#10035), and we will certainly need to add some integration in vscode-ocaml.

Eventually, this work might be useful for other alternate tooling supported by Dune, like Coq, Js_of_ocaml, or even cross-compilation. One can imagine users of these tools wanting to have a library with the same name defined in an OCaml native context and a more specific context.

Drawbacks

Using two Dune contexts might lead to relatively slower builds than using a single context, as code that is 100% shared across Melange and OCaml will have to be built twice. But arguably, this is kind of the case already, as even in the same context, Dune has to generate and execute rules to run both melc and ocamlc/opt, so the performance impact should, in theory, be limited.

Alternatives

Adding a new stanza like melange.library is the main alternative proposed, which can be found in the Melange discussion mentioned above.

Unresolved Questions

Most of the unknowns remain in what specific parts of Dune have to be modified to make this work. We discover new cases as we go and add tests, but we also need guidance from Dune maintainers before proceeding as some changes might be too invasive, or might break the way in which Dune handle specific errors.

Another part that is not 100% clear yet is how editor integration will work, but it should be easy to configure Dune to provide paths to artifacts based on a chosen context.