dune build @check should pass if dune build @all does

[Khady]

I'm working on improving the build system in ahrefs' monorepo. Part of this work is allowing to build as many things as possible in one command, and to speed up the feedback loop from dune. dune build @all tends to be pretty slow, even in subparts of the monorepo. So I wanted to use dune build @check instead, at least during development. But there are directories where dune build @check doesn't pass, while building the @all alias is successful. It feels like a bug, given that @check should be a subset of @all.

[nojb]

Do you have a small example reproducing this behaviour?

[Khady]

Sadly I don't right now, it's complicated to extract a small example from that large repo. Especially as it's not even easy to identify which folder/file/rule is creating a rule with is then executed by dune build @check. But I'll come back with more details if I get some.

[Khady]

So I've identified that in our case the problem happened because of a copy_files# directive, which brings in the folder some files that are not actually used by the binary.

The project structure is something like this:

├── dune-project
├── exe
│   ├── dune
│   └── main.ml
└── lib
    └── f.ml

And you can reproduce the error with

$ cat lib/f.ml 
let return = Lwt.return
$ cat exe/dune
(copy_files# "../lib/*.ml")

(executable (name main))
$ cat exe/main.ml
let () = print_endline "hello world"
$ dune clean
$ dune build --cache=disabled
$ dune build --cache=disabled @check
File "lib/f.ml", line 1, characters 13-23:
Error: Unbound module Lwt

[nojb]

Thanks. My understanding of this example is:

1. the file f.ml is copied to the exe folder
2. the file f.ml is naturally considered part of the executable stanza in that folder,
3. when building @all, the file f.ml is not compiled (probably because ocamldep is run first and found not to be referenced from main.ml), but
4. when building @check the file f.ml is compiled (probably because Dune does not do the ocamldep dance in this case), but fails because the executable stanza does not depend on the lwt library.

Does this match your understanding?

[Khady]

yes, that is also my understanding (except that I believe you made a typo, in point 4 it should say "when building @check).

[nojb]

yes, that is also my understanding (except that I believe you made a typo, in point 4 it should say "when building @check).

Indeed, thanks.

[emillon]

If I remember correctly, the ocaml-ci team encountered the same behaviour.

Personally (I don't know if that's shared by other maintainers), I don't have the expectation that @check should be a subset of @all. What these aliases do is not precisely specified, but I think it's OK to see @check as a quick thing to only have editor support working (fairly independently of how your project is configured), while @all is the project-specific, correct (precise) way to do a full build.

(put differently: @check is fast because it needs to run less of the build, but that means it's less precise)

[rgrinberg]

In this case, I think the issue is with the @all target. It's suppose to
include as many targets as possible, so it shouldn't skip compiling f.ml even
if it's not needed for the executable.

Btw, a way to answer the question if a target belongs to @check is to ask
oneself the following question: do I need the target for full editor
integration. If the answer is yes, then the target belongs to @check.

[emillon]

See also #3182

[Khady]

related to that issue, if I overwrite all or default, I'd expect check to react accordingly. Now I can create an alias to only compile some specific code. But I can't get a check version of that alias. Maybe the problem is not

dune build @check should pass if dune build @all does

but instead

please automatically derive and expose a check version of every alias