Make meta builds (mill-build/build.sc) opt-in and configurable

[lefou]

As a follow up on PR #2377, we need to come up with some way to configure the mill-build/build.sc aka "meta build".

Meta builds need to be disabled, unless the build.sc explicitly enables it.

Meta build location should not be hardcoded but configured when enabled.

// Enable meta-build and specify the location
import $meta.`mill-build`

[lefou]

@lihaoyi wrote a comment to issue #2516, which belongs to this discussion:

Previously we had discussed having the whole mill-build folder be under import $meta.mill-build, where a user could choose what the folder name was. Elsewhere we had also discussed being able to rename build.sc via a flag, e.g. --build-file-name mill-build to avoid conflicts.

What if we combined the two ideas, such that a single flag --build-file-name controls both the meta-build path as well as the meta-build folder? e.g. it would default to build.sc/build/, but a user who has a conflict in build/ could do --build-file-name mill-build to rename both together to mill-build.sc/mill-build/. In this way, the build-file-name.sc file is the source of truth while the build-file-name/ folder is basically the associated extension point if the .sc format becomes too limiting.

This would help ensure that the names of both file and folder remain in-sync, removing a degree of freedom a import $meta.foo provides that is unnecessary, while remaining explicitly customizable and ensuring the relationship between the two is clear to the user. Conceptually it's like having a customizable Shared Primary Key relationship v.s. having an Foreign Key column in a database: the schema enforces the invariants you want by-construction, rather than starting with an unnecessarily flexible schema (import $meta.foo with a different folder name, multiple import $meta.foos...) and having to lock it down after-the-face via assertions.

I think we should not bind the build file name to the directory name of it's potential meta-build. These are two complete different things.

My most important mantra is: A build.sc is completely self-sufficient. It's the single source of truth. (I envision to even move all external settings like the .mill-version into some magic-import or using directives.)

By adding a import $meta we can opt-in a meta-build.

A meta-build is not necessarily tied to a single build.sc. E.g. multiple project can select the same meta-build via import $meta.^.company-defaults.

Choosing a different build script name, e.g. a build-experimental.sc to experiment with some settings, should not force a different meta-build.

A build.sc in a meta-build should be in general equal to a build.sc not within a meta-build. That especially means, that we should not automatically add a ./meta-build/src when compiling a ./meta-build/build.sc, just because we also do not add a ./src when compiling a ./build.sc.

If a build.sc needs to include additional files it can use the existing import $file and probably some new import $source to denote some source folder with .scala or other source files.

[lihaoyi]

A build.sc in a meta-build should be in general equal to a build.sc not within a meta-build. That especially means, that we should not automatically add a ./meta-build/src when compiling a ./meta-build/build.sc, just because we also do not add a ./src when compiling a ./build.sc.

If a build.sc needs to include additional files it can use the existing import $file and probably some new import $source to denote some source folder with .scala or other source files.

I think there's misunderstanding here. Currently, the mill-build/src does not get added to mill-build/build.sc, it gets added to build.sc. And the sources in mill-build/mill-build/src get added to mill-build/build.sc, and so on

The principle here is that the build.sc can be thought of as a shorthand syntax that is equivalent to a long-form ScalaModule: it has sources, ivydeps, and other things we may not have a shorthand syntax for: maybe even submodules, test suites, and so on. So where does is long-form ScalaModule' defined? Currently, the long-form for build.sclives inmill-build/`.

We can certainly add more shorthands like import $sources, but we should be clear that these are all syntactic sugar over the long-form ScalaModule definition.

So the build.sc file and mill-build/ folder are not two completely different things, but rather shortform and longform syntaxes for the same thing. I believe keeping that equivalence will help us keep things simple in the long run, because it clearly specifies how they should behave. Otherwise if we separate them conceptually, we'll be forced to make all sorts of decisions in how each one behaves, and likely end up with a messy outcome where sometimes they behave the same and sometimes they behave differently.

I don't think we should rely on meta-builds for shared/machine-global config. It's just not the right place for that kind of thing to live. If we want something like that, we should add specific support for it.

[lefou]

Ok, thanks for clarifying. It also means, we currently always use mill-build/src. And that's what I don't like so much.

We currently have two situations:

The simple case: A default Mill setup for building build.sc, which is only controlled via a build.sc with magic imports and a small set of config files (like Mill 0.10).

The advanced case / the "long form": A build.sc with a

import $meta.`mill-build`

and we use the setup of the root module in mill-build/build.sc to compile the top-level build.sc.

Question is, if in this long form, every derivation from the defaults needs to be explicitly defined? In that case, my wish to not have a default-included src dir would result in the need to explicitly override the def sources in mill-build/build.scs root module. This would be the most regular setup. I don't have seen many meta builds yet, so I really can't tell whether the need to have a default-included src dir is so common. I think, it is not and that explicitly adding it when needed is perfectly fine. Then, looking at the meta-build makes it perfectly clear why we enabled it in the first place. If that is a common case, we can still introduce a import $source as a shorthand later.

[lihaoyi]

Question is, if in this long form, every derivation from the defaults needs to be explicitly defined? In that case, my wish to not have a default-included src dir would result in the need to explicitly override the def sources in mill-build/build.scs root module. This would be the most regular setup. I don't have seen many meta builds yet, so I really can't tell whether the need to have a default-included src dir is so common. I think, it is not and that explicitly adding it when needed is perfectly fine. Then, looking at the meta-build makes it perfectly clear why we enabled it in the first place. If that is a common case, we can still introduce a import $source as a shorthand later.

Currently, this relies on MillBuildRootModule, which re-uses the ScalaModule config as much as possible except where we needed build-related special cases e.g. parsing build.sc to extract import $ivys to add to ivyDeps, and wrap the scripts to add to allSources.

We could certainly override def sources to make it empty by default, but I would prefer keeping the "same as ScalaModule" behavior as much as possible. Everyone is already familiar with how ScalaModule works, so the less divergence from that better. Furthermore, if we make it opt-in via an override, we can expect that different people would opt-in to different source directories arbitrarily since there's no default, which I would consider worse than having everyone agree to use the same src/ folder that's standardized across all ScalaModules.

I expect that we'll see a lot of people putting things in the mill-build/src/ folder once we enable it. If you look at SBT projects, almost all of them make heavy use of project/*.scala files once they grow beyond a trivial size:

• https://github.com/typelevel/cats/tree/main/project
• https://github.com/akka/akka/tree/main/project
• https://github.com/zio/zio/tree/series/2.x/project
• https://github.com/scala/scala/tree/2.13.x/project
• https://github.com/lampepfl/dotty/tree/main/project

These projects could have put everything in build.sbt and other auxiliary *.sbt files (which SBT does support), but chose to put things in full *.scala files for maintainability. This is the case across the various Scala sub-communities, and I don't think it's a coincidence that they all made the same decision. Once we enable this support on Mill, I expect a similar best practice to emerge here too, so having mill-build/src/ be enabled by default would help make that simpler and also standardize the layout of the *.scala build code folder rather than forcing everyone to arbitrarily choose a folder name

[lefou]

Yeah, that's a valid point. Getting more but smaller files will also improve compilation speed after small changes. Also cache invalidation is probably already more granular because of that.

How about only using a mill-build/src as source directory after we have enabled the meta build with import $meta.mill-build?

Additionally, we could support a pure import $meta without any segments, to use the preferred default name mill-build.

[lefou]

Btw, with using plain Scala files, we should be finally able to define ExternalModules.

[lihaoyi]

How about only using a mill-build/src as source directory after we have enabled the meta build with import $meta.mill-build?

Yes that's what I imagined: that the whole of mill-build/ comes together: mill-build/build.sc, mill-build/src, mill-build/resources, etc. all would be enabled with the same import $meta.

Additionally, we could support a pure import $meta without any segments, to use the preferred default name mill-build.

That sounds like a good idea.