Figure out a plan to stabilize isolated extension usages

[Wyverald]

Isolated extension usages (use_extension(..., isolate=True)) are currently guarded behind the flag --experimental_isolated_extension_usages.

This feature is useful in particular as it allows an extension to easily generate repos scoped to a single user module. However, it is a rather big departure in the mental model of module extensions, as it essentially allows an extension to be evaluated "multiple times" (small print -- big simplification, etc etc); certain extensions that may rely on being "used" by its hosting module might break under this model; and the ID of an extension is not just ".bzl file label + extension name" anymore.

To stabilize this feature, we should evaluate other solutions to the "user-module-scoped repos" problem, and think about ways to mitigate the downsides of isolated extension usages (documentation? intentional limitations?).

[Wyverald]

cc @fmeum @aherrmann

[fmeum]

Just noting some insights I had that are relevant for the evaluation:

1. Isolated extension usages receive independent lockfile entries, which limits the impact of tag changes for non-deterministic extensions.
2. Without isolated extension usages, extensions may need to recommend per-module repo prefixes that include the module version if they need to distinguish module versions in the case of a multiple_version_override (e.g. because the extension relies on its own lockfile, which may conflict if multiple module versions contribute one). Nobody will realistically do this as it causes huge churn on version bumps and is only relevant in this edge case, but it may make multiple_version_override hard to use with rulesets in practice.
3. Isolated extension instances can still share a static list of deps by having the module use_repo them from the main instance.

[aherrmann]

Apologies for the late reply. It didn't get to this earlier.

I'll share how isolated extensions are currently used in rules_nixpkgs and what problem they solve.

• Only isolated extensions, or the root or core module, are allowed to directly define the name of a repo.
• The reason is that non-isolated extensions operate on a global name-space, and if any module could introduce an arbitrary name, then we could encounter name clashes. (Nix packages have too many parameters to come up with a meaningful, unique canonical name.) So, isolated extensions solve the name clash problem by introducing separate namespaces.
• Before isolated extensions were introduced, the extensions were implemented using a hub-repository approach with a custom namespace implementation. Isolated extensions simplified the implementation.
• We do have multiple extensions that need to share data and, indeed, the approach taken is to pass corresponding labels to an attribute, and have a default label that refers to a use_repo'd default in the core module.

The key issue that isolated extensions solve in this case is that of repo namespacing. Apart from that rules_nixpkgs does not further require the separation of extension instances. So, an alternative approach could be a common mechanism to introduce per client module namespaces.

Indeed, the possible presence of multiple extension instances has to be taken into account by each module extension author. For instance, take an extension providing toolchains that expects the generated toolchains repo to only be included by the core module and generates its extension metadata accordingly. If this extension is used in isolated mode, then it should expose the toolchains repo to the invoking module instead.