RFC: Multi-context library installation

[anmonteiro]

Context

"Multi-context" or "Universal" libraries refer to libraries sharing the same name, but enabled in different dune build contexts (through (enabled_if (=%{context_name} ...))). Support for this was added in ocaml/dune#10307.

ocaml/dune#10471 shows that these libraries cannot currently share the same public_name.

Problem statement

• Multi-context libraries may be installed by running dune install and passing the --context flag.
• ocaml/dune#10493 shows the limitation when not passing the --context flag.

Proposed solution

• Build Melange libraries & melange.emit exclusively in their own context.
• Introduce a (melange) field to (context ..) in dune-workspace and phase out (modes melange) from the dune configuration
• Introduce a new (context ..) field in (library ..) that adheres to the ordered set language:
  • :standard defines the default context where most libraries live today
  • libraries can be added to multiple contexts with e.g. (context :standard melange other-context)
  • (context melange) implies (modes melange)
  • (context :standard) (modes melange) becomes forbidden
• Allow libraries with the same public_name to coexist iff one lives exclusively in the default context and the other lives exclusively in the melange context
  • This doesn't represent a problem for installation because melange artifacts are already installed in their own directory: <lib-root>/<lib-name-segments>/melange
  • Installing sources may have to be changed to install Melange sources in the same melange directory mentioned above
• If --context isn't passed to dune install, automatically install the Melange and default contexts in the same prefix

The following example illustrates how a Melange project could be configured in this proposal:

;; dune-project
(lang dune 3.17)

(using melange 0.2) ;; likely implies incrementing the Melange rules version

;; dune-workspace
(lang dune 3.17)

(context default)

;; explicit in this proposal, to be discussed whether it makes sense to
;; automatically define the context if `(using melange <version>)` is
;; present in `dune-project`
(context
 (default
  (name melange)
  (melange <options?>))) ;; <-- NEW

;; dune

(library
 (name foo)
 (public_name foo)
 (context :standard))

(library
 (name foo)
 (public_name foo) ;; same public_name as the native library
 (context melange)) ;; new field `context` in `(library)`

Alternatives Considered

• generate 2 dune install commands in the <project>.opam file
  • this was ruled out because dune should be opam-agnostic, and other tools (e.g. nix) wouldn't have a way of knowing whether they should run dune install twice
• implicitly define a "melange" dune context if (using melange <version>) is present in the dune-project file
  • this isn't ruled out as a future iteration, but it probably warrants further discussion
• Installing Melange libraries into dedicated prefixes (e.g. a sysroot like cross-compilation)
  • this would likely result in unnecessary churn for package managers; in the case of OPAM, cross-compilation setup is quite niche and rather difficult to setup
  • Universal libraries and Melange code aren't exactly cross-compilation, based on the necessity to vary entire implementations based on the target, whereas x-compilation is assumed to compile the same codebase, but varying the (cross-)compiler used

[anmonteiro]

This would likely also solve #10362

[rgrinberg]

The main thing that I dislike about this proposal is that it mixes low level constructs such as contexts and high level constructs as melange settings. The way things were originally, it made sense to me because contexts were expanded to cover more use cases without adding melange specific features. Now it turns that you need melange specific features anyways. In that case, we could have just started with something custom made for melange and the end result wouldn't have some of the sharp edges of this proposal.

What could a custom solution could look like? Well I suppose you could just make libraries built with the melange mode completely have their own sources and compilation artifacts and add conditional source selection for melange (though this would not work for bytecode/native/jsoo modes).

[jchavarri]

If I understand correctly, the main addition in the proposal is the melange field for contexts, which would mark a specific context as "Melange context", so that installation in those cases can be more lenient.

However, telling Dune about Melange-specific compilation at the context level feels like the wrong place (maybe this is what @rgrinberg is also expressing) because inside the contexts, one can put a lot of things besides Melange libraries or emit stanzas. This friction can be noticed in rules like:

• (context :standard) (modes melange) becomes forbidden

I assume a bunch of other things would become forbidden that are not mentioned. For example what happens if I enable some executable stanza only inside a "Melange context"? One would expect that the "context => mode" path (which is the direction I understand the proposal is following) would run into other similar friction like that.

I wonder how a proposed solution would look like if we went the "mode => context" path. Meaning, we leave modes as is, and we somehow infer the context based on it. We could do this transition progressively by adding a config in dune-project e.g. (implicit_melange_context), and then have all libraries and emit stanzas with modes melange be implicitly added to it. That way, there are no conflicts to handle because the source of truth remains at the modes level (which wouldn't need to be phased out).

This would likely also solve #10362

That issue is mostly the same idea that I described above.