Opam's Nix system dependency mechanism

On 22 Apr 2022, three years ago, I opened an issue in the OCaml package manager, opam, ‘depext does not support nixOS’. Last week, my pull request fixing this got merged!

Some background: opam has a ‘external dependency’ (depext) system where packages can declare dependencies on packages that are provided by Operating System package managers rather than opam. One such depext is the GMP C library used by Zarith, which can be installed on Debian with apt install libgmp-dev. The opam repository has virtual conf-* packages which unify dependencies across ecosystems, so conf-gmp contains:

depexts: [
  ["libgmp-dev"] {os-family = "debian"}
  ["libgmp-dev"] {os-family = "ubuntu"}
  ["gmp"] {os = "macos" & os-distribution = "homebrew"}
  ["gmp"] {os-distribution = "macports" & os = "macos"}
  ...
  ["gmp"] {os-distribution = "nixos"}
]

Where depexts entries are filtered according to variables describing the system package manager.

However, NixOS has a rather different notion of installation than other Linux distributions. Specifically, environment variables for linkers to find libraries are set in a Nix derivation, not when installing a package to the system. So attempts to invoke nix-env to provide Nix system dependencies were limited to executables.

Instead, to use GMP, one had to invoke nix-shell -p gmp before invoking the build system. This is suboptimal for two reasons:

1. It requires manual resolution of system dependencies.
2. The resulting binary will contain a reference to a path in the Nix store which isn’t part of a garbage collection (GC) root, so on the next Nix GC the binary will stop working.

The obvious fix for the latter is to build the binary as a Nix derivation, making it a GC root, which is what opam-nix supports. It uses opam to solve dependencies inside a Nix derivation, uses Nix’s Import From Derivation to see the resolved dependencies, and creates Nix derivations for the resulting dependencies. Using the depexts filtered with os-distribution = "nixos" opam-nix is able to provide system dependencies from Nixpkgs.

While working with opam-nix when building Hillingar I found it to be great for deploying OCaml programs on NixOS systems (e.g. Eon), but it was slow and unergonomic for development. Every time a dependency is added or changed, an expensive Nix rebuild is required; it’s a lot faster just to work with Opam.

On 8 Apr 2024 I got funding for a project that included adding depext support for NixOS to opam. There were a few false starts along the way but eventually I implemented a depext mechanism that manages a nix-shell-like environment, setting environment variables with Opam to make system dependencies (depexts) available with Nix. We create a Nix derivation like,

{ pkgs ? import <nixpkgs> {} }:
with pkgs;
stdenv.mkDerivation {
  name = "opam-nix-env";
  nativeBuildInputs = with buildPackages; [ gmp ];

  phases = [ "buildPhase" ];

  buildPhase = ''
while IFS='=' read -r var value; do
  escaped="''$(echo "$value" | sed -e 's/^$/@/' -e 's/ /\\ /g')"
  echo "$var	=	$escaped	Nix" >> "$out"
done < <(env \
  -u BASHOPTS \
  -u HOME \
  -u NIX_BUILD_TOP \
  -u NIX_ENFORCE_PURITY \
  -u NIX_LOG_FD \
  -u NIX_REMOTE \
  -u PPID \
  -u SHELLOPTS \
  -u SSL_CERT_FILE \
  -u TEMP \
  -u TEMPDIR \
  -u TERM \
  -u TMP \
  -u TMPDIR \
  -u TZ \
  -u UID \
  -u PATH \
  -u XDG_DATA_DIRS \
  -u self-referential \
  -u excluded_vars \
  -u excluded_pattern \
  -u phases \
  -u buildPhase \
  -u outputs)

echo "PATH	+=	$PATH	Nix" >> "$out"
echo "XDG_DATA_DIRS	+=	$XDG_DATA_DIRS	Nix" >> "$out"
  '';

  preferLocalBuild = true;
}

Which is very similar to how nix-shell and its successor nix develop work under the hood, and we get the list of variables to exclude and append too from the nix develop source. We build this Nix derivation to output a file in Opam’s environment variable format containing variables to make depexts available. This environment file is a Nix store root, so its dependencies won’t be garbage collected by Nix until the file is removed. This depext mechanism is quite different to the imperative model most other system package managers used, so required a fair amount of refactoring to be plumbed through the codebase.

Opam’s Nix depext mechanism has been merged and released in Opam 2.4~alpha1, which you can use on NixOS with this overlay:

opam = final.overlay-unstable.opam.overrideAttrs (_: rec {
  version = "2.4.0-alpha1";
  src = final.fetchurl {
    url = "https://github.com/ocaml/opam/releases/download/${version}/opam-full-${version}.tar.gz";
    sha256 = "sha256-kRGh8K5sMvmbJtSAEEPIOsim8uUUhrw11I+vVd/nnx4=";
  };
  patches = [ ./pkgs/opam-shebangs.patch ];
});

And can be used from my repository directly:

$ nix shell github:RyanGibb/nixos#legacyPackages.x86_64-linux.nixpkgs.opam

Another part of this project was bridging version solving with Nix¹ in opam-nix-repository which has continued into the Enki project.

Thanks to David, Kate, and Raja for all their help, and to Jane Street for funding this work.