On 22 Apr 2022, three years ago, I opened an issue in the OCaml package manager, opam, ‘depext does not support nixOS’. Last week, my pull request fixing this got merged!
§Let’s Encrypt Example
Before, if we tried installing an OCaml package with a system dependency we would run into:
$ opam --version
2.3.0
$ opam install letsencrypt
[NOTE] External dependency handling not supported for OS family 'nixos'.
You can disable this check using 'opam option --global depext=false'
[NOTE] It seems you have not updated your repositories for a while. Consider updating them with:
opam update
The following actions will be performed:
=== install 41 packages
...
∗ conf-gmp 4 [required by zarith]
∗ conf-pkg-config 4 [required by zarith]
∗ letsencrypt 1.1.0
∗ mirage-crypto-pk 2.0.0 [required by letsencrypt]
∗ zarith 1.14 [required by mirage-crypto-pk]
Proceed with ∗ 41 installations? [y/n] y
<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
⬇ retrieved asn1-combinators.0.3.2 (cached)
⬇ retrieved base64.3.5.1 (cached)
⬇ retrieved conf-gmp.4 (cached)
...
[ERROR] The compilation of conf-gmp.4 failed at "sh -exc cc -c $CFLAGS -I/usr/local/include test.c".
...
#=== ERROR while compiling conf-gmp.4 =========================================#
# context 2.3.0 | linux/x86_64 | ocaml-base-compiler.5.3.0 | https://opam.ocaml.org#4d8fa0fb8fce3b6c8b06f29ebcfa844c292d4f3e
# path ~/.opam/ocaml-base-compiler.5.3.0/.opam-switch/build/conf-gmp.4
# command ~/.opam/opam-init/hooks/sandbox.sh build sh -exc cc -c $CFLAGS -I/usr/local/include test.c
# exit-code 1
# env-file ~/.opam/log/conf-gmp-1821939-442af5.env
# output-file ~/.opam/log/conf-gmp-1821939-442af5.out
### output ###
# + cc -c -I/usr/local/include test.c
# test.c:1:10: fatal error: gmp.h: No such file or directory
# 1 | #include <gmp.h>
# | ^~~~~~~
# compilation terminated.
<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
┌─ The following actions failed
│ λ build conf-gmp 4
└─
...
Now, it looks like:
$ opam --version
2.4.0~alpha1
$ opam install letsencrypt
The following actions will be performed:
=== install 41 packages
...
∗ conf-gmp 4 [required by zarith]
∗ conf-pkg-config 4 [required by zarith]
∗ letsencrypt 1.1.0
∗ mirage-crypto-pk 2.0.0 [required by letsencrypt]
∗ zarith 1.14 [required by mirage-crypto-pk]
Proceed with ∗ 41 installations? [Y/n] y
The following system packages will first need to be installed:
gmp pkg-config
<><> Handling external dependencies <><><><><><><><><><><><><><><><><><><><><><>
opam believes some required external dependencies are missing. opam can:
> 1. Run nix-build to install them (may need root/sudo access)
2. Display the recommended nix-build command and wait while you run it manually (e.g. in another
terminal)
3. Continue anyway, and, upon success, permanently register that this external dependency is present, but
not detectable
4. Abort the installation
[1/2/3/4] 1
+ /run/current-system/sw/bin/nix-build "/home/ryan/.opam/ocaml-base-compiler.5.3.0/.opam-switch/env.nix" "--out-link" "/home/ryan/.opam/ocaml-base-compiler.5.3.0/.opam-switch/nix.env"
- this derivation will be built:
- /nix/store/7ym3yz334i01zr5xk7d1bvdbv34ipa3a-opam-nix-env.drv
- building '/nix/store/7ym3yz334i01zr5xk7d1bvdbv34ipa3a-opam-nix-env.drv'...
- Running phase: buildPhase
- /nix/store/sjvwj70igi44svwj32l8mk9v9g6rrqr4-opam-nix-env
<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
...
⬇ retrieved conf-gmp.4 (cached)
⬇ retrieved conf-gmp-powm-sec.3 (cached)
∗ installed conf-pkg-config.4
∗ installed conf-gmp.4
⬇ retrieved letsencrypt.1.1.0 (cached)
⬇ retrieved mirage-crypto.2.0.0, mirage-crypto-ec.2.0.0, mirage-crypto-pk.2.0.0, mirage-crypto-rng.2.0.0 (cached)
⬇ retrieved zarith.1.14 (cached)
∗ installed zarith.1.14
∗ installed mirage-crypto-pk.2.0.0
∗ installed letsencrypt.1.1.0
Done.
# To update the current shell environment, run: eval $(opam env)
§Implementation
Some background: opam has an ‘external
dependency’ (depext) system where packages can declare dependencies
on packages that are provided by Operating System package managers
rather than opam. One such depext is the GMP C library used by Zarith, which can be
installed on Debian with apt install libgmp-dev. The opam repository has
virtual conf-* packages which unify
dependencies across ecosystems, so conf-gmp contains:
depexts: [
["libgmp-dev"] {os-family = "debian"}
["libgmp-dev"] {os-family = "ubuntu"}
["gmp"] {os = "macos" & os-distribution = "homebrew"}
["gmp"] {os-distribution = "macports" & os = "macos"}
...
["gmp"] {os-distribution = "nixos"}
]
Where depexts entries are filtered according to variables describing the system package manager.
However, NixOS
has a rather
different notion of installation than other Linux distributions.
Specifically, environment variables for linkers to find libraries are
set in a Nix derivation, not when installing a package to the system. So
attempts to invoke
nix-env to provide Nix system dependencies
were limited to executables.
Instead, to use GMP, one had to
invoke nix-shell -p gmp before invoking
the build system. This is suboptimal for two reasons:
- It requires manual resolution of system dependencies.
- The resulting binary will contain a reference to a path in the Nix store which isn’t part of a garbage collection (GC) root, so on the next Nix GC the binary will stop working.
The obvious fix for the latter is to
build the binary as a Nix derivation, making it a GC root, which is what
opam-nix supports. It
uses opam to solve dependencies inside a Nix derivation, uses Nix’s Import
From Derivation to see the resolved dependencies, and creates Nix
derivations for the resulting dependencies. Using the depexts filtered
with os-distribution = "nixos" opam-nix is
able to provide system dependencies from Nixpkgs.
While working with opam-nix when building Hillingar I found it to be great for deploying OCaml programs on NixOS systems (e.g. Eon), but it was slow and unergonomic for development. Every time a dependency is added or changed, an expensive Nix rebuild is required; it’s a lot faster just to work with Opam.
On 8 Apr 2024 I got funding for a
project that included adding depext support for NixOS to opam. There
were a few false
starts along the way
but eventually I implemented a depext mechanism that
manages a nix-shell-like environment,
setting environment variables with Opam to make system dependencies
(depexts) available with Nix. We create a Nix derivation
like,
{ pkgs ? import <nixpkgs> {} }:
with pkgs;
stdenv.mkDerivation {
name = "opam-nix-env";
nativeBuildInputs = with buildPackages; [ pkg-config gmp ];
phases = [ "buildPhase" ];
buildPhase = ''
while IFS='=' read -r var value; do
escaped="''$(echo "$value" | sed -e 's/^$/@/' -e 's/ /\\ /g')"
echo "$var = $escaped Nix" >> "$out"
done < <(env \
-u BASHOPTS \
-u HOME \
-u NIX_BUILD_TOP \
-u NIX_ENFORCE_PURITY \
-u NIX_LOG_FD \
-u NIX_REMOTE \
-u PPID \
-u SHELLOPTS \
-u SSL_CERT_FILE \
-u TEMP \
-u TEMPDIR \
-u TERM \
-u TMP \
-u TMPDIR \
-u TZ \
-u UID \
-u PATH \
-u XDG_DATA_DIRS \
-u self-referential \
-u excluded_vars \
-u excluded_pattern \
-u phases \
-u buildPhase \
-u outputs)
echo "PATH += $PATH Nix" >> "$out"
echo "XDG_DATA_DIRS += $XDG_DATA_DIRS Nix" >> "$out"
'';
preferLocalBuild = true;
}Which is very similar to how nix-shell and its successor nix develop work under the hood, and we get the
list of variables to exclude
and append
too from the nix develop source. We build
this Nix derivation to output a file in Opam’s environment variable
format containing variables to make depexts available. This environment
file is a Nix store root, so its dependencies won’t be garbage collected
by Nix until the file is removed. This depext mechanism is quite
different to the imperative model most other system package managers
used, so required a fair amount of refactoring to be plumbed through the
codebase.
A really cool aspect of this depext mechanism is that it doesn’t interfere with the system environment, so it allows totally isolated environments for different projects. This could be useful to use on even non-NixOS systems as a result.
Opam’s Nix depext mechanism has been merged and released in Opam 2.4~alpha1, which you can use on NixOS with this overlay:
opam = final.overlay-unstable.opam.overrideAttrs (_: rec {
version = "2.4.0-alpha1";
src = final.fetchurl {
url = "https://github.com/ocaml/opam/releases/download/${version}/opam-full-${version}.tar.gz";
sha256 = "sha256-kRGh8K5sMvmbJtSAEEPIOsim8uUUhrw11I+vVd/nnx4=";
};
patches = [ ./pkgs/opam-shebangs.patch ];
});And can be used from my repository directly:
$ nix shell github:RyanGibb/nixos#legacyPackages.x86_64-linux.nixpkgs.opamAnother part of this project was bridging version solving with Nix1 in opam-nix-repository which has continued into the Enki project.
Thanks to David, Kate, and Raja for all their help, and to Jane Street for funding this work.