• blog
• papers
• about

Developing opam-based OCaml projects with Nix Flakes

• 2023-06-07
• last updated 2023-06-07

Lately, I've been writing a significant amount of OCaml as part of my PhD. Instead of using the OCaml package manager (opam) command-line interface (CLI) for these projects, I prefer to use Nix to provide declarative and reproducible development environments and builds. However I still want to be able to interoperate with opam's file format and access packages from the opam repository. In this blog post we'll walk through creating a flake.nix file to do this for a hello world project at github.com/RyanGibb/ocaml-nix-hello. Our aim is to make building an OCaml project, and setting up a development environment, as simple as one command.

Nix?

I've said that Nix can provide declarative and reproducible environments and builds. Let's break down what this means:

• Declarative: instead of using imperative commands to manipulate an opam switch¹ into a desirable state for a project, we instead declare the state we want in a functional Domain Specific Language (DSL) and use Nix to build it for us.
• Reproducible: this declarative specification will give us the same result every time. It does this by pinning the inputs for a build (a 'derivation') by hash and building it in a sandboxed environment².

This aims to solve the problem of 'it works on my machine' but not elsewhere. Container images are also often used for a similar purpose, however in Nix's case we only need to specify the inputs and build rules precisely.

For an introduction to Nix and it's ecosystems, I've written more here.

Flakes

I'm taking an opinionated stance and using Nix Flakes³. Flakes are a new way to specify a source tree as a Nix project using a flake.nix. They provide a lot of benefits: pinning project dependencies using a lockfile flake.lock⁴, resolving Nix expressions in isolation⁵, provide a Nix-native⁶ way of composing Nix projects⁷, and a new CLI⁸ to use Nix. If this sounds a bit complex, just take away despite them being behind a feature flag Nix flakes are the future and are worth using for their benefits now.

To enable flakes on your NixOS system add this fragment to your configuration:

nix.settings.experimental-features = [ "nix-command" "flakes" ];

opam-nix

I've said that I still want to interoperate with opam for 2 reasons:

• If we use the opam file format to specify dependancies we can use other people's opam-based projects comparatively easily, and if other's want to use our project we aren't forcing them to use Nix.
• Relying on the set of OCaml projects packaged in Nixpkgs under ocamlPackages.<name> will leave us with 833 packages instead of the 4229 in github.com/ocaml/opam-repository/ as of 2023-03-20. We also might run into issues with dependency version resolution⁹.

Fortunately a project already exists that solves this for us: github.com/tweag/opam-nix. opam-nix translates opam packages into Nix derivations, so we can use dependencies from opam-repository. It also allows us to declare our project's dependencies in opam's format, so that other users don't have to use Nix. It uses opam's dependency version solver under the hood when building a project. Read more at www.tweag.io/blog/2023-02-16-opam-nix/.

opam-nix also reproducibly provides system dependencies (picking them up from opam depexts) through Nix's mechanisms. Nix provides great support for cross-language project dependencies in general.

A Simple Example

The minimum required to get our project building is:

{
  inputs.opam-nix.url = "github:tweag/opam-nix";

  outputs = { self, opam-nix }:
    let
      system = "x86_64-linux";
      inherit (opam-nix.lib.${system}) buildOpamProject;
      package = "hello";
    in rec {
      packages.${system} = buildOpamProject { } package ./. {
        ocaml-base-compiler = "*";
      };
      defaultPackage.${system} = packages.${system}.${package};
    };
}

Documentation for buildOpamProject can be found at github.com/tweag/opam-nix/#buildOpamProject.

This is sufficient to build the project with:

$ nix build .

We can also get a development shell and build the project outside a Nix derivation -- benefitting from the dune cache -- using:

$ nix develop . -c dune build

Each of the following sections will modify this MVP flake to add new functionality, before we combine them all into the final product.

Development Environment

A user may also want to benefit from developer tools, such as the OCaml LSP server, which can be added to the query made to opam:

 {
   inputs.opam-nix.url = "github:tweag/opam-nix";

-  outputs = { self, opam-nix }:
+  outputs = { self, nixpkgs, opam-nix }:
     let
       system = "x86_64-linux";
+      # instantiate nixpkgs with this system
+      pkgs = nixpkgs.legacyPackages.${system};
       inherit (opam-nix.lib.${system}) buildOpamProject;
       package = "hello";
     in rec {
       packages.${system} = buildOpamProject { } package ./. {
         ocaml-base-compiler = "*";
+        ocaml-lsp-server = "*";
       };
       defaultPackage.${system} = packages.${system}.${package};
+      # create a development environment with ocaml-lsp-server
+      devShells.${system}.default = pkgs.mkShell {
+        inputsFrom = [ defaultPackage.${system} ];
+        buildInputs = [ packages.${system}."ocaml-lsp-server" ];
+      };
     };
 }

Users can then launch an editor with ocaml-lsp-server in the environment with:

$ nix develop . -c $EDITOR `pwd`

For nix develop documentation see nixos.org/manual/nix/stable/command-ref/new-cli/nix3-develop.html.

Managing Dependencies

We might want to specify a specific version of the opam-respository to get more up to date packages, which we can do by tracking it as a seperate input to the flake. We can do the same with the Nixpkgs monorepo¹⁰.

 {
-  inputs.opam-nix.url = "github:tweag/opam-nix";
+  inputs = {
+    nixpkgs.url = "github:NixOS/nixpkgs";
+    opam-nix.url = "github:tweag/opam-nix";
+    opam-repository = {
+      url = "github:ocaml/opam-repository";
+      flake = false;
+    };
+    opam-nix.inputs.opam-repository.follows = "opam-repository";
+    opam-nix.inputs.nixpkgs.follows = "nixpkgs";
+  };

-  outputs = { self, opam-nix }:
+  outputs = { self, opam-nix, ... }:
     let
       system = "x86_64-linux";
       inherit (opam-nix.lib.${system}) buildOpamProject;

The opam-repository can also be chosen granularly opam-nix function call with the repos argument, but we just override opam-nix's opam-repository input. Note that some packages, notably ocamlfind, required patches to work with opam-nix. If you run into errors you can force the resolution of an old version, e.g. ocamlfind = "1.9.5";.

One can pin an input to a specific commit with, e.g.:

nix flake update --override-input opam-repository github:ocaml/opam-repository/<commit>

Materialization

Every time we call buildOpamProject, or an equivalent function that calls queryToScope under the hood, we perform a computationally expensive dependency resolution using a SAT solver. We can save the results of this query to a file with materialization¹¹.

 {
   inputs.opam-nix.url = "github:tweag/opam-nix";

-  outputs = { self, opam-nix }:
+  outputs = { self, opam-nix, ... }:
     let
       system = "x86_64-linux";
-      inherit (opam-nix.lib.${system}) buildOpamProject;
+      inherit (opam-nix.lib.${system})
+        buildOpamProject
+        materializedDefsToScope
+        materializeOpamProject';
       package = "hello";
-    in rec {
-      packages.${system} = buildOpamProject { } package ./. {
+      query = {
         ocaml-base-compiler = "*";
       };
-      defaultPackage.${system} = packages.${system}.${package};
+      resolved-scope = buildOpamProject { } package ./. query;
+      materialized-scope = materializedDefsToScope
+        { sourceMap.${package} = ./.; } ./package-defs.json;
+    in rec {
+      packages = {
+        resolved = resolved-scope;
+        materialized.${system} = materialized-scope;
+        # to generate:
+        #   cat $(nix eval .#package-defs --raw) > package-defs.json
+        ${system}.package-defs = materializeOpamProject' { } ./. query;
+      };
+      defaultPackage.${system} = packages.materialized.${system}.${package};
     };
 }

The package-defs.json file generated by cat $(nix eval .#package-defs --raw) > package-defs.json should be committed to the repository.

Overlays

We can modify derivations with Nix overlays¹².

       system = "x86_64-linux";
       inherit (opam-nix.lib.${system}) buildOpamProject;
       package = "hello";
-    in rec {
-      packages.${system} = buildOpamProject { } package ./. {
-        ocaml-base-compiler = "*";
+      overlay = final: prev: {
+        "${package}" = prev.${package}.overrideAttrs (_: {
+          # override derivation attributes, e.g. add additional dependacies
+          buildInputs = [ ];
+        });
       };
+      overlayed-scope = let
+        scope = buildOpamProject { } package ./. {
+          ocaml-base-compiler = "*";
+        };
+        in scope.overrideScope' overlay;
+    in rec {
+      packages.${system} = overlayed-scope;
       defaultPackage.${system} = packages.${system}.${package};
     };
 }

Multiple Systems

Nix flakes are evaluated hermetically and as a result don't take any arguments¹³. However different systems will have different packages built for them. We essentially parametrize based on system by different derivation paths, e.g. nix build . implicitly builds the derivation packages.${system}.default. We can support multiple systems by creating derivations for each system. flake-utils¹⁴ provides a convient mechanism for creating these derivations.

 {
   inputs.opam-nix.url = "github:tweag/opam-nix";

-  outputs = { self, opam-nix }:
-    let
-      system = "x86_64-linux";
-      inherit (opam-nix.lib.${system}) buildOpamProject;
-      package = "hello";
-    in rec {
-      packages.${system} = buildOpamProject { } package ./. {
-        ocaml-base-compiler = "*";
-      };
-      defaultPackage.${system} = packages.${system}.${package};
-    };
+  outputs = { self, opam-nix, flake-utils }:
+    flake-utils.lib.eachDefaultSystem (system:
+      let
+        system = "x86_64-linux";
+        inherit (opam-nix.lib.${system}) buildOpamProject;
+        package = "hello";
+      in rec {
+        packages.${system} = buildOpamProject { } package ./. {
+          ocaml-base-compiler = "*";
+        };
+        defaultPackage.${system} = packages.${system}.${package};
+      }
+    );
 }

Combining...

We can combine all of:

• § A Simple Example
• § Development Environment
• § Managing Dependancies
• § Materialization
• § Overlays
• § Multiple Systems

To gives us a complete flake for our project:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs";
    opam-nix.url = "github:tweag/opam-nix";
    opam-repository = {
      url = "github:ocaml/opam-repository";
      flake = false;
    };
    opam-nix.inputs.opam-repository.follows = "opam-repository";
    opam-nix.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = { self, nixpkgs, opam-nix, flake-utils, ... }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        system = "x86_64-linux";
        pkgs = nixpkgs.legacyPackages.${system};
        inherit (opam-nix.lib.${system})
          buildOpamProject
          materializedDefsToScope
          materializeOpamProject';
        package = "hello";
        query = {
          ocaml-base-compiler = "*";
        };
        overlay = final: prev: {
          "${package}" = prev.${package}.overrideAttrs (_: {
            # override derivation attributes, e.g. add additional dependacies
            buildInputs = [ ];
          });
        };
        resolved-scope =
          let scope = buildOpamProject { } package ./. query;
          in scope.overrideScope' overlay;
        materialized-scope =
          let scope = materializedDefsToScope
            { sourceMap.${package} = ./.; } ./package-defs.json;
          in scope.overrideScope' overlay;
      in rec {
        packages = {
          resolved = resolved-scope;
          materialized = materialized-scope;
          # to generate:
          #   cat $(nix eval .#package-defs --raw) > package-defs.json
          package-defs = materializeOpamProject' { } ./. query;
        };
        defaultPackage = packages.materialized.${package};
        devShells.default = pkgs.mkShell {
          inputsFrom = [ defaultPackage ];
          buildInputs = [ packages."ocaml-lsp-server" ];
        };
      }
    );
}

Try it out yourself at github.com/RyanGibb/ocaml-nix-hello/commits/main.

Continuous Integration

With a flake, we can easily create a CI job from our Nix flake to build our program. For example, a GitHub action would be:

name: ci
on:
  push:
    branches:
      - 'main'
  pull_request:
    branches:
      - "main"
  workflow_dispatch:
jobs:
  nix:
    name: Build with Nix
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: cachix/install-nix-action@v12
      - run: nix --extra-experimental-features "nix-command flakes" build

See it in action at github.com/RyanGibb/ocaml-nix-hello/actions/runs/5199834104.

Nix Store

The final benefit we'll mentione that this workflow provides is that all dependencies are stored in the global Nix store and transparently shared between projects. When they differ they're duplicated so projects don't interfere with each other. Derivations can be garbage collected to save on disk space when they're no longer used.

To garbage collect globally:

$ nix-collect-garbage

To garbage collect a specific path:

$ PATH=`readlink result`
$ rm result
$ nix-store --delete $(nix-store -qR $PATH)

Real-world Example

A full-featured example of a Nix flake building a project I've been working on recently, an effects-based direct-style Domain Name System implementation written in OCaml, can be found at github.com/RyanGibb/aeon/blob/main/flake.nix.

Conclusion

Now someone getting started with our repository can clone and build it with only:

$ git clone git@github.com:RyanGibb/ocaml-nix-hello.git
$ cd ocaml-nix-hello
$ nix build .

They can set up a development environment with:

$ nix develop -c dune build
$ nix develop -c $EDITOR `pwd`

They could also build it without manually cloning it:

$ nix shell github:RyanGibb/ocaml-nix-hello
$ hello
Hello, World!

They can even run it in a single command!

$ nix run github:ryangibb/ocaml-nix-hello
Hello, World!

If this blog post has made you curious, go try this for your own projects! Feel free to get in touch at ryan@freumh.org.

Thanks

Thanks to Alexander Bantyev (balsoft) for creating and maintaining opam-nix.