Nix, the purely functional build system

This blog post is be about Nix, "The Purely Functional Package Manager", a software project as mind-blowingly useful as it is difficult to succinctly define. What makes Nix so peculiar is that it doesn't seem to aim at implementing specific functionality, instead implementing primitives that make this functionality trivial. This cuts both ways: on the one hand, it results in a radical simplification of a whole slew of build system and package management tooling, on the other hand it can be quite puzzling for beginners.

While there are many resources praising Nix as a package manager, it is rarely talked about as a build system, which it most certainly is. In this post I am going to lay out the problem with traditional build systems and the way Nix solves it, as well as demonstrate its benefits using a real-world example, my color space project HSLuv.

Table of Contents

The build environment problem

Consider this Makefile rule that makes hello.o from hello.c:

hello.o: hello.c
    gcc -c hello.c

Looks simple, but will make hello.o work as expected? This is not certain, because it depends on the presence of GCC and on its version, and this is precisely where traditional build systems throw in the towel. They place the burden of reproducing the build environment on the programmer, often arming him with little more than a list of software to install manually.

This problem is generally confronted at the level of a language ecosystem. For example, in Node.js, you use package.json to specify your build tools. In Python, you might use a Pipfile or requirements.txt. And likewise in every other language, which introduces new problems:

  1. Duplicated effort that goes into build system and dependency management tooling
  2. Duplicated effort that goes into learning the aforementioned tools
  3. Solution is limited to tools that have been packaged for the language at hand, often by reimplementing them in this language. If the tool you are looking for is not available, you have to install it manually or use multiple dependency management tools side-by-side, e.g. NPM and pip.

The Nix solution

Look again at the sample Makefile from the previous section. Imagine yourself as its original developer. How do you make sure that when the build is run by another developer, the output is the same? By making sure the inputs are identical and the build operation is deterministic [1]. In other words, by making sure the build operation is a pure function.

It is not hard to see that a Makefile rule is essentially a function whose arguments and return value are declared in the first line separated by a colon. How would this function be written in a purely functional programming language? Keep reading to find out.

What is Nix?

Nix is defined by its creators as "The Purely Functional Package Manager", but it is helpful to see Nix as part of a set of tools that, like all good software, is structured in layers:

  1. Nix, the purely functional programming language
  2. Nix, the build system (think of GNU Make)
  3. Nix, the package manager
  4. NixOS, the Linux distribution
  5. NixOps, the NixOS deployment/configuration/provisioning tool

The project is hosted on nixos.org and appears to advertise its Linux distribution as the most important layer, but I believe for most developers, most of the benefits can be reaped at the level of the package manager, which can be installed on MacOS or a Linux distro of your choosing.

Nix, the programming language

When Nix is discussed online, one of the top comments is always a criticism of its syntax as if was a fatal flaw of the language. Having actually used Nix, I was puzzled to see this phenomenon. Then I realized where it was coming from and now I happily collapse these threads. The syntax is fine, trust me.

For the purposes of this post it is enough to know that the Nix expression language is really small, covered entirely in a short section of the Nix manual. Alternatively, you can learn it as part of Luca Bruno's excellent Nix Pills tutorial, which in my opinion is the best way to learn Nix.

Nix, the build system

What turns a purely functional programming language into a purely functional build system is the concept of a derivation, a first-class equivalent to a Makefile rule, i.e. a script that produces a set of files from a set of inputs: other files or configuration values. After installing Nix, create a file called default.nix with the following contents:

rec {
  pkgs = import <nixpkgs> {};
  hello = pkgs.stdenv.mkDerivation rec {
    name = "hello";
    builder = builtins.toFile "builder.sh" ''
      source $stdenv/setup
      echo "hello world" > $out
    '';
  };
}

Here we have defined a derivation named hello that creates a text file. The output of a derivation is a file that gets created at the path provided by the environment variable $out. You can do anything inside the builder script, as long as you create a file or directory at $out. What about that pkgs object? Doesn't it look like useless boilerplate? It sure does, but fear not, we will be making use of it shortly.

In the meantime, let's build the derivation by running nix-build -A hello from the same directory as the file above:

$ nix-build -A hello
these derivations will be built:
  /nix/store/yhca2sy0z8ilkgymsyyj2l02xxbqk7i8-hello.drv
building path(s) ‘/nix/store/nydwxxqnxigsqslvri4hm4yd71al8dxy-hello’
/nix/store/nydwxxqnxigsqslvri4hm4yd71al8dxy-hello

Nix stores the output in /nix/store but creates a link in the current directory so you can inspect it:

$ cat result
hello world

You can input files into a derivation by referencing them relative to the .nix file directory. Inputs are defined as arbitary name/value pairs in the mkDerivation parameter and get passed into the builder script as identically named environment variables. This means in the example below, the path to the file ./foo.txt will be passed in as the environment variable $foo:

rec {
  pkgs = import <nixpkgs> {};
  demo = pkgs.stdenv.mkDerivation rec {
    name = "demo";
    foo = ./foo.txt;
    builder = builtins.toFile "builder.sh" ''
      source $stdenv/setup
      mkdir $out
      install $foo $out/foo1.txt
      install $foo $out/foo2.txt
    '';
  };
}

Similarly you can reference other derivations:

rec {
  pkgs = import <nixpkgs> {};

  hello = pkgs.stdenv.mkDerivation rec {
    name = "hello";
    builder = builtins.toFile "builder.sh" ''
      source $stdenv/setup
      echo "hello world" > $out
    '';
  };

  demo = pkgs.stdenv.mkDerivation rec {
    inherit hello;
    name = "demo";
    foo = ./foo.txt;
    builder = builtins.toFile "builder.sh" ''
      source $stdenv/setup
      mkdir $out
      install $foo $out/foo.txt
      install $hello $out/hello.txt
    '';
  };
}

When you build the derivation demo, you automatically build its dependent derivation hello:

$ nix-build -A demo
these derivations will be built:
  /nix/store/q2nmhdn3r1nn50rybgihw59irz40z1gp-hello.drv
  /nix/store/bsa1bnp05dbdls1hhfv9hbsfr8sp06aj-demo.drv
building path(s) ‘/nix/store/gybvj34iq35rcwm2pggrrx5jpdf7rnz6-hello’
building path(s) ‘/nix/store/nwkjm3i41v30zq6k9jb3nq2m4rw8p524-demo’
/nix/store/nwkjm3i41v30zq6k9jb3nq2m4rw8p524-demo
$ ls result
foo.txt         hello.txt

This is all fine and good, but where are the promised build tools? The compilers, the libraries, the minifiers and uglifiers? Read on.

Nixpkgs, the Nix Packages collection

To install the necessary build tools, run the following list of commands. Just kidding. The tools are already there. Where? In that pkgs object we saw in the previous section. See for yourself by creating the following default.nix file:

rec {
  pkgs = import <nixpkgs> {};
  python = pkgs.python36;
}

Believe it or not, python here is a derivation no different from the ones we created in the previous section. Check it out:

$ nix-build -A python
/nix/store/q9xwwp0k8lim1lr511xqakjxs2iv3j67-python3-3.6.4-env
$ result/bin/python
Python 3.6.4 (default, Jan 15 2018, 17:40:15)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

And because python is a derivation like any other, we can pass it in as input to our own derivation. Let's test it out by using Python's handy command line JSON pretty-printer:

rec {
  pkgs = import <nixpkgs> {};
  python = pkgs.python36;

  demo = pkgs.stdenv.mkDerivation rec {
    inherit python;
    name = "demo";
    builder = builtins.toFile "builder.sh" ''
      source $stdenv/setup
      echo "[1,2,3]" | $python/bin/python -m json.tool > $out
    '';
  };
}

And build:

$ nix-build -A demo
/nix/store/fy0ffzashvvjnrfwkf1an80nc1v54da6-demo
$ cat result
[
    1,
    2,
    3
]

Lastly, we need to address the issue of package pinning, ensuring that we are using a specific version of every package. For that let's take another look at the pkgs object:

rec {
  pkgs = import <nixpkgs> {};
  python = pkgs.python36;
}

Clearly it contains whatever we imported from <nixpkgs>, which is actually just a directory in $NIX_PATH. On my system, I can rewrite the above as follows:

rec {
  pkgs = import /nix/var/nix/profiles/per-user/root/channels/nixpkgs {};
  python = pkgs.python36;
}

Let's take a look at that directory:

$ ls /nix/var/nix/profiles/per-user/root/channels/nixpkgs
COPYING         doc             nixos
README.md       lib             pkgs
default.nix     maintainers     svn-revision

Now I'm going to let you in on a secret: that directory is simply a local copy of the nixpkgs repository. All you need for package pinning is to use a specific revision of the nixpkgs repo:

rec {
  pkgsLocal = import <nixpkgs> {};
  pkgs = import (pkgsLocal.fetchzip {
    url = "https://github.com/NixOS/nixpkgs/archive/866717d75b64cb07cab16d0357edfd00cf339c20.zip";
    sha256 = "0ikz6m801gfmgzd4q0la5pcivl46yiviad5gvz0qba0pa7wc8g0g";
  }) {};
  # contents of `pkgs` is pinned to revision 866717d75b64cb07cab16d0357edfd00cf339c20
}

Here we are importing the local version of nixpkgs for its fetchzip function, then letting Nix fetch and unpack an archive with a specific revision of nixpkgs, then we import the unpacked directory the same way we import any other local path. Does this mean Nix will have to fetch and unpack the archive every time we run nix-build? No, that's what the sha256 hash is for. It lets Nix permanently cache the contents in /nix/store such that subsequent invocations are near-instantaneous.

Case Study: HSLuv

For a less trivial example, let's look at how Nix is used for HSLuv.

Static website

A good demonstration at Nix's ability to gather up the necessary tools and put them together in a reproducible manner is building www.hsluv.org. You can try it yourself by cloning our GitHub repo and running:

$ nix-build -A website
...
/nix/store/yvwv7dvfgfm4bb83xbmrilabxcra3bbd-hsluv-website
$ ls result
CNAME           credits         favicon.png     implementations math
comparison      examples        images          index.html      static

Here is what went into building that:

  1. OpenJDK version 1.8.0
  2. Haxe 3.4.4
  3. Node.js 6.12.2
  4. mustache.js 2.3.0
  5. PNG.js 3.0.0
  6. Closure Compiler v20170910 (depends: #1)
  7. Haxe-generated JavaScript from Hsluv.hx (depends: #2)
  8. HSLuv Node.js package (depends: #7)
  9. static demo images from generate-images.js (depends: #3, #5, #8)
  10. static HTML from generate-html.js (depends: #3, #4)
  11. picker.min.js for color picker demo (depends: #7, #6)
  12. full static website (depends: #9, #10, #11)

Not bad for a single command.

Packaging and publishing automation

HSLuv is packaged for NPM, PyPI, Maven, NuGet, LuaRocks, RubyGems et al. To get a sense of the thrill and excitement of working through the peculiarities of these systems, imagine yourself doing taxes in 6 countries simultaneously. It's the kind of joy that's best experienced once. Luckily, we can automate it: the packaging, the signing, the publishing, the retrieval of all the necessary tools, everything.

Let's use Node.js as an example. Take a look at the derivation that creates the Node.js package:

nodePackageDist = makeNodePackage {
  jsFile = haxeJsCompile "hsluv.Hsluv";
  exportFile = ./javascript/api-public.js;
};

Note that it makes use of two functions: makeNodePackage and haxeJsCompile [2]. What kind of functional programming language would Nix be if it didn't allow you to define your functions? In this case, makeNodePackage returns a derivation, which means you can do the following:

$ nix-build -A nodePackageDist
/nix/store/1nvjb8crxq6gw7jnjgj8m5643snlizyy-js-node-package
$ ls result
README.md       hsluv.js        package.json

Now we need to write a script that would publish this package to NPM. Our first instinct is to put our script into a derivation. But should we? Actually not, for the following reasons:

  1. A publishing script is going to make use of secret keys we don't want leaking into /nix/store.
  2. A publishing script really cannot be thought of as a pure function. It doesn't always produce an output, and it is often necessary to rerun the same script with the same inputs, in which case Nix, with its pure-function assumption, will hit the cache (/nix/store) instead.

There is a simple solution to this conundrum. Instead of writing a derivation to perform the publishing we will write a derivation that will produce the script to perform the publishing. Luckily, Nix provides a convenient function just for this purpose: pkgs.writeShellScriptBin:

publishScript = pkgs.writeShellScriptBin "script.sh" ''
  # npm adduser creates .npmrc file in HOME
  TEMP_HOME=`mktemp -d`
  HOME="$TEMP_HOME"
  echo -e "$NPM_USER\n$NPM_PASS\n$NPM_EMAIL\n" | ${pkgs.nodejs}/bin/npm adduser
  ${pkgs.nodejs}/bin/npm publish ${nodePackageDist}
  rm -rf "$TEMP_HOME"
'';

Let's go through this. First of all, note that pkgs.writeShellScriptBin returns a derivation that produces a script, in this case called script.sh with the contents specified in the multiline string. Pay attention to the variables in this script. The curly brace notation (e.g. ${pkgs.nodejs}) is parsed by Nix, not Bash. This means that Nix will insert the location of the npm executable into the script. Likewise for ${nodePackageDist}. Let's see what the script looks like:

$ nix-build -A publishScript
/nix/store/a2ssw0i6a7j7489mf6q545as62n8rjpv-script.sh
$ cat result/bin/script.sh

#!/nix/store/4cvvpbqdkrb0hr4p47q3s9l2a6k04y1g-bash-4.4-p12/bin/bash
# npm adduser creates .npmrc file in HOME
TEMP_HOME=`mktemp -d`
HOME="$TEMP_HOME"
echo -e "$NPM_USER\n$NPM_PASS\n$NPM_EMAIL\n" | /nix/store/628m9fgdqj0vcxji1lwywc0nvagrq41i-nodejs-6.12.2/bin/npm adduser
/nix/store/628m9fgdqj0vcxji1lwywc0nvagrq41i-nodejs-6.12.2/bin/npm publish /nix/store/1nvjb8crxq6gw7jnjgj8m5643snlizyy-js-node-package
rm -rf "$TEMP_HOME"

Note that everything here comes from Nix: the publishing tools, the package to be published, and, if you look at the shebang, even the shell that will execute this script! The build environment is completely isolated from your system. Now all we need to do is run the script, passing in the secret keys as environment variables. In HSLuv, we keep our secret keys in secrets.txt, a file that contains a series of export statements like the following:

export NPM_USER=hsluv
export NPM_PASS=REDACTED
export NPM_EMAIL=hsluvcontributors@gmail.com

export PYPI_USERNAME=hsluv
export PYPI_PASSWORD=REDACTED

...

So to publish to NPM, we need to source secrets.txt and exec result/bin/script.sh. For this purpose I made a tiny wrapper in the root folder of HSLuv called run.sh:

#!/usr/bin/env bash

HSLUV_ROOT="$(dirname ${0})"
SCRIPT_DIR="$(nix-build -A ${1} --no-out-link ${HSLUV_ROOT}/default.nix)"
source "${HSLUV_ROOT}/secrets.txt"
exec "${SCRIPT_DIR}/bin/script.sh"

It takes the name of the publishing script derivation as a command line argument (${1}), so in order to use it to publish the NPM package discussed in this section, you only need one line:

$ ./run.sh publishScript

Now that's what I call automation.

Hope you found this post useful. If you see any errors or have any suggestions, let me know in the comments!

[1]In a certain pedantic sense, every program is deterministic, provided we know the inputs. The problem is that programs have inputs that are not passed in explicitly, such as when they request system time or random numbers from the OS. Nix mitigates many of these sources of "nondeterminism" so this is rarely a problem in practice.
[2]Functions are outside the scope of this blog post. To see the definitions of the functions referenced here, go to default.nix in the HSLuv repository. To learn more about Nix functions, see this quaint little section in the Nix manual.