Configuration management is the foundation that makes modern infrastructure possible. Tools that enable configuration management are required in the toolbox of any operations team, and many development teams as well. Although all the tools aim to solve the same basic set of problems, they adhere to different visions and exhibit different characteristics. The issue is how to choose the tool that best fits each organization's scenarios.
This InfoQ article is part of a series that aims to introduce some of the configuration tools on the market, the principles behind each one and what makes them stand out from each other. You can subscribe to notifications about new articles in the series here.
In this article, I’ll give a short introduction to NixOS, a Linux distribution, and to Nix, the package manager on which NixOS is based. These provide a declarative approach to configuration management with many advantages to users, such as strong reproducibility and atomic upgrades and rollbacks.
The problem
As any system administrator knows, managing the configuration of a system is fraught with peril.
You upgrade one package and discover that other packages on your system no longer work. This turns out to be because some shared dependencies got upgraded as well, and they’re not perfectly backwards-compatible. Windows users call this the DLL hell, but it’s a fairly universal phenomenon.
You upgrade to the latest version of your Linux distribution. It doesn’t work properly. There is no easy way to undo the upgrade. People on the Internet scold you for not knowing you really should have done a clean reinstall.
Your production server dies. Now you need to reproduce its configuration on a new machine, but nobody knows exactly what that configuration was, because it was the result of people making manual changes over a period of time.
Your team develops an application that has many dependencies. Your developers waste a lot of time setting up their build environments. Every time a dependency changes, everybody on the team needs to upgrade their environments manually.
These problems have a number of common causes:
- Configuration and package management tools are not sufficiently declarative: there is no convenient specification that describes the desired configuration of the system, or it’s not guaranteed that the actual configuration corresponds to that specification.
- Configuration changes are realised in a “destructive” way, by overwriting the previous configuration. For instance, a package upgrade on Unix will typically update files in /usr/bin or /etc.
NixOS is a Linux distribution with a fully declarative approach to configuration management designed to overcome these problems. It builds on Nix, a package manager that builds and stores packages in a way that does away with the “destructive” model of package managers like RPM or Apt. NixOS extends this approach to the configuration management of an entire Linux system. This gives many advantages, most importantly:
- Atomic upgrades: during a package or system upgrade, the system remains in a consistent state. If you run a program at any point in time, you’ll get the old or the new version, but not something in between. Similarly, if the system crashes half-way through an upgrade, it will still work. Put in another way, upgrades are transactional.
- Rollbacks: upgrades don’t overwrite the old packages and configuration files, so if a new configuration doesn’t work, you can revert to the previous state. This also makes testing configuration changes less scary.
- Reproducibility: Nix tries very hard to ensure that a package build always produces the same result. This property extends to NixOS: deploying a NixOS configuration on another machine will yield the same result, regardless of whether this machine is a clean install or already had a previous configuration.
A quick tour of Nix
We’ll start with a brief introduction to using the Nix package manager. Installing a package via Nix works pretty much as you’d expect from any package manager. For example, the following will install Git and its dependencies and make it available to the user:
$ nix-env -i git installing 'git-1.9.3' $ git --version git version 1.9.3
However, what sets Nix apart from conventional package managers is the way in which packages are stored:
$ readlink -f $(which git) /nix/store/hn79nsyhnlwqyspwqsbmgzacny35hn3w-git-1.9.3/bin/git
That is, packages don’t live in shared directories such as /usr/bin or /usr/lib, but each have their own directory underneath the Nix store (/nix/store), such as hn79nsyhnlwqyspwqsbmgzacny35hn3w-git-1.9.3. The characters at the start of the name are a cryptographic hash of all inputs used to build the package. These include the source code of the package and the build script, but also dependencies such as the C compiler and libraries against which Git links (such as OpenSSL).
Many of Nix’s advantages flow from the use of these hashes. For instance, let’s look at what happens if we upgrade a package. Nix installs packages from sets of package descriptions called Nix expressions. One such set is the Nix Packages collection (Nixpkgs), which provides thousands of packages. You can get it by cloning its Git repository, but most users get it through a so-called Nix channel. If you’re “subscribed” to the Nixpkgs channel, you can get the most recent set of Nix expressions:
$ nix-channel --update
It may be that the channel brings an updated version of the Nix expression for Git. So we do the upgrade:
$ nix-env -u upgrading 'git-1.9.3' to 'git-2.0.0' $ git --version git version 2.0.0
However, the old version of Git is not gone, because the new one is stored in a different path in the Nix store (for example, /nix/store/5l83x6jlq9kpxf7jk6d7ly12kry8jdkk-git-2.0.0/bin/git). This means that you can roll back to the previous configuration:
$ nix-env --roll-back switching from generation 505 to 504 $ git --version git version 1.9.3
So how does this work? In Nix, whenever you install, upgrade or uninstall a package via nix-env, Nix builds a tree of symbolic links (called a user environment) pointing to the installed packages. The current user environment is reachable from the user’s PATH environment variable, through some indirections. (See Figure 1. The dashed arrows denote symlinks.) Running nix-env -u will first build or download Git and its dependencies, then build a new user environment. This will not affect the old version of Git in any way. Finally, it will update the symlink /nix/var/nix/profiles/default to point at the new user environment (e.g. moving from the old default-42-link to the new default-43-link). The latter step is an atomic action. Thus, running git will either give you the old version or the new version, but not an inconsistent mix of the two (which would be the case if Git’s files were overwritten in place).
(Click on the image to enlarge it)
Figure 1: Profiles
Packages can be removed from a profile by running nix-env -e package-name. This builds a new symlink tree from which the symlinks to the specified package have been removed. So removing a package from a profile doesn’t cause it to be deleted from disk, since the user may want to roll back at some point.
Since disk space is not quite infinite, Nix allows packages to be garbage-collected. This works pretty much like garbage collection in programming languages: Nix will delete any path in the Nix store that is not reachable from a “root”. Roots include the symlinks in /nix/var/nix/profiles but also any open files (to prevent active programs from being garbage-collected). This does require telling Nix that you don’t need to roll back anymore. For example:
$ nix-env --delete-generations 10d $ nix-collect-garbage
The first command tells Nix that you don’t want to roll back to any profile version (“generation”) older than ten days, while the second performs the actual deletion.
Users can have multiple profiles. This makes it easy to experiment with new versions of software, or to keep different versions around for different purposes. For instance, you could have multiple versions of GCC in different profiles. This works because those versions don’t interfere with each other — they’re stored in different paths in the Nix store. Similarly, Nix supports multi-user package management: you don’t have to be root to install software. This is safe because packages installed by one user don’t appear in the profile of other users. But if two users install the exact same package, it will be stored only once.
Building packages
Nix expressions are a simple purely functional language that tell Nix how to build packages. For example, here is a Nix expression for building the Nano editor:
with import <nixpkgs> { };
stdenv.mkDerivation {
name = "nano-2.3.2";
# The source tarball, downloaded into some place in the Nix store.
src = fetchurl {
url = ftp://ftp.gnu.org/pub/gnu/nano/nano-2.3.2.tar.gz;
sha256 = "1s3b21h5p7r8xafw0gahswj16ai6k2vnjhmd15b491hl0x494c7z";
};
# The dependencies, referring to variables in <nixpkgs>.
buildInputs = [ ncurses gettext ];
# This is actually unnecessary:
buildCommand =
''
tar xf $src
cd nano-*
./configure --prefix=$out
make
make install
'';
}
This calls the function stdenv.mkDerivation (a high-level abstraction that provides standard Unix dependencies such as GCC and Make) with arguments that specify the inputs to the build, such as the source code, additional dependencies such as ncurses, and a build script. (In fact, the build script can be omitted in this case because mkDerivation assumes standard Autoconf-style packages by default.)
You can install such a package via nix-env -i nano, but you can also build it without installing it into a profile:
$ nix-build ./nano.nix … /nix/store/22y58w45fskjz6k7xyryx9s6ri22j2bq-nano-2.3.2
This will leave you with a symlink ./result pointing to the result in the Nix store, so you can test the package by running ./result/bin/nano.
Building a package works as follows. Nix evaluates the given Nix expression. The result of the evaluation is a graph whose nodes are package build actions (each producing one path in the Nix store) and whose edges denote dependencies. Nix will then build each package in the graph in the right order, unless its store path already exists. Nix ensures a transactional semantics: builds can always safely be interrupted, restarted, or run in parallel.
You may have noticed that the Nix expression for Nano specifies build dependencies but not run-time dependencies. This is because Nix can figure out run-time dependencies automatically by scanning for cryptographic hashes of store paths inside the output of the build. For instance, if the ncurses dependency evaluates to the store path /nix/store/8h3mfka2jmbjgaqdh1b95h7vh28j8906-ncurses-5.9, and the nano binary contains the string 8h3mfka2… (which will be the case, because of the dynamic library search path embedded in Linux executables), then Nix will conclude that Nano has a run-time dependency on ncurses. The set of all store paths reachable in this way is called the closure of a package. Such a closure can be copied to another machine and should behave there in the same way, since it includes all run-time dependencies. Nix thus makes it easy to send a program to another machine, e.g.
$ nix-copy-closure --to server.example.org $(which firefox)
will copy your exact version of Firefox and all its run-time dependencies to a remote machine via SSH.
A purely functional package manager
So what does it mean to say that Nix is a purely functional package manager? There are several aspects to this.
The Nix expression language is lazy and purely functional, like Haskell. Laziness is particularly important, since it means that variables that refer to packages are not built unless they are needed. For instance, the expression import <nixpkgs> in the example imports the Nix Packages collection. It would be bad if all its packages were built even if they were not used.
The fact that it’s a functional language means that it enables functional abstractions (including higher-order functions such as map) to capture common code patterns.
Package builds are intended to be pure: they should only depend on their declared inputs. This means they shouldn’t download things from the network, depend on the current time, use programs in /usr/bin, and so on. Nix tries hard to enforce this: for instance, it clears the environment so that variables like PATH cannot be used to pass undeclared dependencies to builds, and optionally performs builds in a chroot so that paths like /usr are not visible. But it cannot guarantee purity because current operating systems have no way to enforce determinism; for instance, if a build script makes its output depend on the phase of the moon, there is not much Nix can do about it. But in practice, it works pretty well.
The result of building a package — the paths in the Nix store — can be seen as a purely functional data structure: they are objects that refer to each other, never change after they have been constructed, and are only deleted when they become unreachable. (Another example of the use of persistent purely functional data structures outside of functional programming is Git repositories.) A package can only be “modified” by building a new one, which may use some or all of the dependencies of the old one. This is transitive: if we change the Nix expression for ncurses and run nix-build nano.nix, then both ncurses and Nano will be rebuilt, even if the Nix expression for nano didn’t change.
The purely functional approach does have a price: upgrading a fundamental dependency, like Glibc, may be fairly expensive, since everything that depends on the updated package needs to be rebuilt as well. However, Nix has a feature that ensures that even such upgrades can be done fairly quickly: the binary cache.
Binary caches
As described above, Nix has a source-based model, like Gentoo Linux: Nix expressions describes how to build a package and its dependencies from source. This is convenient for developers, since it makes it easy to modify packages, but not so much for users, since building everything from source is slow. However, Nix allows the best of both the source- and binary-based worlds via its binary cache mechanism, which allows it to transparently “optimise” a build from source into a download of a binary.
This works as follows. Suppose that Nix has evaluated a Nix expression and computed that it needs to build the store path /nix/store/pdskwizjw8ar31hql2wjnnx6g0s6xc50-glibc-2.19. It will then first fetch the URL http://cache.nixos.org/pdskwizjw8ar31hql2wjnnx6g0s6xc50.narinfo. (cache.nixos.org is the default binary cache, but you also create your own, for instance for your internal packages.) If it exists, it will contain (a pointer to) a prebuilt binary tarball, which Nix will unpack into the store instead of building from source.
Developing with Nix
Nix is not only useful as a package manager, but also as a tool for setting up build environments for development projects. A Nix expression can serve as a declarative specification of all the dependencies that your project needs. Nix can then automatically build or download these dependencies in the exact versions required by your project, freeing developers from having to do this manually. In this use case, Nix essentially serves as a more general virtualenv (since it’s language-agnostic), or as a Vagrant that doesn’t require a virtual machine.
Nix usually performs non-interactive builds (e.g. nix-build nano.nix). But it also has a command, nix-shell, that builds or downloads all dependencies of a package, but not the package itself. Instead it drops you into an interactive shell where all necessary environment variables are set so that the dependencies in the Nix store can be found. For instance, for C/C++ dependencies (such as ncurses above) this would be the compiler and linker search paths; for Python dependencies it would be PYTHONPATH; and so on.
For instance, to get a environment for building Nano:
$ nix-shell nano.nix
If your Nix store doesn’t already have the exact versions of ncurses, gcc and the other dependencies required by the Nix expression for Nano, Nix will build or download them. So you can then interactively edit, build and test the software as usual:
[nix-shell]$ tar xf $src [nix-shell]$ cd nano [nix-shell]$ ./configure [nix-shell]$ make [nix-shell]$ ./src/nano [nix-shell]$ emacs src/nano.c … [nix-shell]$ make
Of course, Nano’s dependencies are fairly trivial. But many projects have highly specific dependencies, and changing or adding to them is a pain. Nix makes this trivial. For instance, suppose your project depends on some specific versions of dependencies:
buildInputs = [ boost149 python27 ];
One day you decide to switch to more recent versions of Boost and Python. Instead of every team member having to upgrade those dependencies locally and manually, you just change the Nix expression:
buildInputs = [ boost155 python33 ];
Then when others pull this change and re-run nix-shell, they will get the new build environment, and it will be consistent with what everybody else on the team is using.
NixOS
NixOS is a Linux distribution that uses Nix as its package manager, and more importantly, extends the purely functional approach to package management to configuration management of the entire system. This means that all static bits of a running Linux system are kept in the Nix store, built by Nix expressions. This includes not just packages but also configuration files, boot scripts, and systemd units. For instance, there is a Nix function that builds the configuration file for the OpenSSH server, resulting in a store path like
/nix/store/91cj8hvpj9563ab9kpzmyypsd77il6av-sshd_config
Almost the entire system lives in the Nix store; there is almost no /bin or /usr and only a fairly small /etc.
So why would we want this? The reason is that it lifts all of Nix’s nice properties to the level of the system as a whole:
- The system can be upgraded almost atomically. The building of a new configuration (i.e. the process of building or downloading the store paths of the new configuration) is atomic. So if the power fails halfway through an upgrade, you’ll get either the old or the new configuration, but not something in between, let alone something that doesn’t boot. However, actually switching over to the new configuration requires actions such as restarting system services (e.g., if the PostgreSQL configuration file changed, then the PostgreSQL systemd service will be restarted). Since performing these actions is not instantaneous, the system will briefly be in an inconsistent state. Even so, if you reboot or the system crashes, it will boot entirely into the new configuration.
- The system can be rolled back. This is because nothing in the Nix store gets overwritten when you switch to a new configuration. For example, NixOS’ GRUB boot menu allows you to boot into any previous configuration that hasn’t been garbage-collected:
- System configurations can easily be reproduced elsewhere.
- Configuration changes can easily be tested.
- It’s declarative: the system configuration is entirely specified by a set of Nix expressions.
Here is how a user specifies the desired configuration of a system, for example, running sshd, PostgreSQL and the KDE desktop environment:
{ config, pkgs, ... }:
{
fileSystems."/".device = "/dev/disk/by-label/nixos";
networking.hostName = "mandark";
environment.systemPackages = [ pkgs.firefox ];
services.openssh.enable = true;
services.openssh.forwardX11 = true;
services.postgresql.enable = true;
services.postgresql.enableTCPIP = true;
services.xserver.enable = true;
services.xserver.desktopManager.kde4.enable = true;
}
This file is essentially an input to a Nix function that assembles a complete Linux system by invoking numerous other functions to build the subparts. For instance, the stanza services.openssh.enable = true causes a function to be called that generates a sshd_config file and a systemd unit in the Nix store.
NixOS is modular: every aspect of the system (such as PostgreSQL or KDE) is defined in a separate module, and users can provide their own modules. In fact, the configuration file shown above is a module. In complex deployments, it’s often convenient to factor out commonality between different configurations into shared modules.
You change a system by editing the configuration specification and running the command:
$ nixos-rebuild switch
This builds the new configuration (essentially by invoking Nix to build the whole thing in the Nix store) and then switches over to the new configuration by starting, stopping and restarting system services as needed. For instance, if we remove the line services.openssh.enable = true, then sshd will be stopped; and if we change services.postgresql.enableTCPIP to false, then PostgreSQL will be restarted. (NixOS automatically figures out whether services need to be restarted by comparing the store paths of the corresponding systemd units.)
Thus, when nixos-rebuild finishes, the actual system configuration is always in sync with the specified configuration. This is a big difference with conventional Linux distributions, where the actual configuration is typically the result of “imperative” changes (think RPM post-install scripts scribbling all over /etc). In NixOS, performing a reconfiguration on an existing system will give you the same result as what you would get by doing a clean reinstall (minus mutable stable such as log files and user home directories).
In the configuration management literature, this is called a congruent model, in contrast to a convergent model, where running a configuration management tool makes the actual configuration (hopefully) “converge” on the intended one. For example, in Puppet, if you remove a stanza such as
package { "postgresql":
ensure => "installed"
}
and re-run Puppet, then the package postgresql won’t be removed, resulting in an actual configuration that doesn’t entirely correspond with the specified configuration. In NixOS, the equivalent change will cause the package to disappear from the PATH of users. This is a big deal: it means you don’t have to worry whether a clean redeployment will give you an identical result.
A nice demonstration of the power of declarative, reproducible specifications is the command nixos-rebuild build-vm, which builds a script that runs a virtual machine (using QEMU/KVM) with a configuration identical to what you would get on your actual machine if you ran nixos-rebuild switch. This gives an easy and safe way to try out configuration changes before applying them to the real system.
The declarative approach also extends to networks of machines. NixOS has a tool named NixOps that provisions and deploys networks of NixOS machines from declarative specifications. For example, the following specifies a network of two machines named database and webserver, to be provisioned as VirtualBox instances running on your local machine:
{
database =
{ deployment.targetEnv = "virtualbox";
services.postgresql.enable = true;
services.postgresql.enableTCPIP = true;
...
};
webserver =
{ deployment.targetEnv = "virtualbox";
services.httpd.enable = true;
services.httpd.documentRoot = "/data";
...
};
}
If you run nixops deploy, NixOps will create two VirtualBox instances running NixOS with the given specifications. NixOps supports different target environments; changing the value of deployment.targetEnv to ec2 will cause the machines to be created as Amazon EC2 instances. You reconfigure by editing the specification and running nixops deploy again; it will figure out what actions need to be taken to realise your changes. For instance, if you add a machine to the specification, NixOps will create it; if you remove one, NixOps will destroy it.
(Click on the image to enlarge it)
/filters:no_upscale()/articles/configuration-management-with-nix/en/resources/fig3large.jpg){kind=link}
/filters:no_upscale()/articles/configuration-management-with-nix/en/resources/fig1large.jpg){kind=link}
Figure 2: A NixOps network consisting of three VirtualBox machine
Conclusion
Nix provides a radically different way to manage packages, by borrowing from the way that languages like Haskell deal with memory. This leads to advantages like atomic upgrades and rollbacks, reproducibility, and so on. NixOS extends this to the configuration management of an entire Linux system, resulting in a Linux distribution with a truly declarative configuration model, atomic upgrades and rollbacks, reproducibility, safe testing of configuration changes, and more. NixOps further builds on this by adding automated provisioning of networks of NixOS machines from declarative specifications.
NixOS has an active and growing development community. For instance, the recent 14.04 release had around 130 contributors. Nix, NixOS and NixOps are free software and available from http://nixos.org/.
About the Author
Eelco Dolstra is a Computer Scientist at LogicBlox, Inc. He obtained a PhD in Computer Science at Utrecht University and was a postdoc at Delft University of Technology. He developed the Nix package manager as part of his PhD thesis research.
Configuration management is the foundation that makes modern infrastructure possible. Tools that enable configuration management are required in the toolbox of any operations team, and many development teams as well. Although all the tools aim to solve the same basic set of problems, they adhere to different visions and exhibit different characteristics. The issue is how to choose the tool that best fits each organization's scenarios.
This InfoQ article is part of a series that aims to introduce some of the configuration tools on the market, the principles behind each one and what makes them stand out from each other. You can subscribe to notifications about new articles in the series here.