In the last few months, I migrated both my workstation and my servers (a DigitalOcean VPS and a Raspberry Pi 3) to NixOS. To best summarize the benefits, let's just say that it's like having a "dotfiles" repo, but for your entire system (or multiple!), including custom software, service configuration, drivers, kernel tweaks, etc.

While a similar result could be achieved by more mainstream configuration management tools, such as Puppet or Ansible, they do not integrate deeply into your OS. As such, over a longer period it's likely for your system to accumulate all sorts of manual tweaks done outside of your configuration management framework. The system's configuration is not fully described by your declarative configuration anymore and you will likely not reproduce the system exactly if you have to recreate it from scratch.

NixOS treats the system as mostly immutable and makes it way harder to mess something up: you can't just edit the files under /etc or upgrade globally installed packages by hand. Most meaningful changes you'll only be able to do via edits to configuration.nix and nixos-rebuild switch, which replaces the entire system with a new generation. This also provides an ability to rollback everything easily and effectively, which conventional tools often lack or implement incompletely.

Why this article was made

Despite the situation getting progressively better, while learning Nix and NixOS I still felt like some parts of the stack are under-documented. While there are extensive resources such as the NixOS manual, Nix pills, or nix.dev; what I lacked the most was how-tos or recipes which would show how to connect the tools to achieve your desired goal. Time and time again, I had to resort to reading others' configurations and the source code in the nixpkgs repository.

This series of articles intends to fill this niche somewhat. Most of these methods are the same ones I use to run my own "private cloud". My own infrastructure is quite pedestrian - I only run a few services, including hosting my websites and email - so it's not exactly a guide for "enterprise battle-tested infrastructure". However, once I nailed the process down, I have definitely had a pleasant experience, and my current setup is definitely satisfactory for my needs.

Who this is for

If you - like me a while ago - are getting into Nix / NixOS, but still struggling to put your newly-acquired knowledge to use, you've come to the right place.

I do not intend to teach the core concepts of Nix or the basics of the expression language. The aforementioned resources should get you up and running if you are a complete newbie. Instead, I'll focus on one specific use case - deploying a cloud server running NixOS and managing its configuration from your own workstation.

In this article, we will learn how to:

• Generate a custom NixOS image for DigitalOcean
• Create a virtual NixOS server on the cloud
• Deploy nginx on the server using Morph

Before we begin, let's go over the tools we'll need one by one.

Nix

You should either be running NixOS, or have Nix installed under your preferred Linux distribution.

Again, I assume you have the basic gist of the Nix language. Basically, if you've written at least one Nix derivation, or played around with your configuration.nix, you should be fine.

DigitalOcean

DigitalOcean is a cloud hosting provider. I chose it simply because it is the provider I have been using for the past several years. It works well for me and allows custom images, which is gonna be really important soon.

If you'd like to, you can sign up on DigitalOcean through my referral link and get $100 credit to use freely for 60 days.

Morph

Morph is a tool that allows deploying NixOS machines remotely as easily as you could update your own machine's configuration.

There are multiple competing NixOS deployment tools, including the official NixOps, rival Morph, and a new contender called Colmena. So, why choose Morph specifically?

As Morph's README states, it and similar tools are:

basically fancy wrapper[s] around nix-build, nix copy, nix-env, switch-to-configuration, scp and more

Thus it does not matter that much which tool we use, as all of these tools:

• Use the Nix language itself to define your configuration
• Use similar Nix-native methods for the deployment process itself
• Have a similar layout for the network expressions

Most things you'll learn from this series will be applicable to any of these tools.

As NixOps is currently in the flux somewhere between version 1.7 and 2.0, where 2.0 adds important features, but does not have a stable release yet, it is hard to recommend (although I do use its master branch personally). Colmena might get there soon, as it gains features and polish, but for now, Morph seems like the best alternative.

In particular, Morph has a couple of features I really like. The first is allowing to easily pin a specific version of nixpkgs for your machines. The second is being completely stateless: unlike NixOps, it does not manage a set of deployments in a hidden file somewhere in your home directory. Instead, it does only what you explicitly tell it to do - deploy to machines specified in your .nix file.

Preparation

Enough bikeshedding, let's get down to business. We will need a few things before we are ready.

nix-shell

Let's make a new folder to fit all of our stuff in:

$ mkdir part1 && cd part1

Then, let's define a new shell that will provide the morph package, as well as curl which we will use to query our newly deployed web server. Put this in a new file called shell.nix:

{ pkgs ? import <nixpkgs> { } }:
pkgs.mkShell {
  buildInputs = with pkgs; [ curl morph ];
}

Then enter the shell and you should be able to launch both tools:

$ nix-shell
$ curl --version | head -n 1
curl 7.72.0 (x86_64-pc-linux-gnu) libcurl/7.72.0 OpenSSL/1.1.1i zlib/1.2.11 libssh2/1.9.0 nghttp2/1.41.0
$ morph --version
1.5.0

Great success!

Preparing your DigitalOcean project

Custom image

Besides supporting major distributions out-of-the-box, DigitalOcean also supports custom disk images, which means you can run basically any OS. We will use the DO image generation functionality in nixpkgs to generate one of our own.

Put this in a new file called image.nix:

{ pkgs ? import <nixpkgs> { } }:
let config = {
  imports = [ <nixpkgs/nixos/modules/virtualisation/digital-ocean-image.nix> ];
};
in
(pkgs.nixos config).digitalOceanImage

Then execute:

$ nix-build image.nix

This may take several minutes. Do not worry, we'll only need to do this once. Once the build has finished, you should have a new file under the newly created result/ directory:

$ ls -sh result/*
390M result/nixos.qcow2.gz

A custom image has been built! You can now upload this image to your DigitalOcean account.

SSH key

You should also upload your SSH public key. The custom image we just generated has a hidden superpower: it automatically pulls in the public SSH keys from your DigitalOcean account at creation time. This will let us login via SSH without needing to create a root password.

Create your NixOS droplet!

Finally, we can create a droplet (that's what Digital Ocean calls virtual servers).

A few things to note:

• Remember to choose your custom image, called nixos.qcow2.gz
• Remember to enable your SSH key under the "Authentication" section. For me, the checkbox was checked by default
• I named my droplet nixie. Doing the same will make it easier to follow along
• Feel free to choose the smallest ($5/month) flavor. We will not need anything too powerful

You are charged for all of your droplets hourly, even if they are turned off, so remember to destroy anything you do not need anymore in order to not rack up a huge bill.

Verify

Once the droplet has been created, you will see its IP address on the dashboard. Let's try to connect.

$ ssh root@198.51.100.207
The authenticity of host '198.51.100.207 (198.51.100.207)' can't be established.
ED25519 key fingerprint is SHA256:MtP743nmdmL59nJYYeGrvJVw8sNkqpQGvl2yyD5PMOA.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '198.51.100.207' (ED25519) to the list of known hosts.

[root@nixie:~]# nixos-version
20.09.2290.647cc06986c (Nightingale)
[root@nixie:~]# exit

Works fine!

Note: this initial NixOS version will depend on your workstation's version of nixpkgs. This is not an ideal situation, but it works fine for provisioning. We will solve this later on.

The custom image we built the droplet from has some important directives in its initial configuration.nix. Let's retrieve this file for later use:

$ scp root@198.51.100.207:/etc/nixos/configuration.nix .
$ chmod 644 configuration.nix  # scp keeps the original 444 permissions, but we will need to edit this file

Managing the droplet with Morph

When using NixOS on your machine, you edit your configuration in /etc/nixos/configuration.nix and activate it via sudo nixos-rebuild switch. However, this "every machine governs itself" model will not scale gracefully as you add remote hosts, especially if the amount of them grows large. Instead, we would like to have all the configuration in one place (a Git repo works well) and be able to push out configuration changes from our workstation to all of these machines using a tool like Morph.

However, this approach leaves /etc/nixos/configuration.nix behind. Your system is now running whatever Morph has built the system from, and only that.

As the custom image we generated contains important stuff to run the system on a DigitalOcean droplets, such as bootloader and hard disk configuration, we will need to keep that when moving to Morph.

Let's check what's inside of the current configuration we've downloaded:

$ cat configuration.nix
{ modulesPath, lib, ... }:
{
  imports = lib.optional (builtins.pathExists ./do-userdata.nix) ./do-userdata.nix ++ [
    (modulesPath + "/virtualisation/digital-ocean-config.nix")
  ];
}

Seems a bit short? That's because everything important is hidden away in the digital-ocean-config.nix import. If you are interested what it looks like under the hood, feel free to read the source code in nixpkgs.

For now though, we will rename it to use as a basis for our deployment specification.

$ mv configuration.nix network.nix

Open this file, now called network.nix in your favorite editor. We will do a few small edits to make it look like this instead:

{
  nixie = { modulesPath, lib, ... }: {
    imports = lib.optional (builtins.pathExists ./do-userdata.nix) ./do-userdata.nix ++ [
      (modulesPath + "/virtualisation/digital-ocean-config.nix")
    ];

    deployment.targetHost = "198.51.100.207";
    deployment.targetUser = "root";
  };
}

Whereas previously we've had the function as the only thing in a file, now the top level value is a set with one key nixie pointing to this function. This is how Morph and similar tools define your servers: one key means one machine to deploy to, and the corresponding value is a function that generates that machine's system configuration, just like the one that might exist in your NixOS workstation's configuration.nix.

Besides that, we added important Morph-specific options: the address of the host to deploy to and the user to connect as.

Let's try to deploy now:

$ morph deploy network.nix switch
Selected 1/1 hosts (name filter:-0, limits:-0):
          0: nixie (secrets: 0, health checks: 0)

error: The option `networking.hostName' has conflicting definitions, in `/nix/store/p3bjy5vfknzn1dxq72gb8wwa33bs7hfr-nixos/nixos/modules/virtualisation/digital-ocean-config.nix' and `<unknown-file>'.
<...snip...>

Oops! It seems that the hostname Morph tries to configure automatically conflicts with what is defined in digital-ocean-config.nix. We can fix that by defining it manually in network.nix:

diff --git a/part1/network.nix b/part1/network.nix
index 52e8932..02db4f2 100755
--- a/part1/network.nix
+++ b/part1/network.nix
@@ -1,10 +1,12 @@
 {
-  nixie = { modulesPath, lib, ... }: {
+  nixie = { modulesPath, lib, name, ... }: {
     imports = lib.optional (builtins.pathExists ./do-userdata.nix) ./do-userdata.nix ++ [
       (modulesPath + "/virtualisation/digital-ocean-config.nix")
     ];

     deployment.targetHost = "198.51.100.207";
     deployment.targetUser = "root";
+
+    networking.hostName = name;
   };
 }

For those not fluent in patches, this is what our network.nix looks like so far:

{
  nixie = { modulesPath, lib, name, ... }: {
    imports = lib.optional (builtins.pathExists ./do-userdata.nix) ./do-userdata.nix ++ [
      (modulesPath + "/virtualisation/digital-ocean-config.nix")
    ];

    deployment.targetHost = "198.51.100.207";
    deployment.targetUser = "root";

    networking.hostName = name;
  };
}

The new parameter name that we added to the function definition will be filled out with the same value as the key, in this case, "nixie".

Let's try deploying again:

$ morph deploy network.nix switch

The first deployment will likely take a while. However, on the following deployments, only the parts that changed, e.g. newly added or updated packages, or daemon configuration files, will be rebuilt.

The output of Morph will look like something this:

Selected 1/1 hosts (name filter:-0, limits:-0):
          0: nixie (secrets: 0, health checks: 0)

<...snip...>

Executing 'switch' on matched hosts:

** nixie

<...snip...>

Running healthchecks on nixie (198.51.100.207):
Health checks OK
Done: nixie

Finally some payoff!

Deploying nginx

At last, it's time to deploy something more than just an empty system. We will now make these changes to our configuration:

• Enable the nginx HTTP server
• Add a default virtual host and a simple "OK" response to HTTP requests
• Add a health check that Morph will use to ensure nginx is working properly

diff --git a/part1/network.nix b/part1/network.nix
index 02db4f2..8ab724a 100755
--- a/part1/network.nix
+++ b/part1/network.nix
@@ -8,5 +8,24 @@
     deployment.targetUser = "root";

     networking.hostName = name;
+
+    deployment.healthChecks = {
+      http = [
+        {
+          scheme = "http";
+          port = 80;
+          path = "/";
+          description = "check that nginx is running";
+        }
+      ];
+    };
+
+    services.nginx = {
+      enable = true;
+      virtualHosts.default = {
+        default = true;
+        locations."/".return = "200 \"Hello from Nixie!\"";
+      };
+    };
   };
 }

Time to deploy again!

$ morph deploy network.nix switch
Selected 1/1 hosts (name filter:-0, limits:-0):
          0: nixie (secrets: 0, health checks: 1)
<...snip...>
Running healthchecks on nixie (198.51.100.207):
        * check that nginx is running: Failed (Get "http://198.51.100.207:80/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))

Uh oh. Our health check is failing. Can you spot the issue? An experienced NixOS-er will immediately tell what's wrong: by default, NixOS has firewall enabled and we have not specified that we want to open the port 80. Let's fix that:

diff --git a/part1/network.nix b/part1/network.nix
index 8ab724a..ff078d3 100755
--- a/part1/network.nix
+++ b/part1/network.nix
@@ -20,6 +20,8 @@
       ];
     };

+    networking.firewall.allowedTCPPorts = [ 80 ];
+
     services.nginx = {
       enable = true;
       virtualHosts.default = {

And try our deployment one more time:

$ morph deploy network.nix switch
Selected 1/1 hosts (name filter:-0, limits:-0):
          0: nixie (secrets: 0, health checks: 1)

<...snip...>

Running healthchecks on nixie (198.51.100.207):
        * check that nginx is running: OK
Health checks OK
Done: nixie

Seems to pass now. Might as well check it via curl, just to be sure:

$ curl http://198.51.100.207
Hello from Nixie!

Yay! We have an nginx instance running on our NixOS server.

Pinning nixpkgs

As mentioned before, currently the server's NixOS version depends on the version of Nixpkgs in your local workstation. That is not always the best idea: for example, you might want to run your workstation on the bleeding edge by using nixpkgs-unstable, but keep your servers on a more stable channel.

Luckily, Morph allows us to change what version of nixpkgs is used to deploy our network:

diff --git a/part1/network.nix b/part1/network.nix
index e89b9b1..3643f5f 100755
--- a/part1/network.nix
+++ b/part1/network.nix
@@ -1,4 +1,15 @@
 {
+  network = {
+    pkgs = import
+      (builtins.fetchGit {
+       name = "nixos-21.11-2021-12-19";
+       url = "https://github.com/NixOS/nixpkgs";
+       ref = "refs/heads/nixos-21.11";
+       rev = "e6377ff35544226392b49fa2cf05590f9f0c4b43";
+      })
+      { };
+  };
+
   nixie = { modulesPath, lib, name, ... }: {

The following expression will check out the commit e6377ff3 from the nixos-21.11 branch and use it as the basis for all of the machines in the network. The name is allowed to be an arbitrary string. I use it to mark the date of the last update of the packages.

Now, let's redeploy and check out what version we are running.

$ morph deploy network.nix switch
<...snip...>
$ morph exec network.nix nixos-version
Selected 1/1 hosts (name filter:-0, limits:-0):
          0: nixie (secrets: 0, health checks: 1)

** nixie
21.11pre-git (Nightingale)

While the default nixos-version utility is not willing to return a commit hash, we do see that we are running from a git checkout, rather than the version we observed before. For more information on various nixpkgs channels, check out this page on the unofficial wiki.

State version

It is also a good idea to explicitly set your system.stateVersion. I will refrain from explaining this option in depth, as an explanation can be found in the wiki. Basically, some stateful services (e.g. Postgres) need manual intervention, such as data migration, when upgrading to a new major version. system.stateVersion prevents breakage that could result from upgrading these packages without the necessary precautions: the affected services will keep using compatible versions, even if we upgrade our NixOS version to a new one.

diff --git a/part1/network.nix b/part1/network.nix
index 3643f5f..04eedaf 100755
--- a/part1/network.nix
+++ b/part1/network.nix
@@ -20,6 +20,8 @@

     networking.hostName = name;

+    system.stateVersion = "21.11"; # Do not change lightly!
+
     deployment.healthChecks = {
       http = [
         {

That's it

Here's the finished network.nix:

{
  network = {
    pkgs = import
      (builtins.fetchGit {
        name = "nixos-21.11-2021-12-19";
        url = "https://github.com/NixOS/nixpkgs";
        ref = "refs/heads/nixos-21.11";
        rev = "e6377ff35544226392b49fa2cf05590f9f0c4b43";
      })
      { };
  };

  nixie = { modulesPath, lib, name, ... }: {
    imports = lib.optional (builtins.pathExists ./do-userdata.nix) ./do-userdata.nix ++ [
      (modulesPath + "/virtualisation/digital-ocean-config.nix")
    ];

    deployment.targetHost = "198.51.100.207";
    deployment.targetUser = "root";

    networking.hostName = name;

    system.stateVersion = "21.11"; # Do not change lightly!

    deployment.healthChecks = {
      http = [
        {
          scheme = "http";
          port = 80;
          path = "/";
          description = "check that nginx is running";
        }
      ];
    };

    networking.firewall.allowedTCPPorts = [ 80 ];

    services.nginx = {
      enable = true;
      virtualHosts.default = {
        default = true;
        locations."/".return = "200 \"Hello from Nixie!\"";
      };
    };
  };
}

Let's recap. Using this process, we:

• Created an image that allows us to bootstrap NixOS servers on DigitalOcean, complete with our SSH public key for easy management
• Created a NixOS configuration that we can push to remote machines via Morph
• Deployed an nginx server and added a health check to ensure it still works after changes to configuration
• Pinned this server's packages to a specific commit to ensure stability

The entire source code can be found in a GitHub repository.

What's next?

In the second part of the series, we will learn how to:

• Launch several DigitalOcean servers using Terraform
• Make Nix aware of Terraform state and the servers it manages
• Use that information in our Morph network definition and deploy a setup of several backend servers behind several load balancers.