In this post I’ll describe how to set up a sparse VM image with full disk encryption and NixOS on ZFS, which can be uploaded to a VPS provider and then unlocked on boot using ssh.

First we need to create a virtual machine image file. Initially I tried using qemu-img, but somehow the image file was missing some information and the VM would not recognize a disk. Instead I went with the easy way and used virt-manager to create a new VM with the correct image size. virt-manager seems to create images allocated with the full size:

$ ls -lh /var/lib/libvirt/images
-rw------- 1 root root  161G Aug 24 08:59 nixosVM160GB.qcow2
-rw------- 1 root root   41G Aug 24 00:22 nixosVM40GB.qcow2

Uploading 200GB of mostly empty VM images can take quite a while, lets make them smaller using the wonderful libguestfs-tools.

$ nix-shell -p libguestfs-with-appliance --run "sudo virt-sparsify /var/lib/libvirt/images/nixosVM40GB.qcow2 \
  /var/lib/libvirt/images/nixosVM40GB-sparse.qcow2"
[   0.1] Create overlay file in /tmp to protect source disk
[   0.1] Examine source disk
[   1.8] Copy to destination and make sparse
[  46.6] Sparsify operation completed with no errors.
.virt-sparsify-wrapped: Before deleting the old disk, carefully check that
the target disk boots and works correctly.
$ nix-shell -p libguestfs-with-appliance --run "sudo virt-sparsify /var/lib/libvirt/images/nixosVM160GB.qcow2 \
  /var/lib/libvirt/images/nixosVM160GB-sparse.qcow2"
[   0.0] Create overlay file in /tmp to protect source disk
[   0.1] Examine source disk
[   1.6] Copy to destination and make sparse
[ 172.0] Sparsify operation completed with no errors.
.virt-sparsify-wrapped: Before deleting the old disk, carefully check that
the target disk boots and works correctly.

Now the files are considerably smaller:

$ ls -lh /var/lib/libvirt/images
-rw------- 1 root root  161G Aug 24 08:59 nixosVM160GB.qcow2
-rw-r--r-- 1 root root  195K Aug 28 16:13 nixosVM160GB-sparse.qcow2
-rw------- 1 root root   41G Aug 24 00:22 nixosVM40GB.qcow2
-rw-r--r-- 1 root root  193K Aug 28 16:11 nixosVM40GB-sparse.qcow2

Next install the VMs. In order to do so, create a new VM in virt-manager using the sparse image, add a CDROM type storage device with the NixOS install iso and set up the virtual machine to boot from this device.

The following script will create a virtual machine with the following partition layout, (I’m using ZFS inside lvm in this setup):

sda
├─sda1            GRUB
├─sda2            BOOT
└─sda3            LINUX (LUKS CONTAINER)
  └─cryptroot     LUKS MAPPER
    └─lvmvg-swap  SWAP
    └─lvmvg-root  ZFS
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
#!/usr/bin/env bash
set -e

pprint () {
    local cyan="\e[96m"
    local default="\e[39m"
    # ISO8601 timestamp + ms
    local timestamp
    timestamp=$(date +%FT%T.%3NZ)
    echo -e "${cyan}${timestamp} $1${default}" 1>&2
}

echo "These are the disks available:"
ls -l /dev/disk/by-id/ | awk '{print $11, $10, $9}' | tr -d './' | column -t
echo

# Set DISK
select ENTRY in $(ls /dev/disk/by-id/);
do
    echo $ENTRY
    DISK="/dev/disk/by-id/$ENTRY"
    echo "Installing system on $ENTRY."
    break
done


echo -n "Enter Swap size (e.g. 10G, 5M)"
echo
read SWAP_SIZE
if [[ $SWAP_SIZE =~ ^[0-9]*(G|M)$ ]]
then
  echo "Swap size set to $SWAP_SIZE"
else
  echo "Please enter correct size"
  break
fi

read -p "> Do you want to wipe all data on $ENTRY ?" -n 1 -r
echo # move to a new line
if [[ "$REPLY" =~ ^[Yy]$ ]]
then
    # Clear disk
    wipefs -af "$DISK"
    sgdisk -Zo "$DISK"
fi

pprint "Creating grub partition"
sgdisk -n 0:0:+1000K    -t 1:EF02 "$DISK"
GRUB="$DISK-part1"

pprint "Creating boot partition"
sgdisk -n 0:0:+512M -t 0:EF00 "$DISK"
BOOT="$DISK-part2"

pprint "Creating Linux partition"
sgdisk -n 0:0:0 -t 3:8300 "$DISK"
LINUX="$DISK-part3"

# Inform kernel
partprobe "$DISK"
sleep 1

pprint "Format BOOT partition $BOOT"
mkfs.vfat "$BOOT"

pprint "Creating LUKS container on $LINUX"
cryptsetup --type luks2 luksFormat "$LINUX"

LUKS_DEVICE_NAME=cryptroot
cryptsetup luksOpen "$LINUX" "$LUKS_DEVICE_NAME"

LUKS_DISK="/dev/mapper/$LUKS_DEVICE_NAME"

# Create LVM physical volume
pvcreate $LUKS_DISK

LVM_VOLUME_GROUP=vg0
vgcreate "$LVM_VOLUME_GROUP" "$LUKS_DISK"

lvcreate --name swap --size $SWAP_SIZE "$LVM_VOLUME_GROUP"
SWAP="/dev/$LVM_VOLUME_GROUP/swap"

pprint "Enable SWAP on $SWAP"
mkswap $SWAP
swapon $SWAP

# ZFS partition
lvcreate --name root --extents 100%FREE "$LVM_VOLUME_GROUP"
ZFS="/dev/$LVM_VOLUME_GROUP/root"

pprint "Create ZFS pool on $ZFS"
# -f force
# -m mountpoint
zpool create -f -m none -R /mnt rpool "$ZFS"

pprint "Create ZFS datasets"

zfs create -o mountpoint=legacy rpool/root
zfs create -o mountpoint=legacy rpool/root/nix
zfs create -o mountpoint=legacy rpool/home

zfs set com.sun:auto-snapshot=true rpool/home

zfs snapshot rpool/root@blank

pprint "Mount ZFS datasets"
mount -t zfs rpool/root /mnt

mkdir /mnt/nix
mount -t zfs rpool/root/nix /mnt/nix

mkdir /mnt/home
mount -t zfs rpool/home /mnt/home

mkdir /mnt/boot
mount "$BOOT" /mnt/boot

pprint "Generate NixOS configuration"
nixos-generate-config --root /mnt

# Add LUKS and ZFS configuration
HOSTID=$(head -c8 /etc/machine-id)
LINUX_DISK_UUID=$(blkid --match-tag UUID --output value "$LINUX")

HARDWARE_CONFIG=$(mktemp)
cat <<CONFIG > "$HARDWARE_CONFIG"
  networking.hostId = "$HOSTID";
  boot.initrd.luks.devices."$LUKS_DEVICE_NAME".device = "/dev/disk/by-uuid/$LINUX_DISK_UUID";
  boot.zfs.devNodes = "$ZFS";
CONFIG

pprint "Append configuration to hardware-configuration.nix"
sed -i "\$e cat $HARDWARE_CONFIG" /mnt/etc/nixos/hardware-configuration.nix

After running the script, create the dropbear ssh keys:

$ nix-shell -p dropbear --command "dropbearkey -t ecdsa -f /tmp/initrd-ssh-key"

Copy the key into / as well as /mnt (for some reason the installer seems to fail, when the keys aren’t found in /):

$ sudo mkdir -p /var/dropbear /mnt/var/dropbear
$ sudo cp /tmp/initrd-ssh-key /var/dropbear/
$ sudo cp /tmp/initrd-ssh-key /mnt/var/dropbear/

Add in the boot config:

  boot.loader.grub.device = "/dev/sda";

  boot.initrd.network.enable = true;
  boot.initrd.network.ssh = {
    enable = true;
    port = 2222;
    authorizedKeys = [ "ssh-rsa ..." ];   <-------- this is your list of ssh keys, which are allowed to log in
    hostECDSAKey = /var/dropbear/initrd-ssh-key;
  };
  boot.initrd.network.postCommands = ''
    echo "cryptsetup-askpass" >> /root/.profile
  '';

Finally the VM has to be able to connect to the network, so the initramdisk needs to have the correct kernel modules available. It is possible to find out which kernel modules are necessary for a specific provider by simply launching a VM and looking at lsmod or clicking through the VM configuration and looking for a section on virtualisation drivers.

  boot.initrd.availableKernelModules = [ "e1000e" "virtio_pci" "e1000" ];

Finally install nixos using nixos-install and reboot.

Using different entries in ~/.ssh/config can make rebooting and decrypting the luks devices easier:

Host vm
  Hostname 1.2.3.4
  User user

Host unlock.vm
  Hostname 1.2.3.4
  User root
  Port 2222

This way there won’t be any key conflicts and you can conveniently unlock the VMs using:

$ ssh unlock.vm # alternatively use: ssh -p 2222 root@192.168.122.x
The authenticity of host '[192.168.122.x]:2222 ([192.168.122.x]:2222)' can't be established.
ECDSA key fingerprint is SHA256:...
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[192.168.122.x]:2222' (ECDSA) to the list of known hosts.

Passphrase for /dev/disk/by-uuid/...: *********************************

Waiting 10 seconds for LUKS to request a passphrase.......Connection to 192.168.122.x closed by remote host.
Connection to 192.168.122.x closed.

After installation these are the file sizes:

$ sudo ls -lh /var/lib/libvirt/images/
-rw-r--r-- 1 root root  195K Aug 28 16:54 160GB-sparse-disk-template.qcow2
-rw-r--r-- 1 root root  193K Jul 31 18:36 20GB-sparse-disk-template.qcow2
-rw-r--r-- 1 root root  193K Aug 28 16:54 40GB-sparse-disk-template.qcow2
-rw------- 1 root root  161G Aug 24 08:59 nixosVM160GB.qcow2
-rw-r--r-- 1 root root  1.9G Aug 29 14:11 nixosVM160GB-sparse.qcow2
-rw------- 1 root root   41G Aug 24 00:22 nixosVM40GB.qcow2
-rw-r--r-- 1 root root  2.0G Aug 29 14:01 nixosVM40GB-sparse.qcow2