In this post I’ll describe how to set up a sparse VM image with full disk encryption and NixOS on ZFS, which can be uploaded to a VPS provider and then unlocked on boot using ssh.
First we need to create a virtual machine image file. Initially I tried using
qemu-img
, but somehow the image file was missing some information and the VM
would not recognize a disk. Instead I went with the easy way and used
virt-manager
to create a new VM with the correct image size. virt-manager
seems to create images allocated with the full size:
$ ls -lh /var/lib/libvirt/images
-rw------- 1 root root 161G Aug 24 08:59 nixosVM160GB.qcow2
-rw------- 1 root root 41G Aug 24 00:22 nixosVM40GB.qcow2
Uploading 200GB of mostly empty VM images can take quite a while, lets make them
smaller using the wonderful libguestfs-tools
.
$ nix-shell -p libguestfs-with-appliance --run "sudo virt-sparsify /var/lib/libvirt/images/nixosVM40GB.qcow2 \
/var/lib/libvirt/images/nixosVM40GB-sparse.qcow2"
[ 0.1] Create overlay file in /tmp to protect source disk
[ 0.1] Examine source disk
[ 1.8] Copy to destination and make sparse
[ 46.6] Sparsify operation completed with no errors.
.virt-sparsify-wrapped: Before deleting the old disk, carefully check that
the target disk boots and works correctly.
$ nix-shell -p libguestfs-with-appliance --run "sudo virt-sparsify /var/lib/libvirt/images/nixosVM160GB.qcow2 \
/var/lib/libvirt/images/nixosVM160GB-sparse.qcow2"
[ 0.0] Create overlay file in /tmp to protect source disk
[ 0.1] Examine source disk
[ 1.6] Copy to destination and make sparse
[ 172.0] Sparsify operation completed with no errors.
.virt-sparsify-wrapped: Before deleting the old disk, carefully check that
the target disk boots and works correctly.
Now the files are considerably smaller:
$ ls -lh /var/lib/libvirt/images
-rw------- 1 root root 161G Aug 24 08:59 nixosVM160GB.qcow2
-rw-r--r-- 1 root root 195K Aug 28 16:13 nixosVM160GB-sparse.qcow2
-rw------- 1 root root 41G Aug 24 00:22 nixosVM40GB.qcow2
-rw-r--r-- 1 root root 193K Aug 28 16:11 nixosVM40GB-sparse.qcow2
Next install the VMs. In order to do so, create a new VM in virt-manager using the sparse image, add a CDROM type storage device with the NixOS install iso and set up the virtual machine to boot from this device.
The following script will create a virtual machine with the following partition layout, (I’m using ZFS inside lvm in this setup):
sda
├─sda1 GRUB
├─sda2 BOOT
└─sda3 LINUX (LUKS CONTAINER)
└─cryptroot LUKS MAPPER
└─lvmvg-swap SWAP
└─lvmvg-root ZFS
|
|
After running the script, create the dropbear ssh keys:
$ nix-shell -p dropbear --command "dropbearkey -t ecdsa -f /tmp/initrd-ssh-key"
Copy the key into /
as well as /mnt
(for some reason the installer seems to
fail, when the keys aren’t found in /
):
$ sudo mkdir -p /var/dropbear /mnt/var/dropbear
$ sudo cp /tmp/initrd-ssh-key /var/dropbear/
$ sudo cp /tmp/initrd-ssh-key /mnt/var/dropbear/
Add in the boot config:
boot.loader.grub.device = "/dev/sda";
boot.initrd.network.enable = true;
boot.initrd.network.ssh = {
enable = true;
port = 2222;
authorizedKeys = [ "ssh-rsa ..." ]; <-------- this is your list of ssh keys, which are allowed to log in
hostECDSAKey = /var/dropbear/initrd-ssh-key;
};
boot.initrd.network.postCommands = ''
echo "cryptsetup-askpass" >> /root/.profile
'';
Finally the VM has to be able to connect to the network, so the initramdisk
needs to have the correct kernel modules available. It is possible to find out
which kernel modules are necessary for a specific provider by simply launching a
VM and looking at lsmod
or clicking through the VM configuration and looking
for a section on virtualisation drivers.
boot.initrd.availableKernelModules = [ "e1000e" "virtio_pci" "e1000" ];
Finally install nixos using nixos-install
and reboot.
Using different entries in ~/.ssh/config
can make rebooting and decrypting the luks devices easier:
Host vm
Hostname 1.2.3.4
User user
Host unlock.vm
Hostname 1.2.3.4
User root
Port 2222
This way there won’t be any key conflicts and you can conveniently unlock the VMs using:
$ ssh unlock.vm # alternatively use: ssh -p 2222 root@192.168.122.x
The authenticity of host '[192.168.122.x]:2222 ([192.168.122.x]:2222)' can't be established.
ECDSA key fingerprint is SHA256:...
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[192.168.122.x]:2222' (ECDSA) to the list of known hosts.
Passphrase for /dev/disk/by-uuid/...: *********************************
Waiting 10 seconds for LUKS to request a passphrase.......Connection to 192.168.122.x closed by remote host.
Connection to 192.168.122.x closed.
After installation these are the file sizes:
$ sudo ls -lh /var/lib/libvirt/images/
-rw-r--r-- 1 root root 195K Aug 28 16:54 160GB-sparse-disk-template.qcow2
-rw-r--r-- 1 root root 193K Jul 31 18:36 20GB-sparse-disk-template.qcow2
-rw-r--r-- 1 root root 193K Aug 28 16:54 40GB-sparse-disk-template.qcow2
-rw------- 1 root root 161G Aug 24 08:59 nixosVM160GB.qcow2
-rw-r--r-- 1 root root 1.9G Aug 29 14:11 nixosVM160GB-sparse.qcow2
-rw------- 1 root root 41G Aug 24 00:22 nixosVM40GB.qcow2
-rw-r--r-- 1 root root 2.0G Aug 29 14:01 nixosVM40GB-sparse.qcow2