iac/nixos

NixOS

The whole system of a server, as one flake. asxpio-services runs NixOS; asxpio-ops is still Debian and managed by ../ansible.

flake.nix                       inputs (nixpkgs 26.05, disko, sops-nix), hosts
modules/base.nix                users, SSH, fail2ban, journal cap
modules/hetzner.nix             disk layout and boot for a Hetzner Cloud VM
hosts/asxpio-services/          network, WireGuard, firewall, containers, CI runner
hosts/rehearsal.nix             overrides for a throwaway test server
secrets/*.yaml                  sops-encrypted secrets
migration/                      one-off scripts for the move from Debian

Containers run under Podman, one systemd unit each: systemctl status podman-minecraft, journalctl -u podman-sillytavern. docker is an alias for podman, and /run/docker.sock is Podman's Docker-compatible socket, which the CI runner uses to start jobs.

Changing something

Edit, push a branch (CI evaluates the system), merge to main (CI builds it on the server and switches to it). By hand, over WireGuard:

nixos-rebuild switch --flake .#asxpio-services --target-host root@10.20.0.20

A deploy restarts only the units whose definition changed. Changing a Minecraft container restarts that server; it gets 120 s to save first. A bad deploy is undone with nixos-rebuild switch --rollback on the host, or by picking the previous generation in the boot menu.

To update all packages: nix flake update, commit flake.lock.

Secrets

secrets/asxpio-services.yaml is encrypted with sops for two keys listed in .sops.yaml: the admin's age key and the server's SSH host key. The server decrypts at boot into /run/secrets.

sops secrets/asxpio-services.yaml      # edit; needs ~/.config/sops/age/keys.txt

The server's SSH host key is therefore part of its identity. Keep a copy outside the server (the system archive of a backup contains /etc/ssh), and hand it to a reinstall with --extra-files.

Installing or reinstalling a server

Never delete and recreate a production server: it is billed at an old price that a new server would not get. Reinstall in place.

1. Put the host key where the installer will copy it from:

   mkdir -p extra/etc/ssh        # then place ssh_host_ed25519_key{,.pub}, mode 600
   

2. Install. The running system is replaced; the disk is wiped.

   nix run github:nix-community/nixos-anywhere -- \
     --flake .#asxpio-services --extra-files extra \
     --ssh-option ProxyJump=root@10.20.0.10 root@10.10.0.20
   

   The installer has no WireGuard, so it is reached over the private network through asxpio-ops.

3. Restore data with migration/restore-services.sh. Containers that own data do not start until /opt/.restored exists, so nothing generates a fresh world where the restore is about to land.

If a server is ever lost entirely, terraform apply creates a bare one and the steps above install onto it.

Rehearsal

.#rehearsal is the same system for a throwaway server: plain DHCP, SSH from anywhere, no WireGuard, dummy secrets. Use it to try an install or a risky change before production.

Its secrets are encrypted for a host key that was thrown away with the last test server. For a new one, generate a key, put its age recipient in .sops.yaml and re-encrypt:

ssh-keygen -t ed25519 -N '' -f extra/etc/ssh/ssh_host_ed25519_key
nix run nixpkgs#ssh-to-age < extra/etc/ssh/ssh_host_ed25519_key.pub   # -> &rehearsal
sops updatekeys secrets/rehearsal.yaml