asxpi/iac

ci: evaluate NixOS in the job, build on the server

The first run failed with "home directory /homeless-shelter exists":
job containers are unprivileged, Nix cannot sandbox builds in them, and
unsandboxed builds trip over each other's leftovers.

Evaluate the system in the job instead, which still catches broken
options before main, and let nixos-rebuild build on the target host,
where the Nix daemon has a proper sandbox.

Assisted-by: Claude:fable-5-2

2026-11-12 23:53:07 +04:00

3.1 KiB

Raw Blame History

NixOS

The whole system of a server, as one flake. asxpio-services runs NixOS; asxpio-ops is still Debian and managed by ../ansible.

flake.nix                       inputs (nixpkgs 26.05, disko, sops-nix), hosts
modules/base.nix                users, SSH, fail2ban, journal cap
modules/hetzner.nix             disk layout and boot for a Hetzner Cloud VM
hosts/asxpio-services/          network, WireGuard, firewall, containers, CI runner
hosts/rehearsal.nix             overrides for a throwaway test server
secrets/*.yaml                  sops-encrypted secrets
migration/                      one-off scripts for the move from Debian

Containers run under Podman, one systemd unit each: systemctl status podman-minecraft, journalctl -u podman-sillytavern. docker is an alias for podman, and /run/docker.sock is Podman's Docker-compatible socket, which the CI runner uses to start jobs.

Changing something

Edit, push a branch (CI evaluates the system), merge to main (CI builds it on the server and switches to it). By hand, over WireGuard:

nixos-rebuild switch --flake .#asxpio-services --target-host root@10.20.0.20

A deploy restarts only the units whose definition changed. Changing a Minecraft container restarts that server; it gets 120 s to save first. A bad deploy is undone with nixos-rebuild switch --rollback on the host, or by picking the previous generation in the boot menu.

To update all packages: nix flake update, commit flake.lock.

Secrets

secrets/asxpio-services.yaml is encrypted with sops for two keys listed in .sops.yaml: the admin's age key and the server's SSH host key. The server decrypts at boot into /run/secrets.

sops secrets/asxpio-services.yaml      # edit; needs ~/.config/sops/age/keys.txt

The server's SSH host key is therefore part of its identity. Keep a copy outside the server (the system archive of a backup contains /etc/ssh), and hand it to a reinstall with --extra-files.

Installing or reinstalling a server

Never delete and recreate a production server: it is billed at an old price that a new server would not get. Reinstall in place.

Put the host key where the installer will copy it from:

mkdir -p extra/etc/ssh        # then place ssh_host_ed25519_key{,.pub}, mode 600

Install. The running system is replaced; the disk is wiped.

nix run github:nix-community/nixos-anywhere -- \
  --flake .#asxpio-services --extra-files extra \
  --ssh-option ProxyJump=root@10.20.0.10 root@10.10.0.20

The installer has no WireGuard, so it is reached over the private network through asxpio-ops.

Restore data with migration/restore-services.sh. Containers that own data do not start until /opt/.restored exists, so nothing generates a fresh world where the restore is about to land.

If a server is ever lost entirely, terraform apply creates a bare one and the steps above install onto it.

Rehearsal

.#rehearsal is the same system for a throwaway server: plain DHCP, SSH from anywhere, no WireGuard, dummy secrets. Use it to try an install or a risky change before production.

3.1 KiB Raw Blame History