nixos: runbook for the move, tested restore

Write down the cutover for asxpio-services step by step, with the
times measured in the rehearsal: backup 6 minutes, restore of 29 GB
3 minutes.

The backup script now tolerates files changing under tar in live mode,
which a running Minecraft world always does; in final mode, with the
servers stopped, that remains an error. The restore script can leave
the containers stopped for a restore test.

Also checked on the test server before it was deleted: the production
network layout (interfaces matched by MAC, static IPv6, the private
network on enp7s0) comes up after a reboot with the same routes the
Debian host has.

Assisted-by: Claude:fable-5-2

.gitignore

@@ -1,3 +1,4 @@
 secrets/*
 services
 SECRETS.md
+/rclone.conf.migration

nixos/README.md

@@ -83,3 +83,13 @@ and the steps above install onto it.
 `.#rehearsal` is the same system for a throwaway server: plain DHCP, SSH
 from anywhere, no WireGuard, dummy secrets. Use it to try an install or
 a risky change before production.
+
+Its secrets are encrypted for a host key that was thrown away with the
+last test server. For a new one, generate a key, put its age recipient
+in `.sops.yaml` and re-encrypt:
+
+```sh
+ssh-keygen -t ed25519 -N '' -f extra/etc/ssh/ssh_host_ed25519_key
+nix run nixpkgs#ssh-to-age < extra/etc/ssh/ssh_host_ed25519_key.pub   # -> &rehearsal
+sops updatekeys secrets/rehearsal.yaml
+```

nixos/migration/README.md

@@ -0,0 +1,94 @@
+# Moving asxpio-services from Debian to NixOS
+
+One-off runbook. Delete this directory once the move is done.
+
+Downtime is about 20 to 30 minutes for Minecraft (both servers),
+SillyTavern and CI. asxpio-ops and everything on it stays up. Measured
+on 2026-11-12: backup 6 min, install about 6 min, restore 3 min.
+
+All commands run from a workstation on WireGuard, in `nixos/`.
+
+## Before
+
+1. Announce the window; make sure no CI job is running.
+2. Take a Hetzner snapshot of the server, or check that last night's
+   backup exists. This is the rollback.
+3. Stage what must survive the wipe (both are ignored by git):
+
+   ```sh
+   mkdir -p extra/etc/ssh && chmod 700 extra
+   scp root@10.20.0.20:/etc/ssh/ssh_host_ed25519_key{,.pub} extra/etc/ssh/
+   scp root@10.20.0.20:/root/.config/rclone/rclone.conf extra/rclone.conf
+   mv extra/rclone.conf ../rclone.conf.migration && chmod 600 ../rclone.conf.migration
+   ```
+
+   The host key keeps `known_hosts` valid and is what decrypts the
+   server's secrets.
+
+## Cutover
+
+4. Final backup. Stops the containers, uploads, verifies checksums, and
+   prints `DEST=...`; keep that path.
+
+   ```sh
+   ssh root@10.20.0.20 'bash -s final' < migration/backup-services.sh
+   ```
+
+   Up to here nothing is lost: `docker start` brings everything back.
+
+5. Install. **This wipes the disk.**
+
+   ```sh
+   nix run github:nix-community/nixos-anywhere -- \
+     --flake .#asxpio-services --extra-files extra \
+     --ssh-option ProxyJump=root@10.20.0.10 root@10.10.0.20
+   ```
+
+   The installer boots in RAM first and only then formats. If it cannot
+   be reached at that point, rebooting the server from the Hetzner
+   console returns to the untouched Debian.
+
+6. Check the new system over WireGuard (same host key, no warning):
+
+   ```sh
+   ssh root@10.20.0.20 'nixos-version; systemctl is-system-running; wg show'
+   ```
+
+   If WireGuard is down, the server is still reachable through
+   asxpio-ops: `ssh -J root@10.20.0.10 root@10.10.0.20`.
+
+7. Restore. The object-storage keys exist on the server only for this
+   command.
+
+   ```sh
+   ssh root@10.20.0.20 'umask 077; cat > /run/rclone.conf' < ../rclone.conf.migration
+   ssh root@10.20.0.20 'RCLONE_CONFIG=/run/rclone.conf bash -s <DEST>' < migration/restore-services.sh
+   ssh root@10.20.0.20 'rm -f /run/rclone.conf'
+   ```
+
+8. Verify:
+
+   ```sh
+   ssh root@10.20.0.20 'podman ps; systemctl --failed'
+   ```
+
+   - Both Minecraft logs reach `Done`; join each server once.
+   - Forgejo → Site administration → Runners shows the runner idle.
+   - SillyTavern answers on `http://10.20.0.20:8000`.
+   - Push an empty commit to an app repo and watch one full CI run. Jobs
+     now run on Podman; this is the one thing the rehearsal could only
+     test piece by piece.
+
+## After
+
+9. Merge the `nixos` branch to `main`. The Ansible pipeline then covers
+   asxpio-ops only, and the NixOS pipeline finds nothing to change.
+10. Remove `extra/` and `../rclone.conf.migration`, delete the snapshot
+    after a few days, and delete this directory.
+
+## Rollback
+
+- Before step 5: start the containers again.
+- After step 5: restore the Hetzner snapshot or backup from the console.
+  The server comes back as the Debian system it was, with the data as of
+  the snapshot.

nixos/migration/restore-services.sh

@@ -33,6 +33,12 @@ for name in "${names[@]}"; do
 done
 rm -f "$sums"
 
+# RESTORE_NO_START=1 leaves everything stopped (restore tests)
+if [ -n "${RESTORE_NO_START:-}" ]; then
+  echo "== data restored; containers left stopped"
+  exit 0
+fi
+
 touch /opt/.restored
 echo "== data restored; starting containers"
 systemctl start --no-block \