From 131124e0de45b731f89f0bcb72f3301c834f64fb Mon Sep 17 00:00:00 2001 From: Sergei Poljanski Date: Thu, 12 Nov 2026 17:13:02 +0400 Subject: [PATCH] nixos: runbook for the move, tested restore Write down the cutover for asxpio-services step by step, with the times measured in the rehearsal: backup 6 minutes, restore of 29 GB 3 minutes. The backup script now tolerates files changing under tar in live mode, which a running Minecraft world always does; in final mode, with the servers stopped, that remains an error. The restore script can leave the containers stopped for a restore test. Also checked on the test server before it was deleted: the production network layout (interfaces matched by MAC, static IPv6, the private network on enp7s0) comes up after a reboot with the same routes the Debian host has. Assisted-by: Claude:fable-5-2 --- .gitignore | 1 + nixos/README.md | 10 +++ nixos/migration/README.md | 94 +++++++++++++++++++++++++++++ nixos/migration/restore-services.sh | 6 ++ 4 files changed, 111 insertions(+) create mode 100644 nixos/migration/README.md diff --git a/.gitignore b/.gitignore index f45d67e..aed38c8 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ secrets/* services SECRETS.md +/rclone.conf.migration diff --git a/nixos/README.md b/nixos/README.md index 2e4848e..3e25dcb 100644 --- a/nixos/README.md +++ b/nixos/README.md @@ -83,3 +83,13 @@ and the steps above install onto it. `.#rehearsal` is the same system for a throwaway server: plain DHCP, SSH from anywhere, no WireGuard, dummy secrets. Use it to try an install or a risky change before production. + +Its secrets are encrypted for a host key that was thrown away with the +last test server. For a new one, generate a key, put its age recipient +in `.sops.yaml` and re-encrypt: + +```sh +ssh-keygen -t ed25519 -N '' -f extra/etc/ssh/ssh_host_ed25519_key +nix run nixpkgs#ssh-to-age < extra/etc/ssh/ssh_host_ed25519_key.pub # -> &rehearsal +sops updatekeys secrets/rehearsal.yaml +``` diff --git a/nixos/migration/README.md b/nixos/migration/README.md new file mode 100644 index 0000000..cd7d6dd --- /dev/null +++ b/nixos/migration/README.md @@ -0,0 +1,94 @@ +# Moving asxpio-services from Debian to NixOS + +One-off runbook. Delete this directory once the move is done. + +Downtime is about 20 to 30 minutes for Minecraft (both servers), +SillyTavern and CI. asxpio-ops and everything on it stays up. Measured +on 2026-11-12: backup 6 min, install about 6 min, restore 3 min. + +All commands run from a workstation on WireGuard, in `nixos/`. + +## Before + +1. Announce the window; make sure no CI job is running. +2. Take a Hetzner snapshot of the server, or check that last night's + backup exists. This is the rollback. +3. Stage what must survive the wipe (both are ignored by git): + + ```sh + mkdir -p extra/etc/ssh && chmod 700 extra + scp root@10.20.0.20:/etc/ssh/ssh_host_ed25519_key{,.pub} extra/etc/ssh/ + scp root@10.20.0.20:/root/.config/rclone/rclone.conf extra/rclone.conf + mv extra/rclone.conf ../rclone.conf.migration && chmod 600 ../rclone.conf.migration + ``` + + The host key keeps `known_hosts` valid and is what decrypts the + server's secrets. + +## Cutover + +4. Final backup. Stops the containers, uploads, verifies checksums, and + prints `DEST=...`; keep that path. + + ```sh + ssh root@10.20.0.20 'bash -s final' < migration/backup-services.sh + ``` + + Up to here nothing is lost: `docker start` brings everything back. + +5. Install. **This wipes the disk.** + + ```sh + nix run github:nix-community/nixos-anywhere -- \ + --flake .#asxpio-services --extra-files extra \ + --ssh-option ProxyJump=root@10.20.0.10 root@10.10.0.20 + ``` + + The installer boots in RAM first and only then formats. If it cannot + be reached at that point, rebooting the server from the Hetzner + console returns to the untouched Debian. + +6. Check the new system over WireGuard (same host key, no warning): + + ```sh + ssh root@10.20.0.20 'nixos-version; systemctl is-system-running; wg show' + ``` + + If WireGuard is down, the server is still reachable through + asxpio-ops: `ssh -J root@10.20.0.10 root@10.10.0.20`. + +7. Restore. The object-storage keys exist on the server only for this + command. + + ```sh + ssh root@10.20.0.20 'umask 077; cat > /run/rclone.conf' < ../rclone.conf.migration + ssh root@10.20.0.20 'RCLONE_CONFIG=/run/rclone.conf bash -s ' < migration/restore-services.sh + ssh root@10.20.0.20 'rm -f /run/rclone.conf' + ``` + +8. Verify: + + ```sh + ssh root@10.20.0.20 'podman ps; systemctl --failed' + ``` + + - Both Minecraft logs reach `Done`; join each server once. + - Forgejo → Site administration → Runners shows the runner idle. + - SillyTavern answers on `http://10.20.0.20:8000`. + - Push an empty commit to an app repo and watch one full CI run. Jobs + now run on Podman; this is the one thing the rehearsal could only + test piece by piece. + +## After + +9. Merge the `nixos` branch to `main`. The Ansible pipeline then covers + asxpio-ops only, and the NixOS pipeline finds nothing to change. +10. Remove `extra/` and `../rclone.conf.migration`, delete the snapshot + after a few days, and delete this directory. + +## Rollback + +- Before step 5: start the containers again. +- After step 5: restore the Hetzner snapshot or backup from the console. + The server comes back as the Debian system it was, with the data as of + the snapshot. diff --git a/nixos/migration/restore-services.sh b/nixos/migration/restore-services.sh index 4827cd6..e54f745 100755 --- a/nixos/migration/restore-services.sh +++ b/nixos/migration/restore-services.sh @@ -33,6 +33,12 @@ for name in "${names[@]}"; do done rm -f "$sums" +# RESTORE_NO_START=1 leaves everything stopped (restore tests) +if [ -n "${RESTORE_NO_START:-}" ]; then + echo "== data restored; containers left stopped" + exit 0 +fi + touch /opt/.restored echo "== data restored; starting containers" systemctl start --no-block \