diff --git a/.gitignore b/.gitignore index f45d67e..aed38c8 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ secrets/* services SECRETS.md +/rclone.conf.migration diff --git a/nixos/README.md b/nixos/README.md index 2e4848e..3e25dcb 100644 --- a/nixos/README.md +++ b/nixos/README.md @@ -83,3 +83,13 @@ and the steps above install onto it. `.#rehearsal` is the same system for a throwaway server: plain DHCP, SSH from anywhere, no WireGuard, dummy secrets. Use it to try an install or a risky change before production. + +Its secrets are encrypted for a host key that was thrown away with the +last test server. For a new one, generate a key, put its age recipient +in `.sops.yaml` and re-encrypt: + +```sh +ssh-keygen -t ed25519 -N '' -f extra/etc/ssh/ssh_host_ed25519_key +nix run nixpkgs#ssh-to-age < extra/etc/ssh/ssh_host_ed25519_key.pub # -> &rehearsal +sops updatekeys secrets/rehearsal.yaml +``` diff --git a/nixos/migration/README.md b/nixos/migration/README.md new file mode 100644 index 0000000..cd7d6dd --- /dev/null +++ b/nixos/migration/README.md @@ -0,0 +1,94 @@ +# Moving asxpio-services from Debian to NixOS + +One-off runbook. Delete this directory once the move is done. + +Downtime is about 20 to 30 minutes for Minecraft (both servers), +SillyTavern and CI. asxpio-ops and everything on it stays up. Measured +on 2026-11-12: backup 6 min, install about 6 min, restore 3 min. + +All commands run from a workstation on WireGuard, in `nixos/`. + +## Before + +1. Announce the window; make sure no CI job is running. +2. Take a Hetzner snapshot of the server, or check that last night's + backup exists. This is the rollback. +3. Stage what must survive the wipe (both are ignored by git): + + ```sh + mkdir -p extra/etc/ssh && chmod 700 extra + scp root@10.20.0.20:/etc/ssh/ssh_host_ed25519_key{,.pub} extra/etc/ssh/ + scp root@10.20.0.20:/root/.config/rclone/rclone.conf extra/rclone.conf + mv extra/rclone.conf ../rclone.conf.migration && chmod 600 ../rclone.conf.migration + ``` + + The host key keeps `known_hosts` valid and is what decrypts the + server's secrets. + +## Cutover + +4. Final backup. Stops the containers, uploads, verifies checksums, and + prints `DEST=...`; keep that path. + + ```sh + ssh root@10.20.0.20 'bash -s final' < migration/backup-services.sh + ``` + + Up to here nothing is lost: `docker start` brings everything back. + +5. Install. **This wipes the disk.** + + ```sh + nix run github:nix-community/nixos-anywhere -- \ + --flake .#asxpio-services --extra-files extra \ + --ssh-option ProxyJump=root@10.20.0.10 root@10.10.0.20 + ``` + + The installer boots in RAM first and only then formats. If it cannot + be reached at that point, rebooting the server from the Hetzner + console returns to the untouched Debian. + +6. Check the new system over WireGuard (same host key, no warning): + + ```sh + ssh root@10.20.0.20 'nixos-version; systemctl is-system-running; wg show' + ``` + + If WireGuard is down, the server is still reachable through + asxpio-ops: `ssh -J root@10.20.0.10 root@10.10.0.20`. + +7. Restore. The object-storage keys exist on the server only for this + command. + + ```sh + ssh root@10.20.0.20 'umask 077; cat > /run/rclone.conf' < ../rclone.conf.migration + ssh root@10.20.0.20 'RCLONE_CONFIG=/run/rclone.conf bash -s ' < migration/restore-services.sh + ssh root@10.20.0.20 'rm -f /run/rclone.conf' + ``` + +8. Verify: + + ```sh + ssh root@10.20.0.20 'podman ps; systemctl --failed' + ``` + + - Both Minecraft logs reach `Done`; join each server once. + - Forgejo → Site administration → Runners shows the runner idle. + - SillyTavern answers on `http://10.20.0.20:8000`. + - Push an empty commit to an app repo and watch one full CI run. Jobs + now run on Podman; this is the one thing the rehearsal could only + test piece by piece. + +## After + +9. Merge the `nixos` branch to `main`. The Ansible pipeline then covers + asxpio-ops only, and the NixOS pipeline finds nothing to change. +10. Remove `extra/` and `../rclone.conf.migration`, delete the snapshot + after a few days, and delete this directory. + +## Rollback + +- Before step 5: start the containers again. +- After step 5: restore the Hetzner snapshot or backup from the console. + The server comes back as the Debian system it was, with the data as of + the snapshot. diff --git a/nixos/migration/restore-services.sh b/nixos/migration/restore-services.sh index 4827cd6..e54f745 100755 --- a/nixos/migration/restore-services.sh +++ b/nixos/migration/restore-services.sh @@ -33,6 +33,12 @@ for name in "${names[@]}"; do done rm -f "$sums" +# RESTORE_NO_START=1 leaves everything stopped (restore tests) +if [ -n "${RESTORE_NO_START:-}" ]; then + echo "== data restored; containers left stopped" + exit 0 +fi + touch /opt/.restored echo "== data restored; starting containers" systemctl start --no-block \