iac/nixos/migration

Moving asxpio-services from Debian to NixOS

One-off runbook. Delete this directory once the move is done.

Downtime is about 20 to 30 minutes for Minecraft (both servers), SillyTavern and CI. asxpio-ops and everything on it stays up. Measured on 2026-11-12: backup 6 min, install about 6 min, restore 3 min.

All commands run from a workstation on WireGuard, in nixos/.

Before

1. Announce the window; make sure no CI job is running.

2. Take a Hetzner snapshot of the server, or check that last night's backup exists. This is the rollback.

3. Stage what must survive the wipe (both are ignored by git):

   mkdir -p extra/etc/ssh && chmod 700 extra
   scp root@10.20.0.20:/etc/ssh/ssh_host_ed25519_key{,.pub} extra/etc/ssh/
   scp root@10.20.0.20:/root/.config/rclone/rclone.conf extra/rclone.conf
   mv extra/rclone.conf ../rclone.conf.migration && chmod 600 ../rclone.conf.migration
   

   The host key keeps known_hosts valid and is what decrypts the server's secrets.

Cutover

4. Final backup. Stops the containers, uploads, verifies checksums, and prints DEST=...; keep that path.

   ssh root@10.20.0.20 'bash -s final' < migration/backup-services.sh
   

   Up to here nothing is lost: docker start brings everything back.

5. Install. This wipes the disk.

   nix run github:nix-community/nixos-anywhere -- \
     --flake .#asxpio-services --extra-files extra \
     --ssh-option ProxyJump=root@10.20.0.10 root@10.10.0.20
   

   The installer boots in RAM first and only then formats. If it cannot be reached at that point, rebooting the server from the Hetzner console returns to the untouched Debian.

6. Check the new system over WireGuard (same host key, no warning):

   ssh root@10.20.0.20 'nixos-version; systemctl is-system-running; wg show'
   

   If WireGuard is down, the server is still reachable through asxpio-ops: ssh -J root@10.20.0.10 root@10.10.0.20.

7. Restore. The object-storage keys exist on the server only for this command.

   ssh root@10.20.0.20 'umask 077; cat > /run/rclone.conf' < ../rclone.conf.migration
   ssh root@10.20.0.20 'RCLONE_CONFIG=/run/rclone.conf bash -s <DEST>' < migration/restore-services.sh
   ssh root@10.20.0.20 'rm -f /run/rclone.conf'
   

8. Verify:

   ssh root@10.20.0.20 'podman ps; systemctl --failed'
   

   • Both Minecraft logs reach Done; join each server once.
   • Forgejo → Site administration → Runners shows the runner idle.
   • SillyTavern answers on http://10.20.0.20:8000.
   • Push an empty commit to an app repo and watch one full CI run. Jobs now run on Podman; this is the one thing the rehearsal could only test piece by piece.

After

9. Merge the nixos branch to main. The Ansible pipeline then covers asxpio-ops only, and the NixOS pipeline finds nothing to change.
10. Remove extra/ and ../rclone.conf.migration, delete the snapshot after a few days, and delete this directory.

Rollback

• Before step 5: start the containers again.
• After step 5: restore the Hetzner snapshot or backup from the console. The server comes back as the Debian system it was, with the data as of the snapshot.