Sergei Poljanski
7489f6f485
Write down the cutover for asxpio-services step by step, with the times measured in the rehearsal: backup 6 minutes, restore of 29 GB 3 minutes. The backup script now tolerates files changing under tar in live mode, which a running Minecraft world always does; in final mode, with the servers stopped, that remains an error. The restore script can leave the containers stopped for a restore test. Also checked on the test server before it was deleted: the production network layout (interfaces matched by MAC, static IPv6, the private network on enp7s0) comes up after a reboot with the same routes the Debian host has. Assisted-by: Claude:fable-5-2
94 lines
3.2 KiB
Markdown
94 lines
3.2 KiB
Markdown
# Moving asxpio-services from Debian to NixOS
|
|
|
|
One-off runbook. Delete this directory once the move is done.
|
|
|
|
Downtime is about 20 to 30 minutes for Minecraft (both servers),
|
|
SillyTavern and CI. asxpio-ops and everything on it stays up. Measured
|
|
on 2026-11-12: backup 6 min, install about 6 min, restore 3 min.
|
|
|
|
All commands run from a workstation on WireGuard, in `nixos/`.
|
|
|
|
## Before
|
|
|
|
1. Announce the window; make sure no CI job is running.
|
|
2. Take a Hetzner snapshot of the server, or check that last night's
|
|
backup exists. This is the rollback.
|
|
3. Stage what must survive the wipe (both are ignored by git):
|
|
|
|
```sh
|
|
mkdir -p extra/etc/ssh && chmod 700 extra
|
|
scp root@10.20.0.20:/etc/ssh/ssh_host_ed25519_key{,.pub} extra/etc/ssh/
|
|
scp root@10.20.0.20:/root/.config/rclone/rclone.conf extra/rclone.conf
|
|
mv extra/rclone.conf ../rclone.conf.migration && chmod 600 ../rclone.conf.migration
|
|
```
|
|
|
|
The host key keeps `known_hosts` valid and is what decrypts the
|
|
server's secrets.
|
|
|
|
## Cutover
|
|
|
|
4. Final backup. Stops the containers, uploads, verifies checksums, and
|
|
prints `DEST=...`; keep that path.
|
|
|
|
```sh
|
|
ssh root@10.20.0.20 'bash -s final' < migration/backup-services.sh
|
|
```
|
|
|
|
Up to here nothing is lost: `docker start` brings everything back.
|
|
|
|
5. Install. **This wipes the disk.**
|
|
|
|
```sh
|
|
nix run github:nix-community/nixos-anywhere -- \
|
|
--flake .#asxpio-services --extra-files extra \
|
|
--ssh-option ProxyJump=root@10.20.0.10 root@10.10.0.20
|
|
```
|
|
|
|
The installer boots in RAM first and only then formats. If it cannot
|
|
be reached at that point, rebooting the server from the Hetzner
|
|
console returns to the untouched Debian.
|
|
|
|
6. Check the new system over WireGuard (same host key, no warning):
|
|
|
|
```sh
|
|
ssh root@10.20.0.20 'nixos-version; systemctl is-system-running; wg show'
|
|
```
|
|
|
|
If WireGuard is down, the server is still reachable through
|
|
asxpio-ops: `ssh -J root@10.20.0.10 root@10.10.0.20`.
|
|
|
|
7. Restore. The object-storage keys exist on the server only for this
|
|
command.
|
|
|
|
```sh
|
|
ssh root@10.20.0.20 'umask 077; cat > /run/rclone.conf' < ../rclone.conf.migration
|
|
ssh root@10.20.0.20 'RCLONE_CONFIG=/run/rclone.conf bash -s <DEST>' < migration/restore-services.sh
|
|
ssh root@10.20.0.20 'rm -f /run/rclone.conf'
|
|
```
|
|
|
|
8. Verify:
|
|
|
|
```sh
|
|
ssh root@10.20.0.20 'podman ps; systemctl --failed'
|
|
```
|
|
|
|
- Both Minecraft logs reach `Done`; join each server once.
|
|
- Forgejo → Site administration → Runners shows the runner idle.
|
|
- SillyTavern answers on `http://10.20.0.20:8000`.
|
|
- Push an empty commit to an app repo and watch one full CI run. Jobs
|
|
now run on Podman; this is the one thing the rehearsal could only
|
|
test piece by piece.
|
|
|
|
## After
|
|
|
|
9. Merge the `nixos` branch to `main`. The Ansible pipeline then covers
|
|
asxpio-ops only, and the NixOS pipeline finds nothing to change.
|
|
10. Remove `extra/` and `../rclone.conf.migration`, delete the snapshot
|
|
after a few days, and delete this directory.
|
|
|
|
## Rollback
|
|
|
|
- Before step 5: start the containers again.
|
|
- After step 5: restore the Hetzner snapshot or backup from the console.
|
|
The server comes back as the Debian system it was, with the data as of
|
|
the snapshot.
|