nixos: describe asxpio-services as a NixOS system

asxpio-services is to move from Debian with Docker to NixOS with
Podman. This is the target system; nothing is switched over yet.

One flake holds what was spread over cloud-init, Ansible roles and
hand-made units: network and WireGuard, users and SSH restrictions,
firewall, the weekly image prune, and the four container stacks
(Paper and Fossils Minecraft, SillyTavern, the Forgejo runner), each
as a systemd unit under Podman. The runner keeps running as a
container and starts jobs through Podman's Docker-compatible socket.

Secrets (WireGuard keys, RCON and CurseForge keys) are encrypted with
sops for the admin's age key and the server's SSH host key, so the
repository holds no plaintext secret and CI needs none of them.

Containers that own data wait for /opt/.restored. On a freshly
installed disk a game server would otherwise generate a new world
exactly where the restore is about to land.

Tried on a throwaway server (.#rehearsal): install with nixos-anywhere
keeping a given host key, secret decryption, all stacks, a real Paper
server stopping with a full save, what CI jobs need from the engine
(short image names, the nix-store volume, service containers by name,
a kaniko build, the socket inside a job), an update over SSH and a
reboot.

migration/ has the scripts that move the data through object storage,
and a workflow builds the system on every push and switches the host
on main.

Assisted-by: Claude:fable-5-2

nixos/migration/backup-services.sh

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+# Runs on asxpio-services (Debian). Streams everything worth keeping to
+# object storage as tar.zst archives, one per service, with checksums.
+#
+#   backup-services.sh live     copy while services run (rehearsal, timing)
+#   backup-services.sh final    stop the game servers and the runner first
+#
+# Nothing is written to the local disk. Ownership is stored numerically so
+# a restore on another OS keeps uid/gid.
+set -euo pipefail
+
+mode=${1:?usage: backup-services.sh live|final}
+stamp=$(date -u +%Y%m%dT%H%M%SZ)
+dest="asxpio-bucket-hel:asxpio-backups-main/asxpio-services/nixos-migration/$mode-$stamp"
+# Hetzner needs the region set and rejects the bucket check rclone does
+rc() { rclone --s3-region hel1 --s3-no-check-bucket --s3-chunk-size 64M "$@"; }
+
+if [ "$mode" = final ]; then
+  echo "== stopping services"
+  docker exec minecraft-fossils rcon-cli save-all flush || true
+  docker stop -t 120 minecraft-fossils minecraft
+  docker stop -t 30 minecraft-fossils-wlsync sillytavern forgejo-runner
+fi
+
+sums=$(mktemp)
+archive() { # name, then paths relative to /
+  local name=$1
+  shift
+  echo "== $name: $*"
+  # tar exits 1 when a file changed while being read. That is expected
+  # for a live copy of a running server and an error in a final one.
+  { tar --numeric-owner --acls --xattrs -C / -cf - "$@" || { [ $? -eq 1 ] && [ "$mode" = live ]; }; } \
+    | zstd -T4 -3 \
+    | tee >(sha256sum | sed "s/-\$/$name.tar.zst/" >>"$sums") \
+    | rc rcat "$dest/$name.tar.zst"
+}
+
+archive minecraft opt/minecraft
+archive minecraft-fossils opt/minecraft-fossils
+archive sillytavern opt/sillytavern
+archive forgejo-runner opt/forgejo-runner
+# Not restored on NixOS; kept in case something in them is wanted later
+archive legacy opt/immich opt/wordpress opt/traefik \
+  var/lib/docker/volumes/ppeu-wp_wordpress var/lib/docker/volumes/ppeu-wp_wordpress_db
+archive system etc root home usr/local var/spool/cron
+
+sleep 1
+rc rcat "$dest/SHA256SUMS" <"$sums"
+echo "== uploaded to $dest"
+rc lsl "$dest"
+echo "== verifying checksums against the uploaded objects"
+while read -r sum name; do
+  got=$(rc cat "$dest/$name" | sha256sum | cut -d' ' -f1)
+  [ "$got" = "$sum" ] && echo "ok   $name" || { echo "FAIL $name"; exit 1; }
+done <"$sums"
+rm -f "$sums"
+echo "DEST=$dest"

nixos/migration/restore-services.sh

@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+# Runs on the freshly installed NixOS host. Restores service data from the
+# archives written by backup-services.sh, verifies each against SHA256SUMS,
+# then lets the containers start.
+#
+#   RCLONE_CONFIG=/run/rclone.conf restore-services.sh <remote:path> [name...]
+#
+# Without names it restores everything the host needs. The rclone config
+# with the object-storage keys is supplied by the operator for the run and
+# removed afterwards; it is not part of the system.
+set -euo pipefail
+
+src=${1:?usage: restore-services.sh <remote:path> [name...]}
+shift
+names=("$@")
+[ ${#names[@]} -gt 0 ] || names=(minecraft minecraft-fossils sillytavern forgejo-runner)
+
+rc() { rclone --s3-region hel1 --s3-no-check-bucket "$@"; }
+
+sums=$(mktemp)
+rc cat "$src/SHA256SUMS" >"$sums"
+
+for name in "${names[@]}"; do
+  want=$(awk -v f="$name.tar.zst" '$2 == f {print $1}' "$sums")
+  [ -n "$want" ] || { echo "no checksum for $name" >&2; exit 1; }
+  echo "== restoring $name"
+  got=$(rc cat "$src/$name.tar.zst" \
+    | tee >(zstd -d | tar --numeric-owner --acls --xattrs -xpf - -C /) \
+    | sha256sum | cut -d' ' -f1)
+  wait
+  [ "$got" = "$want" ] || { echo "checksum mismatch for $name" >&2; exit 1; }
+  echo "ok   $name"
+done
+rm -f "$sums"
+
+touch /opt/.restored
+echo "== data restored; starting containers"
+systemctl start --no-block \
+  podman-minecraft.service podman-minecraft-fossils.service \
+  podman-minecraft-fossils-wlsync.service podman-sillytavern.service \
+  podman-forgejo-runner.service