nixos: describe asxpio-services as a NixOS system

asxpio-services is to move from Debian with Docker to NixOS with
Podman. This is the target system; nothing is switched over yet.

One flake holds what was spread over cloud-init, Ansible roles and
hand-made units: network and WireGuard, users and SSH restrictions,
firewall, the weekly image prune, and the four container stacks
(Paper and Fossils Minecraft, SillyTavern, the Forgejo runner), each
as a systemd unit under Podman. The runner keeps running as a
container and starts jobs through Podman's Docker-compatible socket.

Secrets (WireGuard keys, RCON and CurseForge keys) are encrypted with
sops for the admin's age key and the server's SSH host key, so the
repository holds no plaintext secret and CI needs none of them.

Containers that own data wait for /opt/.restored. On a freshly
installed disk a game server would otherwise generate a new world
exactly where the restore is about to land.

Tried on a throwaway server (.#rehearsal): install with nixos-anywhere
keeping a given host key, secret decryption, all stacks, a real Paper
server stopping with a full save, what CI jobs need from the engine
(short image names, the nix-store volume, service containers by name,
a kaniko build, the socket inside a job), an update over SSH and a
reboot.

migration/ has the scripts that move the data through object storage,
and a workflow builds the system on every push and switches the host
on main.

Assisted-by: Claude:fable-5-2

nixos/README.md

@@ -0,0 +1,85 @@
+# NixOS
+
+The whole system of a server, as one flake. `asxpio-services` runs
+NixOS; `asxpio-ops` is still Debian and managed by `../ansible`.
+
+```
+flake.nix                       inputs (nixpkgs 26.05, disko, sops-nix), hosts
+modules/base.nix                users, SSH, fail2ban, journal cap
+modules/hetzner.nix             disk layout and boot for a Hetzner Cloud VM
+hosts/asxpio-services/          network, WireGuard, firewall, containers, CI runner
+hosts/rehearsal.nix             overrides for a throwaway test server
+secrets/*.yaml                  sops-encrypted secrets
+migration/                      one-off scripts for the move from Debian
+```
+
+Containers run under Podman, one systemd unit each:
+`systemctl status podman-minecraft`, `journalctl -u podman-sillytavern`.
+`docker` is an alias for `podman`, and `/run/docker.sock` is Podman's
+Docker-compatible socket, which the CI runner uses to start jobs.
+
+## Changing something
+
+Edit, push a branch (CI builds the system), merge to `main` (CI
+deploys). By hand, over WireGuard:
+
+```sh
+nixos-rebuild switch --flake .#asxpio-services --target-host root@10.20.0.20
+```
+
+A deploy restarts only the units whose definition changed. Changing a
+Minecraft container restarts that server; it gets 120 s to save first.
+A bad deploy is undone with `nixos-rebuild switch --rollback` on the
+host, or by picking the previous generation in the boot menu.
+
+To update all packages: `nix flake update`, commit `flake.lock`.
+
+## Secrets
+
+`secrets/asxpio-services.yaml` is encrypted with
+[sops](https://github.com/getsops/sops) for two keys listed in
+`.sops.yaml`: the admin's age key and the server's SSH host key. The
+server decrypts at boot into `/run/secrets`.
+
+```sh
+sops secrets/asxpio-services.yaml      # edit; needs ~/.config/sops/age/keys.txt
+```
+
+The server's SSH host key is therefore part of its identity. Keep a copy
+outside the server (the `system` archive of a backup contains
+`/etc/ssh`), and hand it to a reinstall with `--extra-files`.
+
+## Installing or reinstalling a server
+
+Never delete and recreate a production server: it is billed at an old
+price that a new server would not get. Reinstall in place.
+
+1. Put the host key where the installer will copy it from:
+
+   ```sh
+   mkdir -p extra/etc/ssh        # then place ssh_host_ed25519_key{,.pub}, mode 600
+   ```
+
+2. Install. The running system is replaced; **the disk is wiped**.
+
+   ```sh
+   nix run github:nix-community/nixos-anywhere -- \
+     --flake .#asxpio-services --extra-files extra \
+     --ssh-option ProxyJump=root@10.20.0.10 root@10.10.0.20
+   ```
+
+   The installer has no WireGuard, so it is reached over the private
+   network through asxpio-ops.
+
+3. Restore data with `migration/restore-services.sh`. Containers that
+   own data do not start until `/opt/.restored` exists, so nothing
+   generates a fresh world where the restore is about to land.
+
+If a server is ever lost entirely, `terraform apply` creates a bare one
+and the steps above install onto it.
+
+## Rehearsal
+
+`.#rehearsal` is the same system for a throwaway server: plain DHCP, SSH
+from anywhere, no WireGuard, dummy secrets. Use it to try an install or
+a risky change before production.