docs: restic backups, XMPP and Xray on services, Fossils retired

Describe the new roles, how to list and restore restic snapshots, and
that the restic password must be kept outside the servers.

Assisted-by: Claude:opus-5-5

README.md

@@ -9,7 +9,7 @@ Ansible. Pushing to `main` deploys: a Forgejo Actions workflow applies
 | Server | Size | Runs |
 |--------|------|------|
 | asxpio-ops | 2 vCPU, 4 GB | Traefik, Forgejo, crawler-alert; app stacks deployed from their own repos |
-| asxpio-services | 8 vCPU, 16 GB | Forgejo Actions runner, Minecraft (Paper and Fossils), SillyTavern |
+| asxpio-services | 8 vCPU, 16 GB | Forgejo Actions runner, Minecraft (Paper), SillyTavern, XMPP (asxp.io), Xray |
 
 Both are reachable for administration only over WireGuard
 (10.20.0.0/24) or the private network (10.10.0.0/20).
@@ -31,7 +31,11 @@ Both are reachable for administration only over WireGuard
 | `forgejo` | ops | Forgejo at git.asxp.io (git.czsk.it is an alias), SSH passthrough |
 | `forgejo_runner` | services | Actions runner and its job labels |
 | `sillytavern` | services | SillyTavern on the WireGuard address |
-| `minecraft`, `minecraft_fossils` | services | game servers, daily restart timer |
+| `minecraft` | services | Paper server |
+| `minecraft_fossils` | services | retired: keeps the Fabric server stopped; world left in `/opt/minecraft-fossils/data` |
+| `xmpp` | services | ejabberd, apushd and tackygram for asxp.io; secrets stay in `/opt/xmpp/state` |
+| `xray` | services | Xray VLESS Reality on :443, config in the vault |
+| `restic_backup` | both | nightly restic backup to object storage, hel1 and fsn1 |
 
 Not managed here: the app stacks on asxpio-ops (asxpio, datacenterbingo,
 blog, czsk, vhpbot, storage). Each deploys from its own repository into
@@ -47,6 +51,23 @@ blog, czsk, vhpbot, storage). Each deploys from its own repository into
 
 A run with nothing to change touches nothing and restarts nothing.
 
+## Backups
+
+Hetzner's server backups are off. Each host runs `restic-backup.timer`
+nightly (ops 03:30, services 04:15 UTC) into two independent
+repositories, `restic/<host>` in `asxpio-backups-main` (hel1) and
+`asxpio-backups-repl` (fsn1). Databases go in as dumps or online
+copies; see `restic_pre_hooks` in `ansible/inventory/host_vars/`.
+A failed run mails its journal.
+
+```sh
+restic-backup hel1 snapshots
+restic-backup fsn1 restore latest --target /tmp/restore --include /opt/xmpp
+```
+
+The password is `vault_restic_password` in the vault. Keep a copy
+outside the servers and this repo's CI: without it nothing can be read.
+
 Things to know:
 
 - **Game servers** are saved and stopped with a 120 s timeout before a

ansible/README.md

@@ -52,7 +52,8 @@ Compose files without variables are plain files under
 ## Roles in site.yaml
 
 `base`, `traefik`, `crawler_alert`, `forgejo`, `forgejo_runner`,
-`sillytavern`, `minecraft`, `minecraft_fossils`. Service roles go
+`sillytavern`, `minecraft`, `minecraft_fossils` (retired, keeps it
+stopped), `xmpp`, `xray`, `restic_backup`. Service roles go
 through `docker_service`, which writes the files, runs
 `docker compose up -d`, restarts on a config change and waits for the
 service to answer.

terraform/README.md

@@ -7,7 +7,8 @@ Hetzner Cloud infrastructure managed via Terraform.
 - **asxpio-ops** (CX23: 2 vCPU, 4GB) - Traefik, Forgejo, app stacks
 - **asxpio-services** (CX43: 8 vCPU, 16GB) - CI runner, game servers, SillyTavern
 
-Both have Hetzner daily backups plus delete and rebuild protection.
+Both have delete and rebuild protection. Hetzner's server backups are
+off; restic backs up the data instead (see ../README.md).
 
 Both servers carry `prevent_destroy` and ignore changes to `user_data`,
 `image` and `ssh_keys`: cloud-init only runs at first boot, and hcloud