ansible: nightly restic backups to both object storage regions

Hetzner's server backups cost 20% of each server's price and only
restore whole disks. Back up what matters with restic instead, into the
backup buckets we already pay for: one repository per host in hel1
(asxpio-backups-main) and one in fsn1 (asxpio-backups-repl), written
independently so either region alone is a full copy.

Postgres goes in as a dump, Forgejo's and vhpbot's SQLite databases
through the online backup API, Garage with a fresh metadata snapshot,
ejabberd through its own backup command. Keeps 7 daily, 4 weekly and
6 monthly snapshots, prunes on Sundays, and mails the journal when a
run fails.

restic is the pinned upstream release, so its version is tracked in
the repo like everything else. The vault gains the restic password
and the Xray config for the role that follows.

Assisted-by: Claude:opus-5-5

ansible/inventory/host_vars/asxpio-ops.yaml

@@ -10,3 +10,22 @@ ssh_git_user: git
 # s3fs is off here: its ensure_diskfree reserve never fit the 38 GB disk
 s3fs_enabled: false
 s3fs_ensure_diskfree: 20000
+
+# restic: Postgres as a dump, Garage with a fresh metadata snapshot,
+# Forgejo's SQLite through the online backup API
+restic_paths:
+  - /etc
+  - /opt
+  - /root/backups/rclone-hetzner.sh
+  - /var/lib/docker/volumes/storage_garage_meta
+  - /var/lib/docker/volumes/storage_garage_data
+restic_excludes:
+  - /opt/backups
+  - /opt/containerd
+restic_pre_hooks:
+  - docker exec storage-postgres pg_dumpall -U postgres > "$STAGE/pg_dumpall.sql"
+  - docker exec storage-postgres pg_dump -U postgres -Fc asxpio > "$STAGE/asxpio.dump"
+  - docker exec storage-garage /garage meta snapshot --all
+  - sqlite3 /opt/forgejo/data/gitea/gitea.db ".backup '$STAGE/forgejo-gitea.db'"
+  - sqlite3 /opt/vhpbot-prod/bot.db ".backup '$STAGE/vhpbot-prod.db'"
+restic_schedule: "*-*-* 03:30:00"

ansible/inventory/host_vars/asxpio-services.yaml

@@ -6,3 +6,21 @@ wireguard_ip: "{{ asxpio_services_wireguard_ip }}"
 ssh_allow_public_git: false
 
 s3fs_ensure_diskfree: 81920  # 80 GB free = max 80 GB cache
+
+# restic: ejabberd's Mnesia through its own backup command, its SQLite
+# through the online backup API. The Paper world is copied live (RCON is
+# off, so there is no save-off); its autosave keeps that close enough.
+restic_paths:
+  - /etc
+  - /opt
+  - /var/lib/docker/volumes/xmpp_database
+  - /var/lib/docker/volumes/xmpp_tackygram
+restic_excludes:
+  - /opt/containerd
+  - /opt/minecraft/cache
+  - /opt/minecraft/libraries
+  - /opt/minecraft/versions
+restic_pre_hooks:
+  - docker exec xmpp-ejabberd-1 ejabberdctl backup /opt/ejabberd/database/nightly.backup
+  - sqlite3 /var/lib/docker/volumes/xmpp_database/_data/ejabberd.sqlite ".backup '$STAGE/ejabberd.sqlite'"
+restic_schedule: "*-*-* 04:15:00"