restic writes both regions itself, but the ad-hoc backups under
asxpio-ops/, the Terraform state and the k8s state only ever went to
hel1. Sync each hel1 bucket into its fsn1 twin every morning from
asxpio-services, with rclone pinned like everything else.
restic/ is excluded, so the independent fsn1 restic repositories are
never touched. A run that would delete more than 1000 objects stops and
mails, so a mistake on the hel1 side cannot empty the replica the next
morning. The first dry run copied 23 objects and deleted none.
Assisted-by: Claude:opus-5-5
ejabberd, apushd and the tackygram gateway ran on the temporary
claude-test-xmpp server, a separate CX23 billed on its own. Move them
to asxpio-services, which has the room now that Fossils is gone, and
manage them like the other stacks.
Compose follows anotherim-server's compose.yaml and
compose.tackygram.yaml with CERTS=files; image versions are variables
so they can be tracked. The TURN and call addresses come from the
host's public IP instead of a hand-edited env file. Secrets, keys and
the certificate stay on the host under /opt/xmpp/state as data and are
covered by the restic backup.
Assisted-by: Claude:opus-5-5
The Fabric server holds 8.8 GB of asxpio-services' 15 GB of RAM, more
than everything else on the host together, and the XMPP server moves
here next. Save the world, remove its containers and the daily restart
timer.
The world stays in /opt/minecraft-fossils/data, and in the restic
backups, until it is deleted by hand.
Assisted-by: Claude:opus-5-5
Hetzner's server backups cost 20% of each server's price and only
restore whole disks. Back up what matters with restic instead, into the
backup buckets we already pay for: one repository per host in hel1
(asxpio-backups-main) and one in fsn1 (asxpio-backups-repl), written
independently so either region alone is a full copy.
Postgres goes in as a dump, Forgejo's and vhpbot's SQLite databases
through the online backup API, Garage with a fresh metadata snapshot,
ejabberd through its own backup command. Keeps 7 daily, 4 weekly and
6 monthly snapshots, prunes on Sundays, and mails the journal when a
run fails.
restic is the pinned upstream release, so its version is tracked in
the repo like everything else. The vault gains the restic password
and the Xray config for the role that follows.
Assisted-by: Claude:opus-5-5
The roles had drifted far from production: Forgejo was pinned at 12
while 15.0.11 runs, its config lacked the mailer, packages and the real
database path, and several services existed only on the hosts. Running
a role would have rolled production back. Take the servers as the truth
and make the repository match them.
site.yaml now holds the desired state of both hosts:
- base: unattended upgrades, fail2ban, Docker daemon config and the
weekly image prune, as found on the hosts. New: a 500M journal cap,
and the Docker systemd override is kept absent. That override once
opened dockerd on tcp/2375 without authentication.
- traefik: templates as deployed, version in a variable (v3.7.18).
The Coraza WAF files are removed; no router ever used them.
- forgejo: 15.0.11, app.ini generated from the live file with the
secrets taken from the vault.
- crawler_alert, sillytavern, minecraft, minecraft_fossils: added
from the hosts.
Forgejo's canonical name becomes git.asxp.io; git.czsk.it stays as an
alias on the same router.
docker_service restarts a stack when its config changes, no longer
forces ownership on existing data directories, and hides template
diffs because they carry secrets. Containers listed in
service_graceful_stop are saved and stopped with a long timeout before
compose recreates them: Docker's 10 s default is too short for a
Minecraft world. Both game servers get stop_grace_period for the same
reason.
Removed: the full-deployment and runner playbooks with their prompts,
the Forgejo backup and restore roles for the old bare-metal host, and
inventory entries for services that were never deployed. SSH hardening
stays a manual playbook, since a mistake there locks everyone out.
New secrets (SMTP, RCON, CurseForge key) went into the vault.
Assisted-by: Claude:fable-5-2
