The ejabberd certificate (asxp.io, conference., xmpp.) was issued by
hand and expires 2027-01-28 with nothing renewing it. Traefik now
obtains it alongside the sites' certificates, through a router that
only exists to request it, and a systemd path unit on ops pushes it to
services whenever acme.json changes, with a daily run as fallback.
The push uses its own SSH key, accepted from ops' VPC address only and
forced to xmpp-cert-install, which checks the certificate is valid for
another day and matches its key before replacing server.pem and
reloading ejabberd.
Two alerts: the sync hasn't succeeded for two days, or ejabberd serves
an older certificate than Traefik holds for six hours. The existing
expiry alert already covers port 5223.
Assisted-by: Claude:opus-5-5
The XMPP certificate covers xmpp.asxp.io and conference.asxp.io, which
point at asxpio-services, so Let's Encrypt validates them there. A small
nginx on port 80 passes /.well-known/acme-challenge/ over the VPC to
Traefik, which holds all certificates, and answers 404 to the rest.
Assisted-by: Claude:opus-5-5
Nothing watched the servers: tonight's asxpio crash loop was found by
hand. On asxpio-ops, Prometheus (30 days), Loki (14 days) and Grafana
at gf.asxp.io (WireGuard only), with blackbox_exporter probing every
service from outside and postgres_exporter. On both hosts node_exporter
and cAdvisor on the VPC address, and Alloy shipping container logs, the
journal and Traefik's access log to Loki over the VPC.
Traefik and Forgejo expose their metrics; restic and the bucket replica
record their last success for node_exporter. Dashboards: an overview as
home, plus hosts, containers, Traefik, probes, Postgres and logs.
Alerts by email to me@asxp.io: host or service down, certificate under
14 days, disk or memory nearly full, a container restart-looping,
Postgres down, backup or replica stale.
Assisted-by: Claude:opus-5-5
The interop project kept a cx23 (claude-test-interop-runner, about
EUR 6/month with its IPv4) only to run its Forgejo runner. Run it here
instead, with the same registration (UUID + token, token in the vault)
and the same three labels, pinned.
Its PR checks are untrusted, so unlike our own runner its jobs get
neither the Docker socket nor host volumes. nix-host, which on the old
host used the host's Nix store as a cache, is a throwaway container for
now; a second, trusted-only registration can bring the cache back.
Rollback: remove the role from site.yaml (or stop the container) and
start the old runner VM again.
Assisted-by: Claude:opus-5-5
The server is stopped and its data deleted (archived to object storage
as minecraft-fossils-final-20261112.tar.zst), so the role that kept it
stopped has nothing left to do.
Assisted-by: Claude:opus-5-5
restic writes both regions itself, but the ad-hoc backups under
asxpio-ops/, the Terraform state and the k8s state only ever went to
hel1. Sync each hel1 bucket into its fsn1 twin every morning from
asxpio-services, with rclone pinned like everything else.
restic/ is excluded, so the independent fsn1 restic repositories are
never touched. A run that would delete more than 1000 objects stops and
mails, so a mistake on the hel1 side cannot empty the replica the next
morning. The first dry run copied 23 objects and deleted none.
Assisted-by: Claude:opus-5-5
ejabberd, apushd and the tackygram gateway ran on the temporary
claude-test-xmpp server, a separate CX23 billed on its own. Move them
to asxpio-services, which has the room now that Fossils is gone, and
manage them like the other stacks.
Compose follows anotherim-server's compose.yaml and
compose.tackygram.yaml with CERTS=files; image versions are variables
so they can be tracked. The TURN and call addresses come from the
host's public IP instead of a hand-edited env file. Secrets, keys and
the certificate stay on the host under /opt/xmpp/state as data and are
covered by the restic backup.
Assisted-by: Claude:opus-5-5
The Fabric server holds 8.8 GB of asxpio-services' 15 GB of RAM, more
than everything else on the host together, and the XMPP server moves
here next. Save the world, remove its containers and the daily restart
timer.
The world stays in /opt/minecraft-fossils/data, and in the restic
backups, until it is deleted by hand.
Assisted-by: Claude:opus-5-5
Hetzner's server backups cost 20% of each server's price and only
restore whole disks. Back up what matters with restic instead, into the
backup buckets we already pay for: one repository per host in hel1
(asxpio-backups-main) and one in fsn1 (asxpio-backups-repl), written
independently so either region alone is a full copy.
Postgres goes in as a dump, Forgejo's and vhpbot's SQLite databases
through the online backup API, Garage with a fresh metadata snapshot,
ejabberd through its own backup command. Keeps 7 daily, 4 weekly and
6 monthly snapshots, prunes on Sundays, and mails the journal when a
run fails.
restic is the pinned upstream release, so its version is tracked in
the repo like everything else. The vault gains the restic password
and the Xray config for the role that follows.
Assisted-by: Claude:opus-5-5
The roles had drifted far from production: Forgejo was pinned at 12
while 15.0.11 runs, its config lacked the mailer, packages and the real
database path, and several services existed only on the hosts. Running
a role would have rolled production back. Take the servers as the truth
and make the repository match them.
site.yaml now holds the desired state of both hosts:
- base: unattended upgrades, fail2ban, Docker daemon config and the
weekly image prune, as found on the hosts. New: a 500M journal cap,
and the Docker systemd override is kept absent. That override once
opened dockerd on tcp/2375 without authentication.
- traefik: templates as deployed, version in a variable (v3.7.18).
The Coraza WAF files are removed; no router ever used them.
- forgejo: 15.0.11, app.ini generated from the live file with the
secrets taken from the vault.
- crawler_alert, sillytavern, minecraft, minecraft_fossils: added
from the hosts.
Forgejo's canonical name becomes git.asxp.io; git.czsk.it stays as an
alias on the same router.
docker_service restarts a stack when its config changes, no longer
forces ownership on existing data directories, and hides template
diffs because they carry secrets. Containers listed in
service_graceful_stop are saved and stopped with a long timeout before
compose recreates them: Docker's 10 s default is too short for a
Minecraft world. Both game servers get stop_grace_period for the same
reason.
Removed: the full-deployment and runner playbooks with their prompts,
the Forgejo backup and restore roles for the old bare-metal host, and
inventory entries for services that were never deployed. SSH hardening
stays a manual playbook, since a mistake there locks everyone out.
New secrets (SMTP, RCON, CurseForge key) went into the vault.
Assisted-by: Claude:fable-5-2
