The XMPP certificate covers xmpp.asxp.io and conference.asxp.io, which
point at asxpio-services, so Let's Encrypt validates them there. A small
nginx on port 80 passes /.well-known/acme-challenge/ over the VPC to
Traefik, which holds all certificates, and answers 404 to the rest.
Assisted-by: Claude:opus-5-5
The pipeline's check-mode pass doesn't write the new unit files, so
enabling crawler-metrics.timer failed there and stopped the apply.
Assisted-by: Claude:opus-5-5
crawler-alert.py --metrics (5-minute timer) reads what the Traefik
access log gained since the last run, judges every request that names
an AI bot exactly like the weekly report (vendor IP ranges, forward-
confirmed rDNS, Meta's ASN) and writes cumulative counters to the
node_exporter textfile directory. Verdicts are cached per bot and IP
for a week, so DNS only sees new addresses; range files are refetched
once a day. It covers every router, while the mails keep watching
Forgejo only.
New dashboard "AI crawlers", and a row of it on the overview.
The overview's load gauge divided node_load1 by a count with fewer
labels, which matches nothing; aggregate both sides by host.
Assisted-by: Claude:opus-5-5
ops has 12 GB free on a 38 GB disk and the Traefik access log alone
is ~600 MB, so logs can't grow there. From 2026-11-14 Loki writes
chunks and index to Hetzner Object Storage (asxpio-loki-main, hel1)
with the existing project key, passed through loki.env and
-config.expand-env. Today's filesystem period ages out by retention.
Log retention goes from 14 to 90 days. Prometheus metrics are small
(~20 MB so far) and stay local, now for 90 days instead of 30.
Assisted-by: Claude:opus-5-5
blackbox_exporter matches query_response line by line, and ejabberd
writes its stream features and <proceed/> without newlines, so the
STARTTLS dialogue always timed out and the probe alert fired while
XMPP was fine. Port 5222 gets a TCP connect; the certificate is still
checked through direct TLS on 5223.
Assisted-by: Claude:opus-5-5
Each SSH login from Ansible and CI deploys creates its own
session-N.scope unit. As a journal label that made 3400+ streams and
Loki rejected new ones with 429, dropping container logs. Map them all
to session.scope.
cAdvisor's image healthcheck probes localhost:8080, but it listens on
the VPC address on 9180, so both containers showed unhealthy.
Assisted-by: Claude:opus-5-5
Nothing watched the servers: tonight's asxpio crash loop was found by
hand. On asxpio-ops, Prometheus (30 days), Loki (14 days) and Grafana
at gf.asxp.io (WireGuard only), with blackbox_exporter probing every
service from outside and postgres_exporter. On both hosts node_exporter
and cAdvisor on the VPC address, and Alloy shipping container logs, the
journal and Traefik's access log to Loki over the VPC.
Traefik and Forgejo expose their metrics; restic and the bucket replica
record their last success for node_exporter. Dashboards: an overview as
home, plus hosts, containers, Traefik, probes, Postgres and logs.
Alerts by email to me@asxp.io: host or service down, certificate under
14 days, disk or memory nearly full, a container restart-looping,
Postgres down, backup or replica stale.
Assisted-by: Claude:opus-5-5
A dry run changes nothing to verify, and for a service that is not on
the host yet the check waited a minute and failed the dry run.
Assisted-by: Claude:opus-5-5
The interop project kept a cx23 (claude-test-interop-runner, about
EUR 6/month with its IPv4) only to run its Forgejo runner. Run it here
instead, with the same registration (UUID + token, token in the vault)
and the same three labels, pinned.
Its PR checks are untrusted, so unlike our own runner its jobs get
neither the Docker socket nor host volumes. nix-host, which on the old
host used the host's Nix store as a cache, is a throwaway container for
now; a second, trusted-only registration can bring the cache back.
Rollback: remove the role from site.yaml (or stop the container) and
start the old runner VM again.
Assisted-by: Claude:opus-5-5
Floating tags (17-alpine, 3.4-slim, nginx:alpine, :latest, :debug, @v7)
change under a deploy without anyone deciding it: the storage deploy
moved Postgres to 17.12 on its own because of postgres:17-alpine. Pin
every image and action to its exact current version, so each change
arrives as a reviewed Renovate PR and a rollback is a revert.
Assisted-by: Claude:opus-5-5
restic writes both regions itself, but the ad-hoc backups under
asxpio-ops/, the Terraform state and the k8s state only ever went to
hel1. Sync each hel1 bucket into its fsn1 twin every morning from
asxpio-services, with rclone pinned like everything else.
restic/ is excluded, so the independent fsn1 restic repositories are
never touched. A run that would delete more than 1000 objects stops and
mails, so a mistake on the hel1 side cannot empty the replica the next
morning. The first dry run copied 23 objects and deleted none.
Assisted-by: Claude:opus-5-5
The VLESS Reality proxy ran from a hand-installed binary on the
temporary server. Run the upstream image instead, pinned and on the
host network, with only CAP_NET_BIND_SERVICE; its config (keys and
client id) is in the vault.
Assisted-by: Claude:opus-5-5
ejabberd, apushd and the tackygram gateway ran on the temporary
claude-test-xmpp server, a separate CX23 billed on its own. Move them
to asxpio-services, which has the room now that Fossils is gone, and
manage them like the other stacks.
Compose follows anotherim-server's compose.yaml and
compose.tackygram.yaml with CERTS=files; image versions are variables
so they can be tracked. The TURN and call addresses come from the
host's public IP instead of a hand-edited env file. Secrets, keys and
the certificate stay on the host under /opt/xmpp/state as data and are
covered by the restic backup.
Assisted-by: Claude:opus-5-5
In check mode nothing is written, so for a service that is not on the
host yet the restart task finds no project directory and fails the
whole dry run. Ignore errors there in check mode, as the compose task
already does.
Assisted-by: Claude:opus-5-5
The Fabric server holds 8.8 GB of asxpio-services' 15 GB of RAM, more
than everything else on the host together, and the XMPP server moves
here next. Save the world, remove its containers and the daily restart
timer.
The world stays in /opt/minecraft-fossils/data, and in the restic
backups, until it is deleted by hand.
Assisted-by: Claude:opus-5-5
Hetzner's server backups cost 20% of each server's price and only
restore whole disks. Back up what matters with restic instead, into the
backup buckets we already pay for: one repository per host in hel1
(asxpio-backups-main) and one in fsn1 (asxpio-backups-repl), written
independently so either region alone is a full copy.
Postgres goes in as a dump, Forgejo's and vhpbot's SQLite databases
through the online backup API, Garage with a fresh metadata snapshot,
ejabberd through its own backup command. Keeps 7 daily, 4 weekly and
6 monthly snapshots, prunes on Sundays, and mails the journal when a
run fails.
restic is the pinned upstream release, so its version is tracked in
the repo like everything else. The vault gains the restic password
and the Xray config for the role that follows.
Assisted-by: Claude:opus-5-5
The prune job now lives in /etc/cron.d on both hosts and root's crontab
entry is gone, so the tasks that removed it have nothing left to do.
Assisted-by: Claude:fable-5-2
grep -v exits 1 when it prints nothing, and with pipefail that failed
the task on both hosts, although the crontab had been rewritten
correctly. Ignore grep's exit status.
Assisted-by: Claude:fable-5-2
The roles had drifted far from production: Forgejo was pinned at 12
while 15.0.11 runs, its config lacked the mailer, packages and the real
database path, and several services existed only on the hosts. Running
a role would have rolled production back. Take the servers as the truth
and make the repository match them.
site.yaml now holds the desired state of both hosts:
- base: unattended upgrades, fail2ban, Docker daemon config and the
weekly image prune, as found on the hosts. New: a 500M journal cap,
and the Docker systemd override is kept absent. That override once
opened dockerd on tcp/2375 without authentication.
- traefik: templates as deployed, version in a variable (v3.7.18).
The Coraza WAF files are removed; no router ever used them.
- forgejo: 15.0.11, app.ini generated from the live file with the
secrets taken from the vault.
- crawler_alert, sillytavern, minecraft, minecraft_fossils: added
from the hosts.
Forgejo's canonical name becomes git.asxp.io; git.czsk.it stays as an
alias on the same router.
docker_service restarts a stack when its config changes, no longer
forces ownership on existing data directories, and hides template
diffs because they carry secrets. Containers listed in
service_graceful_stop are saved and stopped with a long timeout before
compose recreates them: Docker's 10 s default is too short for a
Minecraft world. Both game servers get stop_grace_period for the same
reason.
Removed: the full-deployment and runner playbooks with their prompts,
the Forgejo backup and restore roles for the old bare-metal host, and
inventory entries for services that were never deployed. SSH hardening
stays a manual playbook, since a mistake there locks everyone out.
New secrets (SMTP, RCON, CurseForge key) went into the vault.
Assisted-by: Claude:fable-5-2
The role had drifted from what runs on asxpio-services: capacity 4,
a bridge network per workflow so service containers start, the Docker
socket automounted into jobs, and a nix-latest label backed by a
persistent nix-store volume.
That volume caused a CI outage on 2026-10-18. nixos/nix:latest moved on,
and since the image's /bin/sh and /etc/nix/nix.conf link into /nix/store,
the old volume left them dangling. Pin the image through
forgejo_runner_nix_version and name the volume after it, so a bump is
deliberate and always gets a store seeded from its own image.
Assisted-by: Claude:opus-5-5
The ops server has 3.7 GB RAM and no swap. Forgejo, MinIO and the apps
fit with room to spare, but a spike (a big push, git gc, parallel
image pulls) would OOM-kill a service instead of paging out.
swappiness 10, so it only swaps under real pressure.
Assisted-by: Claude:opus-5
On the 38 GB ops disk the s3fs cache reserve (ensure_diskfree) never
fit, so all three mounts failed at every boot since 2026-09-22. Backups
now go to Hetzner S3 with rclone instead.
Add s3fs_enabled (default true); when false the role unmounts the
buckets and drops their fstab entries, keeping the credentials file
that the rclone wrapper reads.
Assisted-by: Claude:opus-5
