The ejabberd certificate (asxp.io, conference., xmpp.) was issued by
hand and expires 2027-01-28 with nothing renewing it. Traefik now
obtains it alongside the sites' certificates, through a router that
only exists to request it, and a systemd path unit on ops pushes it to
services whenever acme.json changes, with a daily run as fallback.
The push uses its own SSH key, accepted from ops' VPC address only and
forced to xmpp-cert-install, which checks the certificate is valid for
another day and matches its key before replacing server.pem and
reloading ejabberd.
Two alerts: the sync hasn't succeeded for two days, or ejabberd serves
an older certificate than Traefik holds for six hours. The existing
expiry alert already covers port 5223.
Assisted-by: Claude:opus-5-5
The XMPP certificate covers xmpp.asxp.io and conference.asxp.io, which
point at asxpio-services, so Let's Encrypt validates them there. A small
nginx on port 80 passes /.well-known/acme-challenge/ over the VPC to
Traefik, which holds all certificates, and answers 404 to the rest.
Assisted-by: Claude:opus-5-5
xmpp.asxp.io and conference.asxp.io point at services, so the Let's
Encrypt HTTP challenge for the XMPP certificate arrives there. Only
/.well-known/acme-challenge/ is answered (forwarded to Traefik on ops);
everything else gets a 404.
Assisted-by: Claude:opus-5-5
The pipeline's check-mode pass doesn't write the new unit files, so
enabling crawler-metrics.timer failed there and stopped the apply.
Assisted-by: Claude:opus-5-5
crawler-alert.py --metrics (5-minute timer) reads what the Traefik
access log gained since the last run, judges every request that names
an AI bot exactly like the weekly report (vendor IP ranges, forward-
confirmed rDNS, Meta's ASN) and writes cumulative counters to the
node_exporter textfile directory. Verdicts are cached per bot and IP
for a week, so DNS only sees new addresses; range files are refetched
once a day. It covers every router, while the mails keep watching
Forgejo only.
New dashboard "AI crawlers", and a row of it on the overview.
The overview's load gauge divided node_load1 by a count with fewer
labels, which matches nothing; aggregate both sides by host.
Assisted-by: Claude:opus-5-5
ops has 12 GB free on a 38 GB disk and the Traefik access log alone
is ~600 MB, so logs can't grow there. From 2026-11-14 Loki writes
chunks and index to Hetzner Object Storage (asxpio-loki-main, hel1)
with the existing project key, passed through loki.env and
-config.expand-env. Today's filesystem period ages out by retention.
Log retention goes from 14 to 90 days. Prometheus metrics are small
(~20 MB so far) and stay local, now for 90 days instead of 30.
Assisted-by: Claude:opus-5-5
blackbox_exporter matches query_response line by line, and ejabberd
writes its stream features and <proceed/> without newlines, so the
STARTTLS dialogue always timed out and the probe alert fired while
XMPP was fine. Port 5222 gets a TCP connect; the certificate is still
checked through direct TLS on 5223.
Assisted-by: Claude:opus-5-5
Superuser login for the asxp.io admin moves to admin.asxp.io, which
Traefik only serves to WireGuard and the VPC. The public A record is
there for the Let's Encrypt HTTP challenge, like gf; clients reach it
through split-horizon hosts entries.
Assisted-by: Claude:opus-5-5
Each SSH login from Ansible and CI deploys creates its own
session-N.scope unit. As a journal label that made 3400+ streams and
Loki rejected new ones with 429, dropping container logs. Map them all
to session.scope.
cAdvisor's image healthcheck probes localhost:8080, but it listens on
the VPC address on 9180, so both containers showed unhealthy.
Assisted-by: Claude:opus-5-5
Grafana is WireGuard-only; WireGuard clients map the name to the ops
WireGuard address. A public AAAA record made browsers prefer the public
path, where the allowlist answers 403.
Assisted-by: Claude:opus-5-5
Nothing watched the servers: tonight's asxpio crash loop was found by
hand. On asxpio-ops, Prometheus (30 days), Loki (14 days) and Grafana
at gf.asxp.io (WireGuard only), with blackbox_exporter probing every
service from outside and postgres_exporter. On both hosts node_exporter
and cAdvisor on the VPC address, and Alloy shipping container logs, the
journal and Traefik's access log to Loki over the VPC.
Traefik and Forgejo expose their metrics; restic and the bucket replica
record their last success for node_exporter. Dashboards: an overview as
home, plus hosts, containers, Traefik, probes, Postgres and logs.
Alerts by email to me@asxp.io: host or service down, certificate under
14 days, disk or memory nearly full, a container restart-looping,
Postgres down, backup or replica stale.
Assisted-by: Claude:opus-5-5
A dry run changes nothing to verify, and for a service that is not on
the host yet the check waited a minute and failed the dry run.
Assisted-by: Claude:opus-5-5
Same pattern as traefik.asxp.io: public records to asxpio-ops, so
Traefik gets a Let's Encrypt certificate over HTTP-01, while an IP
allowlist limits it to WireGuard.
Assisted-by: Claude:opus-5-5
The interop project kept a cx23 (claude-test-interop-runner, about
EUR 6/month with its IPv4) only to run its Forgejo runner. Run it here
instead, with the same registration (UUID + token, token in the vault)
and the same three labels, pinned.
Its PR checks are untrusted, so unlike our own runner its jobs get
neither the Docker socket nor host volumes. nix-host, which on the old
host used the host's Nix store as a cache, is a throwaway container for
now; a second, trusted-only registration can bring the cache back.
Rollback: remove the role from site.yaml (or stop the container) and
start the old runner VM again.
Assisted-by: Claude:opus-5-5
Under runner 13 the run that recreates the runner ends as failed, not
cancelled (run 450); say so, and where its log stops, so it is not
mistaken for a real failure.
Assisted-by: Claude:opus-5-5
Floating tags (17-alpine, 3.4-slim, nginx:alpine, :latest, :debug, @v7)
change under a deploy without anyone deciding it: the storage deploy
moved Postgres to 17.12 on its own because of postgres:17-alpine. Pin
every image and action to its exact current version, so each change
arrives as a reviewed Renovate PR and a rollback is a revert.
Assisted-by: Claude:opus-5-5
Renovate now runs from its own repository for every repo, with the
shared rules in its preset. Extend that preset here and keep only what
is specific to this repo (the Ansible var manager, the Paper JRE, the
runner and nix notes); drop the workflow that ran it for iac alone.
Assisted-by: Claude:opus-5-5
Both recreate the runner mid-job: the applying run shows as cancelled,
leaves its job container behind, and needs a later run to confirm.
Write the follow-up steps down where the Renovate PRs send you.
Assisted-by: Claude:opus-5-5
The unquoted value contains ": ", which YAML reads as a nested mapping,
so Forgejo rejected the whole workflow and the last two pushes ran
nothing. Quote it.
Assisted-by: Claude:opus-5-5
The server is stopped and its data deleted (archived to object storage
as minecraft-fossils-final-20261112.tar.zst), so the role that kept it
stopped has nothing left to do.
Assisted-by: Claude:opus-5-5
Every run was listed as "deploy", so dry runs on Renovate branches
looked like deployments. Name each run after what it does and the
branch it ran on.
Assisted-by: Claude:opus-5-5
The repository was renamed from infrastructure to iac. Renovate looks
repositories up by name, and with push-to-create on, the old name now
belongs to a different, empty repository.
Assisted-by: Claude:opus-5-5
restic writes both regions itself, but the ad-hoc backups under
asxpio-ops/, the Terraform state and the k8s state only ever went to
hel1. Sync each hel1 bucket into its fsn1 twin every morning from
asxpio-services, with rclone pinned like everything else.
restic/ is excluded, so the independent fsn1 restic repositories are
never touched. A run that would delete more than 1000 objects stops and
mails, so a mistake on the hel1 side cannot empty the replica the next
morning. The first dry run copied 23 objects and deleted none.
Assisted-by: Claude:opus-5-5
The mounts were only ever read by one-off migration scripts, and their
cache had grown to 9.7 GB. restic and the bucket replica talk to object
storage directly, so nothing needs a filesystem view of the buckets.
Disable s3fs like on asxpio-ops; the bootstrap playbook unmounts and
drops the fstab lines (applied by hand, cache removed).
The playbook still created Immich directories under the mount and its
summary needed the cache size, which a disabled host no longer sets.
Assisted-by: Claude:opus-5-5
The zones were edited by hand in the console, and it showed: dozens of
records for servers that no longer existed, a CNAME and a TXT record
missing their trailing dot, and a 24 h default TTL that made today's
XMPP move linger in resolver caches.
Now both zones live in the asxpio project, so describe them here,
adopted with import blocks (the first apply imported 87 and changed
nothing but protection). Our servers' records reference hcloud_server
instead of literal IPs. Every record set is change-protected except
those of the test XMPP servers, and both zones are delete-protected.
Assisted-by: Claude:opus-5-5
Immich has not run for months and its photos are now in Google Photos.
asxpio-immich-main and asxpio-immich-repl (about 140 GB each) are
deleted; remove them from Terraform (state too) and from the s3fs
mounts, so nothing recreates or remounts them. The mount on
asxpio-services was taken down and its fstab line removed by hand.
Assisted-by: Claude:opus-5-5
/opt/vhpbot-prod/bot.db is an empty directory that Docker created for a
bind mount whose file never existed; the bot keeps no database there.
The hook could only fail and mark every nightly run as failed.
Assisted-by: Claude:opus-5-5
Versions are pinned across the Ansible vars, compose files, Terraform
providers and workflows, and nothing says when one is behind. Run
Renovate daily from Forgejo Actions: one PR per available bump, with
the upstream changelog in the body, which the deploy pipeline then
dry-runs like any other branch.
Variables are tracked through a "# renovate: datasource=... depName=..."
comment above them; Forgejo, Traefik, the runner and its nix image get
one here. The Paper JRE tag is left alone, the runner and nix bumps
carry their manual follow-up steps in the PR, and Renovate's own image
is bumped monthly.
Needs the repository secrets RENOVATE_TOKEN (bot user with write
access) and GH_COM_TOKEN (read-only github.com token for release
notes).
Assisted-by: Claude:opus-5-5
Describe the new roles, how to list and restore restic snapshots, and
that the restic password must be kept outside the servers.
Assisted-by: Claude:opus-5-5
Open the ports of the XMPP server and Xray, which move to
asxpio-services, and close 25566 with Minecraft Fossils gone.
Turn off Hetzner's server backups on both hosts: restic now backs up
the data to object storage in two regions, and the server backups cost
20% of each server's price. Disabling them deletes the existing backup
images, so apply that part only after restic has a verified snapshot.
Assisted-by: Claude:opus-5-5
The VLESS Reality proxy ran from a hand-installed binary on the
temporary server. Run the upstream image instead, pinned and on the
host network, with only CAP_NET_BIND_SERVICE; its config (keys and
client id) is in the vault.
Assisted-by: Claude:opus-5-5
ejabberd, apushd and the tackygram gateway ran on the temporary
claude-test-xmpp server, a separate CX23 billed on its own. Move them
to asxpio-services, which has the room now that Fossils is gone, and
manage them like the other stacks.
Compose follows anotherim-server's compose.yaml and
compose.tackygram.yaml with CERTS=files; image versions are variables
so they can be tracked. The TURN and call addresses come from the
host's public IP instead of a hand-edited env file. Secrets, keys and
the certificate stay on the host under /opt/xmpp/state as data and are
covered by the restic backup.
Assisted-by: Claude:opus-5-5
In check mode nothing is written, so for a service that is not on the
host yet the restart task finds no project directory and fails the
whole dry run. Ignore errors there in check mode, as the compose task
already does.
Assisted-by: Claude:opus-5-5
The Fabric server holds 8.8 GB of asxpio-services' 15 GB of RAM, more
than everything else on the host together, and the XMPP server moves
here next. Save the world, remove its containers and the daily restart
timer.
The world stays in /opt/minecraft-fossils/data, and in the restic
backups, until it is deleted by hand.
Assisted-by: Claude:opus-5-5
Hetzner's server backups cost 20% of each server's price and only
restore whole disks. Back up what matters with restic instead, into the
backup buckets we already pay for: one repository per host in hel1
(asxpio-backups-main) and one in fsn1 (asxpio-backups-repl), written
independently so either region alone is a full copy.
Postgres goes in as a dump, Forgejo's and vhpbot's SQLite databases
through the online backup API, Garage with a fresh metadata snapshot,
ejabberd through its own backup command. Keeps 7 daily, 4 weekly and
6 monthly snapshots, prunes on Sundays, and mails the journal when a
run fails.
restic is the pinned upstream release, so its version is tracked in
the repo like everything else. The vault gains the restic password
and the Xray config for the role that follows.
Assisted-by: Claude:opus-5-5
