The ejabberd certificate (asxp.io, conference., xmpp.) was issued by
hand and expires 2027-01-28 with nothing renewing it. Traefik now
obtains it alongside the sites' certificates, through a router that
only exists to request it, and a systemd path unit on ops pushes it to
services whenever acme.json changes, with a daily run as fallback.
The push uses its own SSH key, accepted from ops' VPC address only and
forced to xmpp-cert-install, which checks the certificate is valid for
another day and matches its key before replacing server.pem and
reloading ejabberd.
Two alerts: the sync hasn't succeeded for two days, or ejabberd serves
an older certificate than Traefik holds for six hours. The existing
expiry alert already covers port 5223.
Assisted-by: Claude:opus-5-5
crawler-alert.py --metrics (5-minute timer) reads what the Traefik
access log gained since the last run, judges every request that names
an AI bot exactly like the weekly report (vendor IP ranges, forward-
confirmed rDNS, Meta's ASN) and writes cumulative counters to the
node_exporter textfile directory. Verdicts are cached per bot and IP
for a week, so DNS only sees new addresses; range files are refetched
once a day. It covers every router, while the mails keep watching
Forgejo only.
New dashboard "AI crawlers", and a row of it on the overview.
The overview's load gauge divided node_load1 by a count with fewer
labels, which matches nothing; aggregate both sides by host.
Assisted-by: Claude:opus-5-5
ops has 12 GB free on a 38 GB disk and the Traefik access log alone
is ~600 MB, so logs can't grow there. From 2026-11-14 Loki writes
chunks and index to Hetzner Object Storage (asxpio-loki-main, hel1)
with the existing project key, passed through loki.env and
-config.expand-env. Today's filesystem period ages out by retention.
Log retention goes from 14 to 90 days. Prometheus metrics are small
(~20 MB so far) and stay local, now for 90 days instead of 30.
Assisted-by: Claude:opus-5-5
blackbox_exporter matches query_response line by line, and ejabberd
writes its stream features and <proceed/> without newlines, so the
STARTTLS dialogue always timed out and the probe alert fired while
XMPP was fine. Port 5222 gets a TCP connect; the certificate is still
checked through direct TLS on 5223.
Assisted-by: Claude:opus-5-5
Nothing watched the servers: tonight's asxpio crash loop was found by
hand. On asxpio-ops, Prometheus (30 days), Loki (14 days) and Grafana
at gf.asxp.io (WireGuard only), with blackbox_exporter probing every
service from outside and postgres_exporter. On both hosts node_exporter
and cAdvisor on the VPC address, and Alloy shipping container logs, the
journal and Traefik's access log to Loki over the VPC.
Traefik and Forgejo expose their metrics; restic and the bucket replica
record their last success for node_exporter. Dashboards: an overview as
home, plus hosts, containers, Traefik, probes, Postgres and logs.
Alerts by email to me@asxp.io: host or service down, certificate under
14 days, disk or memory nearly full, a container restart-looping,
Postgres down, backup or replica stale.
Assisted-by: Claude:opus-5-5
