Checked against the Hetzner project:
- Both servers have daily backups and delete and rebuild protection
switched on, while the code said backups = false and set no
protection. An apply would have turned all three off and discarded
the existing backups. Record them as they are.
- The firewalls had drifted both ways. Live, asxpio-ops also accepts
XMPP ports and tcp/5000, and asxpio-services accepts 80, 443, XMPP,
TURN, Valheim and Factorio ports; none of those services run. The
code in turn missed tcp/25566, where the second Minecraft server
listens.
Describe the firewalls as they should be: internal networks, plus the
public ports of services that run. Public SSH to asxpio-services goes
too; sshd and iptables already refuse it from outside the VPN.
This is not applied. The next plan will show the stale ports being
closed and nothing else.
Assisted-by: Claude:fable-5-2
The prune job now lives in /etc/cron.d on both hosts and root's crontab
entry is gone, so the tasks that removed it have nothing left to do.
Assisted-by: Claude:fable-5-2
grep -v exits 1 when it prints nothing, and with pipefail that failed
the task on both hosts, although the crontab had been rewritten
correctly. Ignore grep's exit status.
Assisted-by: Claude:fable-5-2
The READMEs still tracked the 2025 migration from bare metal and listed
Grafana, Immich, Valheim, Prosody and others that do not run. Replace
them with what is deployed, how a change reaches production and where
the secrets live.
Assisted-by: Claude:fable-5-2
Service changes were made by hand on the servers, which is how the
repository drifted. Apply site.yaml from a workflow instead: every
push dry-runs it against production, and a push to main applies it.
Jobs run in containers on asxpio-services. asxpio-ops is reached over
the private network; the services host is reached through asxpio-ops,
because a container connecting to its own host arrives from a Docker
bridge address that sshd and the firewall refuse. Host keys are pinned
in known_hosts.
Ansible comes from a pinned nixpkgs revision so every run uses the
same version.
Assisted-by: Claude:fable-5-2
The roles had drifted far from production: Forgejo was pinned at 12
while 15.0.11 runs, its config lacked the mailer, packages and the real
database path, and several services existed only on the hosts. Running
a role would have rolled production back. Take the servers as the truth
and make the repository match them.
site.yaml now holds the desired state of both hosts:
- base: unattended upgrades, fail2ban, Docker daemon config and the
weekly image prune, as found on the hosts. New: a 500M journal cap,
and the Docker systemd override is kept absent. That override once
opened dockerd on tcp/2375 without authentication.
- traefik: templates as deployed, version in a variable (v3.7.18).
The Coraza WAF files are removed; no router ever used them.
- forgejo: 15.0.11, app.ini generated from the live file with the
secrets taken from the vault.
- crawler_alert, sillytavern, minecraft, minecraft_fossils: added
from the hosts.
Forgejo's canonical name becomes git.asxp.io; git.czsk.it stays as an
alias on the same router.
docker_service restarts a stack when its config changes, no longer
forces ownership on existing data directories, and hides template
diffs because they carry secrets. Containers listed in
service_graceful_stop are saved and stopped with a long timeout before
compose recreates them: Docker's 10 s default is too short for a
Minecraft world. Both game servers get stop_grace_period for the same
reason.
Removed: the full-deployment and runner playbooks with their prompts,
the Forgejo backup and restore roles for the old bare-metal host, and
inventory entries for services that were never deployed. SSH hardening
stays a manual playbook, since a mistake there locks everyone out.
New secrets (SMTP, RCON, CurseForge key) went into the vault.
Assisted-by: Claude:fable-5-2
hcloud replaces a server when user_data, image or ssh_keys change, and
cloud-init only runs at first boot anyway. An edit to cloud-init, like
adding a user, would therefore plan the destruction of a production
machine. Ignore changes to those three attributes and set
prevent_destroy on both servers, so cloud-init describes a rebuild and
nothing else.
Add user zerrtika to cloud-init to match the servers; the user was
created on them by playbook 04.
Assisted-by: Claude:fable-5-2
The role had drifted from what runs on asxpio-services: capacity 4,
a bridge network per workflow so service containers start, the Docker
socket automounted into jobs, and a nix-latest label backed by a
persistent nix-store volume.
That volume caused a CI outage on 2026-10-18. nixos/nix:latest moved on,
and since the image's /bin/sh and /etc/nix/nix.conf link into /nix/store,
the old volume left them dangling. Pin the image through
forgejo_runner_nix_version and name the volume after it, so a bump is
deliberate and always gets a store seeded from its own image.
Assisted-by: Claude:opus-5-5
The ops server has 3.7 GB RAM and no swap. Forgejo, MinIO and the apps
fit with room to spare, but a spike (a big push, git gc, parallel
image pulls) would OOM-kill a service instead of paging out.
swappiness 10, so it only swaps under real pressure.
Assisted-by: Claude:opus-5
On the 38 GB ops disk the s3fs cache reserve (ensure_diskfree) never
fit, so all three mounts failed at every boot since 2026-09-22. Backups
now go to Hetzner S3 with rclone instead.
Add s3fs_enabled (default true); when false the role unmounts the
buckets and drops their fstab entries, keeping the credentials file
that the rclone wrapper reads.
Assisted-by: Claude:opus-5
