Write down the cutover for asxpio-services step by step, with the
times measured in the rehearsal: backup 6 minutes, restore of 29 GB
3 minutes.
The backup script now tolerates files changing under tar in live mode,
which a running Minecraft world always does; in final mode, with the
servers stopped, that remains an error. The restore script can leave
the containers stopped for a restore test.
Also checked on the test server before it was deleted: the production
network layout (interfaces matched by MAC, static IPv6, the private
network on enp7s0) comes up after a reboot with the same routes the
Debian host has.
Assisted-by: Claude:fable-5-2
The first run failed with "home directory /homeless-shelter exists":
job containers are unprivileged, Nix cannot sandbox builds in them, and
unsandboxed builds trip over each other's leftovers.
Evaluate the system in the job instead, which still catches broken
options before main, and let nixos-rebuild build on the target host,
where the Nix daemon has a proper sandbox.
Assisted-by: Claude:fable-5-2
An install reads the server's SSH host key from nixos/extra/, and that
key also decrypts the server's secrets. Ignore the directory, along
with nix build result links.
Assisted-by: Claude:fable-5-2
With asxpio-services described in nixos/, Ansible only has asxpio-ops
left. Drop the runner, SillyTavern and Minecraft roles, the services
host from both inventories, and the WireGuard client play.
The s3fs role and playbook go as well: the mounts were switched off on
asxpio-ops, and on asxpio-services nothing read them. The credentials
file the role once wrote on asxpio-ops stays, because the backup script
there reads it.
site.yaml against asxpio-ops still changes nothing.
Assisted-by: Claude:fable-5-2
asxpio-services is to move from Debian with Docker to NixOS with
Podman. This is the target system; nothing is switched over yet.
One flake holds what was spread over cloud-init, Ansible roles and
hand-made units: network and WireGuard, users and SSH restrictions,
firewall, the weekly image prune, and the four container stacks
(Paper and Fossils Minecraft, SillyTavern, the Forgejo runner), each
as a systemd unit under Podman. The runner keeps running as a
container and starts jobs through Podman's Docker-compatible socket.
Secrets (WireGuard keys, RCON and CurseForge keys) are encrypted with
sops for the admin's age key and the server's SSH host key, so the
repository holds no plaintext secret and CI needs none of them.
Containers that own data wait for /opt/.restored. On a freshly
installed disk a game server would otherwise generate a new world
exactly where the restore is about to land.
Tried on a throwaway server (.#rehearsal): install with nixos-anywhere
keeping a given host key, secret decryption, all stacks, a real Paper
server stopping with a full save, what CI jobs need from the engine
(short image names, the nix-store volume, service containers by name,
a kaniko build, the socket inside a job), an update over SSH and a
reboot.
migration/ has the scripts that move the data through object storage,
and a workflow builds the system on every push and switches the host
on main.
Assisted-by: Claude:fable-5-2
The cost table was a year old. Use the amounts billed now.
Both servers still cost less than the list price for their type
(3.99 against 5.49, 11.99 against 15.99). That price belongs to the
existing server, so say plainly that a reinstall has to happen in
place.
Assisted-by: Claude:fable-5-2
With provider 1.54 the plan replaces both servers. Hetzner stopped
returning a server's datacenter, the old provider derived the location
from it, and a location that reads back empty differs from "hel1",
which forces replacement. Only prevent_destroy stood in the way.
Provider 1.70.2 reads the location properly: the same plan then shows
the two firewall updates and nothing else. The lock file is not
tracked, so raise the version constraint to keep the old provider out.
Run terraform init -upgrade once in existing checkouts.
Assisted-by: Claude:fable-5-2
Checked against the Hetzner project:
- Both servers have daily backups and delete and rebuild protection
switched on, while the code said backups = false and set no
protection. An apply would have turned all three off and discarded
the existing backups. Record them as they are.
- The firewalls had drifted both ways. Live, asxpio-ops also accepts
XMPP ports and tcp/5000, and asxpio-services accepts 80, 443, XMPP,
TURN, Valheim and Factorio ports; none of those services run. The
code in turn missed tcp/25566, where the second Minecraft server
listens.
Describe the firewalls as they should be: internal networks, plus the
public ports of services that run. Public SSH to asxpio-services goes
too; sshd and iptables already refuse it from outside the VPN.
This is not applied. The next plan will show the stale ports being
closed and nothing else.
Assisted-by: Claude:fable-5-2
The prune job now lives in /etc/cron.d on both hosts and root's crontab
entry is gone, so the tasks that removed it have nothing left to do.
Assisted-by: Claude:fable-5-2
grep -v exits 1 when it prints nothing, and with pipefail that failed
the task on both hosts, although the crontab had been rewritten
correctly. Ignore grep's exit status.
Assisted-by: Claude:fable-5-2
The READMEs still tracked the 2025 migration from bare metal and listed
Grafana, Immich, Valheim, Prosody and others that do not run. Replace
them with what is deployed, how a change reaches production and where
the secrets live.
Assisted-by: Claude:fable-5-2
Service changes were made by hand on the servers, which is how the
repository drifted. Apply site.yaml from a workflow instead: every
push dry-runs it against production, and a push to main applies it.
Jobs run in containers on asxpio-services. asxpio-ops is reached over
the private network; the services host is reached through asxpio-ops,
because a container connecting to its own host arrives from a Docker
bridge address that sshd and the firewall refuse. Host keys are pinned
in known_hosts.
Ansible comes from a pinned nixpkgs revision so every run uses the
same version.
Assisted-by: Claude:fable-5-2
The roles had drifted far from production: Forgejo was pinned at 12
while 15.0.11 runs, its config lacked the mailer, packages and the real
database path, and several services existed only on the hosts. Running
a role would have rolled production back. Take the servers as the truth
and make the repository match them.
site.yaml now holds the desired state of both hosts:
- base: unattended upgrades, fail2ban, Docker daemon config and the
weekly image prune, as found on the hosts. New: a 500M journal cap,
and the Docker systemd override is kept absent. That override once
opened dockerd on tcp/2375 without authentication.
- traefik: templates as deployed, version in a variable (v3.7.18).
The Coraza WAF files are removed; no router ever used them.
- forgejo: 15.0.11, app.ini generated from the live file with the
secrets taken from the vault.
- crawler_alert, sillytavern, minecraft, minecraft_fossils: added
from the hosts.
Forgejo's canonical name becomes git.asxp.io; git.czsk.it stays as an
alias on the same router.
docker_service restarts a stack when its config changes, no longer
forces ownership on existing data directories, and hides template
diffs because they carry secrets. Containers listed in
service_graceful_stop are saved and stopped with a long timeout before
compose recreates them: Docker's 10 s default is too short for a
Minecraft world. Both game servers get stop_grace_period for the same
reason.
Removed: the full-deployment and runner playbooks with their prompts,
the Forgejo backup and restore roles for the old bare-metal host, and
inventory entries for services that were never deployed. SSH hardening
stays a manual playbook, since a mistake there locks everyone out.
New secrets (SMTP, RCON, CurseForge key) went into the vault.
Assisted-by: Claude:fable-5-2
hcloud replaces a server when user_data, image or ssh_keys change, and
cloud-init only runs at first boot anyway. An edit to cloud-init, like
adding a user, would therefore plan the destruction of a production
machine. Ignore changes to those three attributes and set
prevent_destroy on both servers, so cloud-init describes a rebuild and
nothing else.
Add user zerrtika to cloud-init to match the servers; the user was
created on them by playbook 04.
Assisted-by: Claude:fable-5-2
The role had drifted from what runs on asxpio-services: capacity 4,
a bridge network per workflow so service containers start, the Docker
socket automounted into jobs, and a nix-latest label backed by a
persistent nix-store volume.
That volume caused a CI outage on 2026-10-18. nixos/nix:latest moved on,
and since the image's /bin/sh and /etc/nix/nix.conf link into /nix/store,
the old volume left them dangling. Pin the image through
forgejo_runner_nix_version and name the volume after it, so a bump is
deliberate and always gets a store seeded from its own image.
Assisted-by: Claude:opus-5-5
The ops server has 3.7 GB RAM and no swap. Forgejo, MinIO and the apps
fit with room to spare, but a spike (a big push, git gc, parallel
image pulls) would OOM-kill a service instead of paging out.
swappiness 10, so it only swaps under real pressure.
Assisted-by: Claude:opus-5
On the 38 GB ops disk the s3fs cache reserve (ensure_diskfree) never
fit, so all three mounts failed at every boot since 2026-09-22. Backups
now go to Hetzner S3 with rclone instead.
Add s3fs_enabled (default true); when false the role unmounts the
buckets and drops their fstab entries, keeping the credentials file
that the rclone wrapper reads.
Assisted-by: Claude:opus-5
