asxp.io infrastructure

Two Hetzner Cloud servers, described by Terraform and configured by Ansible. Pushing to main deploys: a Forgejo Actions workflow applies ansible/site.yaml to both servers.

Servers

Server	Size	Runs
asxpio-ops	2 vCPU, 4 GB	Traefik, Forgejo, crawler-alert; app stacks deployed from their own repos
asxpio-services	8 vCPU, 16 GB	Forgejo Actions runner, Minecraft (Paper), SillyTavern, XMPP (asxp.io), Xray

Both are reachable for administration only over WireGuard (10.20.0.0/24) or the private network (10.10.0.0/20).

Layout

• terraform/: servers, network, firewalls, object storage buckets.
• ansible/site.yaml: everything the pipeline manages (see below).
• ansible/playbooks/: host bootstrap, run by hand.
• .forgejo/workflows/deploy.yaml: the pipeline.

What the pipeline manages

Role	Host	What
base	both	unattended upgrades, fail2ban, journal size cap, Docker daemon config, weekly image prune
traefik	ops	reverse proxy, TLS, dashboard on WireGuard
crawler_alert	ops	daily and weekly crawler report timers
forgejo	ops	Forgejo at git.asxp.io (git.czsk.it is an alias), SSH passthrough
forgejo_runner	services	Actions runner and its job labels
sillytavern	services	SillyTavern on the WireGuard address
minecraft	services	Paper server
xmpp	services	ejabberd, apushd and tackygram for asxp.io; secrets stay in /opt/xmpp/state
xray	services	Xray VLESS Reality on :443, config in the vault
restic_backup	both	nightly restic backup to object storage, hel1 and fsn1

Not managed here: the app stacks on asxpio-ops (asxpio, datacenterbingo, blog, czsk, vhpbot, storage). Each deploys from its own repository into /opt/<app>.

Changing something

1. Edit the role or variable, for example an image tag in ansible/inventory/group_vars/ or a compose file under ansible/roles/<role>/files/.
2. Push a branch. The pipeline does a dry run against production.
3. Merge to main. The pipeline dry-runs again, then applies.

A run with nothing to change touches nothing and restarts nothing.

Backups

Hetzner's server backups are off. Each host runs restic-backup.timer nightly (ops 03:30, services 04:15 UTC) into two independent repositories, restic/<host> in asxpio-backups-main (hel1) and asxpio-backups-repl (fsn1). Databases go in as dumps or online copies; see restic_pre_hooks in ansible/inventory/host_vars/. A failed run mails its journal.

restic-backup hel1 snapshots
restic-backup fsn1 restore latest --target /tmp/restore --include /opt/xmpp

The password is vault_restic_password in the vault. Keep a copy outside the servers and this repo's CI: without it nothing can be read.

Things to know:

• Game servers are saved and stopped with a 120 s timeout before a compose change recreates them.
• The runner executes the pipeline. A change to it recreates the container mid-job: the change lands, that job is reported as failed, and the next run confirms it.
• Forgejo restarts on a config change; the job log may pause for a few seconds.

To run it yourself, over WireGuard:

cd ansible
ansible-playbook site.yaml --check --diff      # dry run
ansible-playbook site.yaml --tags traefik      # one service

Secrets

Nothing secret is stored in the clear. Service secrets live in ansible/inventory/group_vars/all/vault.yaml, encrypted with Ansible Vault. The pipeline needs two repository secrets:

• ANSIBLE_VAULT_PASSWORD: decrypts the vault.
• SSH_DEPLOY_KEY: private key of user deploy on both servers.

Locally, put the vault password in ansible/.vault_pass (ignored by git).

Bootstrap (by hand)

ansible/playbooks/ holds what is done once per host or can lock you out: WireGuard, SSH hardening, s3fs mounts, swap, user creation. See ansible/README.md.