Sergei Poljanski
59b5b4468a
Hetzner's server backups cost 20% of each server's price and only restore whole disks. Back up what matters with restic instead, into the backup buckets we already pay for: one repository per host in hel1 (asxpio-backups-main) and one in fsn1 (asxpio-backups-repl), written independently so either region alone is a full copy. Postgres goes in as a dump, Forgejo's and vhpbot's SQLite databases through the online backup API, Garage with a fresh metadata snapshot, ejabberd through its own backup command. Keeps 7 daily, 4 weekly and 6 monthly snapshots, prunes on Sundays, and mails the journal when a run fails. restic is the pinned upstream release, so its version is tracked in the repo like everything else. The vault gains the restic password and the Xray config for the role that follows. Assisted-by: Claude:opus-5-5 |
||
|---|---|---|
| .. | ||
| inventory | ||
| playbooks | ||
| roles | ||
| .gitignore | ||
| ansible.cfg | ||
| known_hosts | ||
| README.md | ||
| site.yaml | ||
Ansible
site.yaml is the desired state of both servers and is applied by the
deploy pipeline on every push to main. The playbooks under
playbooks/ are host bootstrap and are run by hand.
Running
echo '<vault password>' > .vault_pass && chmod 600 .vault_pass
ansible-playbook site.yaml --check --diff # dry run, over WireGuard
ansible-playbook site.yaml # apply
ansible-playbook site.yaml --tags forgejo # one role
On NixOS: nix shell nixpkgs#ansible -c ansible-playbook ….
Templates that carry secrets are never shown in --diff output.
Inventories
inventory/hosts.yaml: default. Connects as root to the WireGuard addresses.inventory/ci.yaml: used by the pipeline. Connects asdeployover the private network, reaching asxpio-services through asxpio-ops.
known_hosts pins both servers' SSH host keys for the pipeline.
Variables
inventory/
├── group_vars/all/
│ ├── network.yaml # VPC and WireGuard addresses
│ ├── paths.yaml # /opt
│ ├── s3.yaml # object storage buckets and s3fs mounts
│ ├── wireguard.yaml # WireGuard peers
│ └── vault.yaml # encrypted secrets
├── group_vars/ops_servers/
│ ├── traefik.yaml # traefik_version
│ └── forgejo.yaml # forgejo_version, domain and aliases
├── group_vars/services_servers/
│ ├── forgejo_runner.yaml # runner version, labels' nix image
│ └── minecraft.yaml
└── host_vars/ # per-host addresses, SSH and s3fs settings
Compose files without variables are plain files under
roles/<role>/files/; edit the image tag there.
Roles in site.yaml
base, traefik, crawler_alert, forgejo, forgejo_runner,
sillytavern, minecraft, minecraft_fossils. Service roles go
through docker_service, which writes the files, runs
docker compose up -d, restarts on a config change and waits for the
service to answer.
crawler_alert installs a copy of
traefik-crawlers-statistics;
update roles/crawler_alert/files/ from there.
Bootstrap playbooks
| Playbook | Purpose |
|---|---|
01-wireguard-setup.yaml |
WireGuard hub on asxpio-ops, peer on asxpio-services, client configs |
02-ssh-hardening.yaml |
restrict SSH to WireGuard and the private network |
03-setup-s3fs.yaml |
s3fs mounts on asxpio-services (off on asxpio-ops) |
04-add-user-zerrtika.yaml |
add a user to running servers |
05-swap.yaml |
2 GB swapfile on asxpio-ops |
Always dry-run these first with --check --diff.
03-setup-s3fs.yamlrewrites/etc/s3fs/.passwd-s3fs, which the backup script on asxpio-ops also reads.- On asxpio-services the WireGuard interface is named
asxpio-services, notwg0. Check01-wireguard-setup.yamlagainst the host before re-running it there.
Runner notes
The nix-latest label runs nixos/nix:<forgejo_runner_nix_version>
with a nix-store-<version> volume over /nix. To bump it: change the
variable, let the pipeline apply it, then remove the old volume.