Sergei Poljanski
59415515a1
Describe the new roles, how to list and restore restic snapshots, and that the restic password must be kept outside the servers. Assisted-by: Claude:opus-5-5
104 lines
4 KiB
Markdown
104 lines
4 KiB
Markdown
# asxp.io infrastructure
|
|
|
|
Two Hetzner Cloud servers, described by Terraform and configured by
|
|
Ansible. Pushing to `main` deploys: a Forgejo Actions workflow applies
|
|
`ansible/site.yaml` to both servers.
|
|
|
|
## Servers
|
|
|
|
| Server | Size | Runs |
|
|
|--------|------|------|
|
|
| asxpio-ops | 2 vCPU, 4 GB | Traefik, Forgejo, crawler-alert; app stacks deployed from their own repos |
|
|
| asxpio-services | 8 vCPU, 16 GB | Forgejo Actions runner, Minecraft (Paper), SillyTavern, XMPP (asxp.io), Xray |
|
|
|
|
Both are reachable for administration only over WireGuard
|
|
(10.20.0.0/24) or the private network (10.10.0.0/20).
|
|
|
|
## Layout
|
|
|
|
- `terraform/`: servers, network, firewalls, object storage buckets.
|
|
- `ansible/site.yaml`: everything the pipeline manages (see below).
|
|
- `ansible/playbooks/`: host bootstrap, run by hand.
|
|
- `.forgejo/workflows/deploy.yaml`: the pipeline.
|
|
|
|
## What the pipeline manages
|
|
|
|
| Role | Host | What |
|
|
|------|------|------|
|
|
| `base` | both | unattended upgrades, fail2ban, journal size cap, Docker daemon config, weekly image prune |
|
|
| `traefik` | ops | reverse proxy, TLS, dashboard on WireGuard |
|
|
| `crawler_alert` | ops | daily and weekly crawler report timers |
|
|
| `forgejo` | ops | Forgejo at git.asxp.io (git.czsk.it is an alias), SSH passthrough |
|
|
| `forgejo_runner` | services | Actions runner and its job labels |
|
|
| `sillytavern` | services | SillyTavern on the WireGuard address |
|
|
| `minecraft` | services | Paper server |
|
|
| `minecraft_fossils` | services | retired: keeps the Fabric server stopped; world left in `/opt/minecraft-fossils/data` |
|
|
| `xmpp` | services | ejabberd, apushd and tackygram for asxp.io; secrets stay in `/opt/xmpp/state` |
|
|
| `xray` | services | Xray VLESS Reality on :443, config in the vault |
|
|
| `restic_backup` | both | nightly restic backup to object storage, hel1 and fsn1 |
|
|
|
|
Not managed here: the app stacks on asxpio-ops (asxpio, datacenterbingo,
|
|
blog, czsk, vhpbot, storage). Each deploys from its own repository into
|
|
`/opt/<app>`.
|
|
|
|
## Changing something
|
|
|
|
1. Edit the role or variable, for example an image tag in
|
|
`ansible/inventory/group_vars/` or a compose file under
|
|
`ansible/roles/<role>/files/`.
|
|
2. Push a branch. The pipeline does a dry run against production.
|
|
3. Merge to `main`. The pipeline dry-runs again, then applies.
|
|
|
|
A run with nothing to change touches nothing and restarts nothing.
|
|
|
|
## Backups
|
|
|
|
Hetzner's server backups are off. Each host runs `restic-backup.timer`
|
|
nightly (ops 03:30, services 04:15 UTC) into two independent
|
|
repositories, `restic/<host>` in `asxpio-backups-main` (hel1) and
|
|
`asxpio-backups-repl` (fsn1). Databases go in as dumps or online
|
|
copies; see `restic_pre_hooks` in `ansible/inventory/host_vars/`.
|
|
A failed run mails its journal.
|
|
|
|
```sh
|
|
restic-backup hel1 snapshots
|
|
restic-backup fsn1 restore latest --target /tmp/restore --include /opt/xmpp
|
|
```
|
|
|
|
The password is `vault_restic_password` in the vault. Keep a copy
|
|
outside the servers and this repo's CI: without it nothing can be read.
|
|
|
|
Things to know:
|
|
|
|
- **Game servers** are saved and stopped with a 120 s timeout before a
|
|
compose change recreates them.
|
|
- **The runner** executes the pipeline. A change to it recreates the
|
|
container mid-job: the change lands, that job is reported as failed,
|
|
and the next run confirms it.
|
|
- **Forgejo** restarts on a config change; the job log may pause for a
|
|
few seconds.
|
|
|
|
To run it yourself, over WireGuard:
|
|
|
|
```sh
|
|
cd ansible
|
|
ansible-playbook site.yaml --check --diff # dry run
|
|
ansible-playbook site.yaml --tags traefik # one service
|
|
```
|
|
|
|
## Secrets
|
|
|
|
Nothing secret is stored in the clear. Service secrets live in
|
|
`ansible/inventory/group_vars/all/vault.yaml`, encrypted with Ansible
|
|
Vault. The pipeline needs two repository secrets:
|
|
|
|
- `ANSIBLE_VAULT_PASSWORD`: decrypts the vault.
|
|
- `SSH_DEPLOY_KEY`: private key of user `deploy` on both servers.
|
|
|
|
Locally, put the vault password in `ansible/.vault_pass` (ignored by git).
|
|
|
|
## Bootstrap (by hand)
|
|
|
|
`ansible/playbooks/` holds what is done once per host or can lock you
|
|
out: WireGuard, SSH hardening, s3fs mounts, swap, user creation.
|
|
See `ansible/README.md`.
|