# asxp.io infrastructure

Two Hetzner Cloud servers, described by Terraform and configured by
Ansible. Pushing to `main` deploys: a Forgejo Actions workflow applies
`ansible/site.yaml` to both servers.

## Servers

| Server | Size | Runs |
|--------|------|------|
| asxpio-ops | 2 vCPU, 4 GB | Traefik, Forgejo, crawler-alert; app stacks deployed from their own repos |
| asxpio-services | 8 vCPU, 16 GB | Forgejo Actions runner, Minecraft (Paper), SillyTavern, XMPP (asxp.io), Xray |

Both are reachable for administration only over WireGuard
(10.20.0.0/24) or the private network (10.10.0.0/20).

## Layout

- `terraform/`: servers, network, firewalls, object storage buckets.
- `ansible/site.yaml`: everything the pipeline manages (see below).
- `ansible/playbooks/`: host bootstrap, run by hand.
- `.forgejo/workflows/deploy.yaml`: the pipeline.

## What the pipeline manages

| Role | Host | What |
|------|------|------|
| `base` | both | unattended upgrades, fail2ban, journal size cap, Docker daemon config, weekly image prune |
| `traefik` | ops | reverse proxy, TLS, dashboard on WireGuard |
| `crawler_alert` | ops | daily and weekly crawler report timers |
| `forgejo` | ops | Forgejo at git.asxp.io (git.czsk.it is an alias), SSH passthrough |
| `forgejo_runner` | services | Actions runner and its job labels |
| `sillytavern` | services | SillyTavern on the WireGuard address |
| `minecraft` | services | Paper server |
| `xmpp` | services | ejabberd, apushd and tackygram for asxp.io; secrets stay in `/opt/xmpp/state` |
| `xray` | services | Xray VLESS Reality on :443, config in the vault |
| `restic_backup` | both | nightly restic backup to object storage, hel1 and fsn1 |

Not managed here: the app stacks on asxpio-ops (asxpio, datacenterbingo,
blog, czsk, vhpbot, storage). Each deploys from its own repository into
`/opt/<app>`.

## Changing something

1. Edit the role or variable, for example an image tag in
   `ansible/inventory/group_vars/` or a compose file under
   `ansible/roles/<role>/files/`.
2. Push a branch. The pipeline does a dry run against production.
3. Merge to `main`. The pipeline dry-runs again, then applies.

A run with nothing to change touches nothing and restarts nothing.

## Backups

Hetzner's server backups are off. Each host runs `restic-backup.timer`
nightly (ops 03:30, services 04:15 UTC) into two independent
repositories, `restic/<host>` in `asxpio-backups-main` (hel1) and
`asxpio-backups-repl` (fsn1). Databases go in as dumps or online
copies; see `restic_pre_hooks` in `ansible/inventory/host_vars/`.
A failed run mails its journal.

```sh
restic-backup hel1 snapshots
restic-backup fsn1 restore latest --target /tmp/restore --include /opt/xmpp
```

The password is `vault_restic_password` in the vault. Keep a copy
outside the servers and this repo's CI: without it nothing can be read.

Things to know:

- **Game servers** are saved and stopped with a 120 s timeout before a
  compose change recreates them.
- **The runner** executes the pipeline. A change to it recreates the
  container mid-job: the change lands, that job is reported as failed,
  and the next run confirms it.
- **Forgejo** restarts on a config change; the job log may pause for a
  few seconds.

To run it yourself, over WireGuard:

```sh
cd ansible
ansible-playbook site.yaml --check --diff      # dry run
ansible-playbook site.yaml --tags traefik      # one service
```

## Secrets

Nothing secret is stored in the clear. Service secrets live in
`ansible/inventory/group_vars/all/vault.yaml`, encrypted with Ansible
Vault. The pipeline needs two repository secrets:

- `ANSIBLE_VAULT_PASSWORD`: decrypts the vault.
- `SSH_DEPLOY_KEY`: private key of user `deploy` on both servers.

Locally, put the vault password in `ansible/.vault_pass` (ignored by git).

## Bootstrap (by hand)

`ansible/playbooks/` holds what is done once per host or can lock you
out: WireGuard, SSH hardening, s3fs mounts, swap, user creation.
See `ansible/README.md`.