docs: describe the servers as they are

The READMEs still tracked the 2025 migration from bare metal and listed
Grafana, Immich, Valheim, Prosody and others that do not run. Replace
them with what is deployed, how a change reaches production and where
the secrets live.

Assisted-by: Claude:fable-5-2
This commit is contained in:
Sergei Poljanski 2026-11-12 06:47:09 +04:00
commit e76d2a022c
2 changed files with 149 additions and 98 deletions

111
README.md
View file

@ -1,58 +1,83 @@
# asxp.io Infrastructure
# asxp.io infrastructure
Infrastructure as Code for asxp.io servers managed via Terraform, Ansible, and Forgejo CI/CD.
Two Hetzner Cloud servers, described by Terraform and configured by
Ansible. Pushing to `main` deploys: a Forgejo Actions workflow applies
`ansible/site.yaml` to both servers.
## Servers
| Server | Type | vCPU | RAM | Services |
|--------|------|------|-----|----------|
| asxpio-ops | CX22 | 2 | 4GB | traefik, forgejo, grafana, prometheus, loki, static_sites |
| asxpio-services | CX43 | 8 | 16GB | forgejo_runner, immich, minecraft, valheim, factorio, prosody, coturn |
| Server | Size | Runs |
|--------|------|------|
| asxpio-ops | 2 vCPU, 4 GB | Traefik, Forgejo, crawler-alert; app stacks deployed from their own repos |
| asxpio-services | 8 vCPU, 16 GB | Forgejo Actions runner, Minecraft (Paper and Fossils), SillyTavern |
## Directory Structure
Both are reachable for administration only over WireGuard
(10.20.0.0/24) or the private network (10.10.0.0/20).
- **terraform/** - Cloud infrastructure provisioning (Hetzner Cloud)
- **ansible/** - Server bootstrap and configuration management
- **services/** - Docker Compose files for container orchestration
- **.forgejo/workflows/** - CI/CD pipeline automation (planned)
## Layout
## Migration TODO
- `terraform/`: servers, network, firewalls, object storage buckets.
- `ansible/site.yaml`: everything the pipeline manages (see below).
- `ansible/playbooks/`: host bootstrap, run by hand.
- `.forgejo/workflows/deploy.yaml`: the pipeline.
### Phase 0: Preparation
- [x] Terraform infrastructure code
- [x] Ansible inventory and playbooks
- [x] Create S3 buckets (hel1-primary, fsn1-secondary)
- [x] Provision servers via Terraform
- [x] Backup all services from bare metal
## What the pipeline manages
### Phase 1: Core Infrastructure
- [x] Deploy WireGuard VPN on ops-server
- [x] Deploy Traefik reverse proxy
- [ ] Deploy monitoring stack (Grafana, Prometheus, Loki)
| Role | Host | What |
|------|------|------|
| `base` | both | unattended upgrades, fail2ban, journal size cap, Docker daemon config, weekly image prune |
| `traefik` | ops | reverse proxy, TLS, dashboard on WireGuard |
| `crawler_alert` | ops | daily and weekly crawler report timers |
| `forgejo` | ops | Forgejo at git.asxp.io (git.czsk.it is an alias), SSH passthrough |
| `forgejo_runner` | services | Actions runner and its job labels |
| `sillytavern` | services | SillyTavern on the WireGuard address |
| `minecraft`, `minecraft_fossils` | services | game servers, daily restart timer |
### Phase 2: Git & CI/CD
- [x] Migrate Forgejo from bare metal
- [x] Deploy Forgejo Runner on services-server
Not managed here: the app stacks on asxpio-ops (asxpio, datacenterbingo,
blog, czsk, vhpbot, storage). Each deploys from its own repository into
`/opt/<app>`.
### Phase 3: Media & Storage
- [x] Upload Immich photos to S3 (135GB)
- [x] Migrate Immich to services-server
- [/] Configure S3 geo-replication (hel1 → fsn1) (Turn on service)
## Changing something
### Phase 4: Gaming Services
- [ ] Migrate Minecraft (20GB local + 80GB S3 backups)
- [ ] Migrate Valheim (9.5GB)
- [ ] Migrate Factorio (31MB)
1. Edit the role or variable, for example an image tag in
`ansible/inventory/group_vars/` or a compose file under
`ansible/roles/<role>/files/`.
2. Push a branch. The pipeline does a dry run against production.
3. Merge to `main`. The pipeline dry-runs again, then applies.
### Phase 5: Communication
- [ ] Migrate Prosody XMPP
- [ ] Migrate Turnserver
A run with nothing to change touches nothing and restarts nothing.
### Phase 6: Final Services
- [ ] Migrate static sites (asxp.io, czsk.it)
- [ ] Configure automated backups to S3
- [x] DNS cutover and validation
- [x] Decommission bare metal (after 30 days)
Things to know:
### Later
- [ ] Setup Terraform to manage DNS
- **Game servers** are saved and stopped with a 120 s timeout before a
compose change recreates them.
- **The runner** executes the pipeline. A change to it recreates the
container mid-job: the change lands, that job is reported as failed,
and the next run confirms it.
- **Forgejo** restarts on a config change; the job log may pause for a
few seconds.
To run it yourself, over WireGuard:
```sh
cd ansible
ansible-playbook site.yaml --check --diff # dry run
ansible-playbook site.yaml --tags traefik # one service
```
## Secrets
Nothing secret is stored in the clear. Service secrets live in
`ansible/inventory/group_vars/all/vault.yaml`, encrypted with Ansible
Vault. The pipeline needs two repository secrets:
- `ANSIBLE_VAULT_PASSWORD`: decrypts the vault.
- `SSH_DEPLOY_KEY`: private key of user `deploy` on both servers.
Locally, put the vault password in `ansible/.vault_pass` (ignored by git).
## Bootstrap (by hand)
`ansible/playbooks/` holds what is done once per host or can lock you
out: WireGuard, SSH hardening, s3fs mounts, swap, user creation.
See `ansible/README.md`.