From e76d2a022c35db810dd61bfe6f25fd8a1344bda7 Mon Sep 17 00:00:00 2001 From: Sergei Poljanski Date: Thu, 12 Nov 2026 06:47:09 +0400 Subject: [PATCH] docs: describe the servers as they are The READMEs still tracked the 2025 migration from bare metal and listed Grafana, Immich, Valheim, Prosody and others that do not run. Replace them with what is deployed, how a change reaches production and where the secrets live. Assisted-by: Claude:fable-5-2 --- README.md | 111 +++++++++++++++++++++++++++----------------- ansible/README.md | 116 ++++++++++++++++++++++++++++------------------ 2 files changed, 139 insertions(+), 88 deletions(-) diff --git a/README.md b/README.md index d76fcae..8993dbc 100644 --- a/README.md +++ b/README.md @@ -1,58 +1,83 @@ -# asxp.io Infrastructure +# asxp.io infrastructure -Infrastructure as Code for asxp.io servers managed via Terraform, Ansible, and Forgejo CI/CD. +Two Hetzner Cloud servers, described by Terraform and configured by +Ansible. Pushing to `main` deploys: a Forgejo Actions workflow applies +`ansible/site.yaml` to both servers. ## Servers -| Server | Type | vCPU | RAM | Services | -|--------|------|------|-----|----------| -| asxpio-ops | CX22 | 2 | 4GB | traefik, forgejo, grafana, prometheus, loki, static_sites | -| asxpio-services | CX43 | 8 | 16GB | forgejo_runner, immich, minecraft, valheim, factorio, prosody, coturn | +| Server | Size | Runs | +|--------|------|------| +| asxpio-ops | 2 vCPU, 4 GB | Traefik, Forgejo, crawler-alert; app stacks deployed from their own repos | +| asxpio-services | 8 vCPU, 16 GB | Forgejo Actions runner, Minecraft (Paper and Fossils), SillyTavern | -## Directory Structure +Both are reachable for administration only over WireGuard +(10.20.0.0/24) or the private network (10.10.0.0/20). -- **terraform/** - Cloud infrastructure provisioning (Hetzner Cloud) -- **ansible/** - Server bootstrap and configuration management -- **services/** - Docker Compose files for container orchestration -- **.forgejo/workflows/** - CI/CD pipeline automation (planned) +## Layout -## Migration TODO +- `terraform/`: servers, network, firewalls, object storage buckets. +- `ansible/site.yaml`: everything the pipeline manages (see below). +- `ansible/playbooks/`: host bootstrap, run by hand. +- `.forgejo/workflows/deploy.yaml`: the pipeline. -### Phase 0: Preparation -- [x] Terraform infrastructure code -- [x] Ansible inventory and playbooks -- [x] Create S3 buckets (hel1-primary, fsn1-secondary) -- [x] Provision servers via Terraform -- [x] Backup all services from bare metal +## What the pipeline manages -### Phase 1: Core Infrastructure -- [x] Deploy WireGuard VPN on ops-server -- [x] Deploy Traefik reverse proxy -- [ ] Deploy monitoring stack (Grafana, Prometheus, Loki) +| Role | Host | What | +|------|------|------| +| `base` | both | unattended upgrades, fail2ban, journal size cap, Docker daemon config, weekly image prune | +| `traefik` | ops | reverse proxy, TLS, dashboard on WireGuard | +| `crawler_alert` | ops | daily and weekly crawler report timers | +| `forgejo` | ops | Forgejo at git.asxp.io (git.czsk.it is an alias), SSH passthrough | +| `forgejo_runner` | services | Actions runner and its job labels | +| `sillytavern` | services | SillyTavern on the WireGuard address | +| `minecraft`, `minecraft_fossils` | services | game servers, daily restart timer | -### Phase 2: Git & CI/CD -- [x] Migrate Forgejo from bare metal -- [x] Deploy Forgejo Runner on services-server +Not managed here: the app stacks on asxpio-ops (asxpio, datacenterbingo, +blog, czsk, vhpbot, storage). Each deploys from its own repository into +`/opt/`. -### Phase 3: Media & Storage -- [x] Upload Immich photos to S3 (135GB) -- [x] Migrate Immich to services-server -- [/] Configure S3 geo-replication (hel1 → fsn1) (Turn on service) +## Changing something -### Phase 4: Gaming Services -- [ ] Migrate Minecraft (20GB local + 80GB S3 backups) -- [ ] Migrate Valheim (9.5GB) -- [ ] Migrate Factorio (31MB) +1. Edit the role or variable, for example an image tag in + `ansible/inventory/group_vars/` or a compose file under + `ansible/roles//files/`. +2. Push a branch. The pipeline does a dry run against production. +3. Merge to `main`. The pipeline dry-runs again, then applies. -### Phase 5: Communication -- [ ] Migrate Prosody XMPP -- [ ] Migrate Turnserver +A run with nothing to change touches nothing and restarts nothing. -### Phase 6: Final Services -- [ ] Migrate static sites (asxp.io, czsk.it) -- [ ] Configure automated backups to S3 -- [x] DNS cutover and validation -- [x] Decommission bare metal (after 30 days) +Things to know: -### Later -- [ ] Setup Terraform to manage DNS +- **Game servers** are saved and stopped with a 120 s timeout before a + compose change recreates them. +- **The runner** executes the pipeline. A change to it recreates the + container mid-job: the change lands, that job is reported as failed, + and the next run confirms it. +- **Forgejo** restarts on a config change; the job log may pause for a + few seconds. + +To run it yourself, over WireGuard: + +```sh +cd ansible +ansible-playbook site.yaml --check --diff # dry run +ansible-playbook site.yaml --tags traefik # one service +``` + +## Secrets + +Nothing secret is stored in the clear. Service secrets live in +`ansible/inventory/group_vars/all/vault.yaml`, encrypted with Ansible +Vault. The pipeline needs two repository secrets: + +- `ANSIBLE_VAULT_PASSWORD`: decrypts the vault. +- `SSH_DEPLOY_KEY`: private key of user `deploy` on both servers. + +Locally, put the vault password in `ansible/.vault_pass` (ignored by git). + +## Bootstrap (by hand) + +`ansible/playbooks/` holds what is done once per host or can lock you +out: WireGuard, SSH hardening, s3fs mounts, swap, user creation. +See `ansible/README.md`. diff --git a/ansible/README.md b/ansible/README.md index 62d4e34..e581645 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -1,60 +1,86 @@ -# ASXP.IO Ansible Infrastructure +# Ansible -Automated deployment for asxpio servers: WireGuard VPN, SSH hardening, Traefik, Forgejo, and S3 storage. +`site.yaml` is the desired state of both servers and is applied by the +deploy pipeline on every push to `main`. The playbooks under +`playbooks/` are host bootstrap and are run by hand. -## Quick Start +## Running -```bash -# 1. Setup vault -echo "your-password" > .vault_pass && chmod 600 .vault_pass -./scripts/setup-vault.sh encrypt +```sh +echo '' > .vault_pass && chmod 600 .vault_pass -# 2. Stage 1: WireGuard VPN -ansible-playbook playbooks/01-wireguard-setup.yaml -# Connect to VPN, update inventory/hosts.yaml to use WireGuard IPs - -# 3. Stage 2: Full deployment (SSH hardening, Traefik, Forgejo, Runner) -ansible-playbook playbooks/02-full-deployment.yaml - -# 4. Stage 3: S3 storage mounts -ansible-playbook playbooks/03-setup-s3fs.yaml +ansible-playbook site.yaml --check --diff # dry run, over WireGuard +ansible-playbook site.yaml # apply +ansible-playbook site.yaml --tags forgejo # one role ``` -## Infrastructure +On NixOS: `nix shell nixpkgs#ansible -c ansible-playbook …`. -**Servers:** -- **asxpio-ops** (CX22): Forgejo, Traefik, WireGuard VPN hub -- **asxpio-services** (CX43): Forgejo Runner, Immich, game servers +Templates that carry secrets are never shown in `--diff` output. -**Networks:** -- VPC: 10.10.0.0/20 -- WireGuard: 10.20.0.0/24 +## Inventories -**S3 Buckets (mounted via s3fs):** -- asxpio-terraform-main → /mnt/s3/terraform -- asxpio-backups-main → /mnt/s3/backups -- asxpio-immich-main → /mnt/s3/immich +- `inventory/hosts.yaml`: default. Connects as root to the WireGuard + addresses. +- `inventory/ci.yaml`: used by the pipeline. Connects as `deploy` over + the private network, reaching asxpio-services through asxpio-ops. -## Playbook Workflow +`known_hosts` pins both servers' SSH host keys for the pipeline. -1. **WireGuard Setup** - Deploy VPN, generate client configs -2. **Full Deployment** - SSH hardening (no lockout), Traefik, Forgejo, Runner -3. **S3 Storage** - Mount S3 buckets with automatic cache management - -All playbooks are idempotent - safe to rerun without breaking existing setup. - -## Configuration Files +## Variables ``` inventory/ -├── hosts.yaml # Server inventory, IPs, groups -├── group_vars/all/ # Variables applied to all hosts -│ ├── main.yaml # Forgejo version, ports, runner config -│ ├── network.yaml # VPC (10.10.0.0/20), WireGuard (10.20.0.0/24) IPs -│ ├── services.yaml # Service paths (/opt/*), ports, Docker networks -│ ├── s3.yaml # S3 buckets, mount points, credentials reference -│ └── vault.yaml # ENCRYPTED: S3 keys, Forgejo secrets, runner token -└── host_vars/ # Host-specific overrides - ├── asxpio-ops.yaml # WireGuard server, clients list, S3 cache: 10GB - └── asxpio-services.yaml # S3 cache: 80GB +├── group_vars/all/ +│ ├── network.yaml # VPC and WireGuard addresses +│ ├── paths.yaml # /opt +│ ├── s3.yaml # object storage buckets and s3fs mounts +│ ├── wireguard.yaml # WireGuard peers +│ └── vault.yaml # encrypted secrets +├── group_vars/ops_servers/ +│ ├── traefik.yaml # traefik_version +│ └── forgejo.yaml # forgejo_version, domain and aliases +├── group_vars/services_servers/ +│ ├── forgejo_runner.yaml # runner version, labels' nix image +│ └── minecraft.yaml +└── host_vars/ # per-host addresses, SSH and s3fs settings ``` + +Compose files without variables are plain files under +`roles//files/`; edit the image tag there. + +## Roles in site.yaml + +`base`, `traefik`, `crawler_alert`, `forgejo`, `forgejo_runner`, +`sillytavern`, `minecraft`, `minecraft_fossils`. Service roles go +through `docker_service`, which writes the files, runs +`docker compose up -d`, restarts on a config change and waits for the +service to answer. + +`crawler_alert` installs a copy of +[traefik-crawlers-statistics](https://git.asxp.io/asxpi/traefik-crawlers-statistics); +update `roles/crawler_alert/files/` from there. + +## Bootstrap playbooks + +| Playbook | Purpose | +|----------|---------| +| `01-wireguard-setup.yaml` | WireGuard hub on asxpio-ops, peer on asxpio-services, client configs | +| `02-ssh-hardening.yaml` | restrict SSH to WireGuard and the private network | +| `03-setup-s3fs.yaml` | s3fs mounts on asxpio-services (off on asxpio-ops) | +| `04-add-user-zerrtika.yaml` | add a user to running servers | +| `05-swap.yaml` | 2 GB swapfile on asxpio-ops | + +Always dry-run these first with `--check --diff`. + +- `03-setup-s3fs.yaml` rewrites `/etc/s3fs/.passwd-s3fs`, which the + backup script on asxpio-ops also reads. +- On asxpio-services the WireGuard interface is named + `asxpio-services`, not `wg0`. Check `01-wireguard-setup.yaml` against + the host before re-running it there. + +## Runner notes + +The `nix-latest` label runs `nixos/nix:` +with a `nix-store-` volume over `/nix`. To bump it: change the +variable, let the pipeline apply it, then remove the old volume.