diff --git a/README.md b/README.md index d76fcae..8993dbc 100644 --- a/README.md +++ b/README.md @@ -1,58 +1,83 @@ -# asxp.io Infrastructure +# asxp.io infrastructure -Infrastructure as Code for asxp.io servers managed via Terraform, Ansible, and Forgejo CI/CD. +Two Hetzner Cloud servers, described by Terraform and configured by +Ansible. Pushing to `main` deploys: a Forgejo Actions workflow applies +`ansible/site.yaml` to both servers. ## Servers -| Server | Type | vCPU | RAM | Services | -|--------|------|------|-----|----------| -| asxpio-ops | CX22 | 2 | 4GB | traefik, forgejo, grafana, prometheus, loki, static_sites | -| asxpio-services | CX43 | 8 | 16GB | forgejo_runner, immich, minecraft, valheim, factorio, prosody, coturn | +| Server | Size | Runs | +|--------|------|------| +| asxpio-ops | 2 vCPU, 4 GB | Traefik, Forgejo, crawler-alert; app stacks deployed from their own repos | +| asxpio-services | 8 vCPU, 16 GB | Forgejo Actions runner, Minecraft (Paper and Fossils), SillyTavern | -## Directory Structure +Both are reachable for administration only over WireGuard +(10.20.0.0/24) or the private network (10.10.0.0/20). -- **terraform/** - Cloud infrastructure provisioning (Hetzner Cloud) -- **ansible/** - Server bootstrap and configuration management -- **services/** - Docker Compose files for container orchestration -- **.forgejo/workflows/** - CI/CD pipeline automation (planned) +## Layout -## Migration TODO +- `terraform/`: servers, network, firewalls, object storage buckets. +- `ansible/site.yaml`: everything the pipeline manages (see below). +- `ansible/playbooks/`: host bootstrap, run by hand. +- `.forgejo/workflows/deploy.yaml`: the pipeline. -### Phase 0: Preparation -- [x] Terraform infrastructure code -- [x] Ansible inventory and playbooks -- [x] Create S3 buckets (hel1-primary, fsn1-secondary) -- [x] Provision servers via Terraform -- [x] Backup all services from bare metal +## What the pipeline manages -### Phase 1: Core Infrastructure -- [x] Deploy WireGuard VPN on ops-server -- [x] Deploy Traefik reverse proxy -- [ ] Deploy monitoring stack (Grafana, Prometheus, Loki) +| Role | Host | What | +|------|------|------| +| `base` | both | unattended upgrades, fail2ban, journal size cap, Docker daemon config, weekly image prune | +| `traefik` | ops | reverse proxy, TLS, dashboard on WireGuard | +| `crawler_alert` | ops | daily and weekly crawler report timers | +| `forgejo` | ops | Forgejo at git.asxp.io (git.czsk.it is an alias), SSH passthrough | +| `forgejo_runner` | services | Actions runner and its job labels | +| `sillytavern` | services | SillyTavern on the WireGuard address | +| `minecraft`, `minecraft_fossils` | services | game servers, daily restart timer | -### Phase 2: Git & CI/CD -- [x] Migrate Forgejo from bare metal -- [x] Deploy Forgejo Runner on services-server +Not managed here: the app stacks on asxpio-ops (asxpio, datacenterbingo, +blog, czsk, vhpbot, storage). Each deploys from its own repository into +`/opt/`. -### Phase 3: Media & Storage -- [x] Upload Immich photos to S3 (135GB) -- [x] Migrate Immich to services-server -- [/] Configure S3 geo-replication (hel1 → fsn1) (Turn on service) +## Changing something -### Phase 4: Gaming Services -- [ ] Migrate Minecraft (20GB local + 80GB S3 backups) -- [ ] Migrate Valheim (9.5GB) -- [ ] Migrate Factorio (31MB) +1. Edit the role or variable, for example an image tag in + `ansible/inventory/group_vars/` or a compose file under + `ansible/roles//files/`. +2. Push a branch. The pipeline does a dry run against production. +3. Merge to `main`. The pipeline dry-runs again, then applies. -### Phase 5: Communication -- [ ] Migrate Prosody XMPP -- [ ] Migrate Turnserver +A run with nothing to change touches nothing and restarts nothing. -### Phase 6: Final Services -- [ ] Migrate static sites (asxp.io, czsk.it) -- [ ] Configure automated backups to S3 -- [x] DNS cutover and validation -- [x] Decommission bare metal (after 30 days) +Things to know: -### Later -- [ ] Setup Terraform to manage DNS +- **Game servers** are saved and stopped with a 120 s timeout before a + compose change recreates them. +- **The runner** executes the pipeline. A change to it recreates the + container mid-job: the change lands, that job is reported as failed, + and the next run confirms it. +- **Forgejo** restarts on a config change; the job log may pause for a + few seconds. + +To run it yourself, over WireGuard: + +```sh +cd ansible +ansible-playbook site.yaml --check --diff # dry run +ansible-playbook site.yaml --tags traefik # one service +``` + +## Secrets + +Nothing secret is stored in the clear. Service secrets live in +`ansible/inventory/group_vars/all/vault.yaml`, encrypted with Ansible +Vault. The pipeline needs two repository secrets: + +- `ANSIBLE_VAULT_PASSWORD`: decrypts the vault. +- `SSH_DEPLOY_KEY`: private key of user `deploy` on both servers. + +Locally, put the vault password in `ansible/.vault_pass` (ignored by git). + +## Bootstrap (by hand) + +`ansible/playbooks/` holds what is done once per host or can lock you +out: WireGuard, SSH hardening, s3fs mounts, swap, user creation. +See `ansible/README.md`. diff --git a/ansible/README.md b/ansible/README.md index 62d4e34..e581645 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -1,60 +1,86 @@ -# ASXP.IO Ansible Infrastructure +# Ansible -Automated deployment for asxpio servers: WireGuard VPN, SSH hardening, Traefik, Forgejo, and S3 storage. +`site.yaml` is the desired state of both servers and is applied by the +deploy pipeline on every push to `main`. The playbooks under +`playbooks/` are host bootstrap and are run by hand. -## Quick Start +## Running -```bash -# 1. Setup vault -echo "your-password" > .vault_pass && chmod 600 .vault_pass -./scripts/setup-vault.sh encrypt +```sh +echo '' > .vault_pass && chmod 600 .vault_pass -# 2. Stage 1: WireGuard VPN -ansible-playbook playbooks/01-wireguard-setup.yaml -# Connect to VPN, update inventory/hosts.yaml to use WireGuard IPs - -# 3. Stage 2: Full deployment (SSH hardening, Traefik, Forgejo, Runner) -ansible-playbook playbooks/02-full-deployment.yaml - -# 4. Stage 3: S3 storage mounts -ansible-playbook playbooks/03-setup-s3fs.yaml +ansible-playbook site.yaml --check --diff # dry run, over WireGuard +ansible-playbook site.yaml # apply +ansible-playbook site.yaml --tags forgejo # one role ``` -## Infrastructure +On NixOS: `nix shell nixpkgs#ansible -c ansible-playbook …`. -**Servers:** -- **asxpio-ops** (CX22): Forgejo, Traefik, WireGuard VPN hub -- **asxpio-services** (CX43): Forgejo Runner, Immich, game servers +Templates that carry secrets are never shown in `--diff` output. -**Networks:** -- VPC: 10.10.0.0/20 -- WireGuard: 10.20.0.0/24 +## Inventories -**S3 Buckets (mounted via s3fs):** -- asxpio-terraform-main → /mnt/s3/terraform -- asxpio-backups-main → /mnt/s3/backups -- asxpio-immich-main → /mnt/s3/immich +- `inventory/hosts.yaml`: default. Connects as root to the WireGuard + addresses. +- `inventory/ci.yaml`: used by the pipeline. Connects as `deploy` over + the private network, reaching asxpio-services through asxpio-ops. -## Playbook Workflow +`known_hosts` pins both servers' SSH host keys for the pipeline. -1. **WireGuard Setup** - Deploy VPN, generate client configs -2. **Full Deployment** - SSH hardening (no lockout), Traefik, Forgejo, Runner -3. **S3 Storage** - Mount S3 buckets with automatic cache management - -All playbooks are idempotent - safe to rerun without breaking existing setup. - -## Configuration Files +## Variables ``` inventory/ -├── hosts.yaml # Server inventory, IPs, groups -├── group_vars/all/ # Variables applied to all hosts -│ ├── main.yaml # Forgejo version, ports, runner config -│ ├── network.yaml # VPC (10.10.0.0/20), WireGuard (10.20.0.0/24) IPs -│ ├── services.yaml # Service paths (/opt/*), ports, Docker networks -│ ├── s3.yaml # S3 buckets, mount points, credentials reference -│ └── vault.yaml # ENCRYPTED: S3 keys, Forgejo secrets, runner token -└── host_vars/ # Host-specific overrides - ├── asxpio-ops.yaml # WireGuard server, clients list, S3 cache: 10GB - └── asxpio-services.yaml # S3 cache: 80GB +├── group_vars/all/ +│ ├── network.yaml # VPC and WireGuard addresses +│ ├── paths.yaml # /opt +│ ├── s3.yaml # object storage buckets and s3fs mounts +│ ├── wireguard.yaml # WireGuard peers +│ └── vault.yaml # encrypted secrets +├── group_vars/ops_servers/ +│ ├── traefik.yaml # traefik_version +│ └── forgejo.yaml # forgejo_version, domain and aliases +├── group_vars/services_servers/ +│ ├── forgejo_runner.yaml # runner version, labels' nix image +│ └── minecraft.yaml +└── host_vars/ # per-host addresses, SSH and s3fs settings ``` + +Compose files without variables are plain files under +`roles//files/`; edit the image tag there. + +## Roles in site.yaml + +`base`, `traefik`, `crawler_alert`, `forgejo`, `forgejo_runner`, +`sillytavern`, `minecraft`, `minecraft_fossils`. Service roles go +through `docker_service`, which writes the files, runs +`docker compose up -d`, restarts on a config change and waits for the +service to answer. + +`crawler_alert` installs a copy of +[traefik-crawlers-statistics](https://git.asxp.io/asxpi/traefik-crawlers-statistics); +update `roles/crawler_alert/files/` from there. + +## Bootstrap playbooks + +| Playbook | Purpose | +|----------|---------| +| `01-wireguard-setup.yaml` | WireGuard hub on asxpio-ops, peer on asxpio-services, client configs | +| `02-ssh-hardening.yaml` | restrict SSH to WireGuard and the private network | +| `03-setup-s3fs.yaml` | s3fs mounts on asxpio-services (off on asxpio-ops) | +| `04-add-user-zerrtika.yaml` | add a user to running servers | +| `05-swap.yaml` | 2 GB swapfile on asxpio-ops | + +Always dry-run these first with `--check --diff`. + +- `03-setup-s3fs.yaml` rewrites `/etc/s3fs/.passwd-s3fs`, which the + backup script on asxpio-ops also reads. +- On asxpio-services the WireGuard interface is named + `asxpio-services`, not `wg0`. Check `01-wireguard-setup.yaml` against + the host before re-running it there. + +## Runner notes + +The `nix-latest` label runs `nixos/nix:` +with a `nix-store-` volume over `/nix`. To bump it: change the +variable, let the pipeline apply it, then remove the old volume.