docs: describe the servers as they are
The READMEs still tracked the 2025 migration from bare metal and listed Grafana, Immich, Valheim, Prosody and others that do not run. Replace them with what is deployed, how a change reaches production and where the secrets live. Assisted-by: Claude:fable-5-2
This commit is contained in:
parent
63bd67a86d
commit
e76d2a022c
2 changed files with 149 additions and 98 deletions
111
README.md
111
README.md
|
|
@ -1,58 +1,83 @@
|
||||||
# asxp.io Infrastructure
|
# asxp.io infrastructure
|
||||||
|
|
||||||
Infrastructure as Code for asxp.io servers managed via Terraform, Ansible, and Forgejo CI/CD.
|
Two Hetzner Cloud servers, described by Terraform and configured by
|
||||||
|
Ansible. Pushing to `main` deploys: a Forgejo Actions workflow applies
|
||||||
|
`ansible/site.yaml` to both servers.
|
||||||
|
|
||||||
## Servers
|
## Servers
|
||||||
|
|
||||||
| Server | Type | vCPU | RAM | Services |
|
| Server | Size | Runs |
|
||||||
|--------|------|------|-----|----------|
|
|--------|------|------|
|
||||||
| asxpio-ops | CX22 | 2 | 4GB | traefik, forgejo, grafana, prometheus, loki, static_sites |
|
| asxpio-ops | 2 vCPU, 4 GB | Traefik, Forgejo, crawler-alert; app stacks deployed from their own repos |
|
||||||
| asxpio-services | CX43 | 8 | 16GB | forgejo_runner, immich, minecraft, valheim, factorio, prosody, coturn |
|
| asxpio-services | 8 vCPU, 16 GB | Forgejo Actions runner, Minecraft (Paper and Fossils), SillyTavern |
|
||||||
|
|
||||||
## Directory Structure
|
Both are reachable for administration only over WireGuard
|
||||||
|
(10.20.0.0/24) or the private network (10.10.0.0/20).
|
||||||
|
|
||||||
- **terraform/** - Cloud infrastructure provisioning (Hetzner Cloud)
|
## Layout
|
||||||
- **ansible/** - Server bootstrap and configuration management
|
|
||||||
- **services/** - Docker Compose files for container orchestration
|
|
||||||
- **.forgejo/workflows/** - CI/CD pipeline automation (planned)
|
|
||||||
|
|
||||||
## Migration TODO
|
- `terraform/`: servers, network, firewalls, object storage buckets.
|
||||||
|
- `ansible/site.yaml`: everything the pipeline manages (see below).
|
||||||
|
- `ansible/playbooks/`: host bootstrap, run by hand.
|
||||||
|
- `.forgejo/workflows/deploy.yaml`: the pipeline.
|
||||||
|
|
||||||
### Phase 0: Preparation
|
## What the pipeline manages
|
||||||
- [x] Terraform infrastructure code
|
|
||||||
- [x] Ansible inventory and playbooks
|
|
||||||
- [x] Create S3 buckets (hel1-primary, fsn1-secondary)
|
|
||||||
- [x] Provision servers via Terraform
|
|
||||||
- [x] Backup all services from bare metal
|
|
||||||
|
|
||||||
### Phase 1: Core Infrastructure
|
| Role | Host | What |
|
||||||
- [x] Deploy WireGuard VPN on ops-server
|
|------|------|------|
|
||||||
- [x] Deploy Traefik reverse proxy
|
| `base` | both | unattended upgrades, fail2ban, journal size cap, Docker daemon config, weekly image prune |
|
||||||
- [ ] Deploy monitoring stack (Grafana, Prometheus, Loki)
|
| `traefik` | ops | reverse proxy, TLS, dashboard on WireGuard |
|
||||||
|
| `crawler_alert` | ops | daily and weekly crawler report timers |
|
||||||
|
| `forgejo` | ops | Forgejo at git.asxp.io (git.czsk.it is an alias), SSH passthrough |
|
||||||
|
| `forgejo_runner` | services | Actions runner and its job labels |
|
||||||
|
| `sillytavern` | services | SillyTavern on the WireGuard address |
|
||||||
|
| `minecraft`, `minecraft_fossils` | services | game servers, daily restart timer |
|
||||||
|
|
||||||
### Phase 2: Git & CI/CD
|
Not managed here: the app stacks on asxpio-ops (asxpio, datacenterbingo,
|
||||||
- [x] Migrate Forgejo from bare metal
|
blog, czsk, vhpbot, storage). Each deploys from its own repository into
|
||||||
- [x] Deploy Forgejo Runner on services-server
|
`/opt/<app>`.
|
||||||
|
|
||||||
### Phase 3: Media & Storage
|
## Changing something
|
||||||
- [x] Upload Immich photos to S3 (135GB)
|
|
||||||
- [x] Migrate Immich to services-server
|
|
||||||
- [/] Configure S3 geo-replication (hel1 → fsn1) (Turn on service)
|
|
||||||
|
|
||||||
### Phase 4: Gaming Services
|
1. Edit the role or variable, for example an image tag in
|
||||||
- [ ] Migrate Minecraft (20GB local + 80GB S3 backups)
|
`ansible/inventory/group_vars/` or a compose file under
|
||||||
- [ ] Migrate Valheim (9.5GB)
|
`ansible/roles/<role>/files/`.
|
||||||
- [ ] Migrate Factorio (31MB)
|
2. Push a branch. The pipeline does a dry run against production.
|
||||||
|
3. Merge to `main`. The pipeline dry-runs again, then applies.
|
||||||
|
|
||||||
### Phase 5: Communication
|
A run with nothing to change touches nothing and restarts nothing.
|
||||||
- [ ] Migrate Prosody XMPP
|
|
||||||
- [ ] Migrate Turnserver
|
|
||||||
|
|
||||||
### Phase 6: Final Services
|
Things to know:
|
||||||
- [ ] Migrate static sites (asxp.io, czsk.it)
|
|
||||||
- [ ] Configure automated backups to S3
|
|
||||||
- [x] DNS cutover and validation
|
|
||||||
- [x] Decommission bare metal (after 30 days)
|
|
||||||
|
|
||||||
### Later
|
- **Game servers** are saved and stopped with a 120 s timeout before a
|
||||||
- [ ] Setup Terraform to manage DNS
|
compose change recreates them.
|
||||||
|
- **The runner** executes the pipeline. A change to it recreates the
|
||||||
|
container mid-job: the change lands, that job is reported as failed,
|
||||||
|
and the next run confirms it.
|
||||||
|
- **Forgejo** restarts on a config change; the job log may pause for a
|
||||||
|
few seconds.
|
||||||
|
|
||||||
|
To run it yourself, over WireGuard:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cd ansible
|
||||||
|
ansible-playbook site.yaml --check --diff # dry run
|
||||||
|
ansible-playbook site.yaml --tags traefik # one service
|
||||||
|
```
|
||||||
|
|
||||||
|
## Secrets
|
||||||
|
|
||||||
|
Nothing secret is stored in the clear. Service secrets live in
|
||||||
|
`ansible/inventory/group_vars/all/vault.yaml`, encrypted with Ansible
|
||||||
|
Vault. The pipeline needs two repository secrets:
|
||||||
|
|
||||||
|
- `ANSIBLE_VAULT_PASSWORD`: decrypts the vault.
|
||||||
|
- `SSH_DEPLOY_KEY`: private key of user `deploy` on both servers.
|
||||||
|
|
||||||
|
Locally, put the vault password in `ansible/.vault_pass` (ignored by git).
|
||||||
|
|
||||||
|
## Bootstrap (by hand)
|
||||||
|
|
||||||
|
`ansible/playbooks/` holds what is done once per host or can lock you
|
||||||
|
out: WireGuard, SSH hardening, s3fs mounts, swap, user creation.
|
||||||
|
See `ansible/README.md`.
|
||||||
|
|
|
||||||
|
|
@ -1,60 +1,86 @@
|
||||||
# ASXP.IO Ansible Infrastructure
|
# Ansible
|
||||||
|
|
||||||
Automated deployment for asxpio servers: WireGuard VPN, SSH hardening, Traefik, Forgejo, and S3 storage.
|
`site.yaml` is the desired state of both servers and is applied by the
|
||||||
|
deploy pipeline on every push to `main`. The playbooks under
|
||||||
|
`playbooks/` are host bootstrap and are run by hand.
|
||||||
|
|
||||||
## Quick Start
|
## Running
|
||||||
|
|
||||||
```bash
|
```sh
|
||||||
# 1. Setup vault
|
echo '<vault password>' > .vault_pass && chmod 600 .vault_pass
|
||||||
echo "your-password" > .vault_pass && chmod 600 .vault_pass
|
|
||||||
./scripts/setup-vault.sh encrypt
|
|
||||||
|
|
||||||
# 2. Stage 1: WireGuard VPN
|
ansible-playbook site.yaml --check --diff # dry run, over WireGuard
|
||||||
ansible-playbook playbooks/01-wireguard-setup.yaml
|
ansible-playbook site.yaml # apply
|
||||||
# Connect to VPN, update inventory/hosts.yaml to use WireGuard IPs
|
ansible-playbook site.yaml --tags forgejo # one role
|
||||||
|
|
||||||
# 3. Stage 2: Full deployment (SSH hardening, Traefik, Forgejo, Runner)
|
|
||||||
ansible-playbook playbooks/02-full-deployment.yaml
|
|
||||||
|
|
||||||
# 4. Stage 3: S3 storage mounts
|
|
||||||
ansible-playbook playbooks/03-setup-s3fs.yaml
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Infrastructure
|
On NixOS: `nix shell nixpkgs#ansible -c ansible-playbook …`.
|
||||||
|
|
||||||
**Servers:**
|
Templates that carry secrets are never shown in `--diff` output.
|
||||||
- **asxpio-ops** (CX22): Forgejo, Traefik, WireGuard VPN hub
|
|
||||||
- **asxpio-services** (CX43): Forgejo Runner, Immich, game servers
|
|
||||||
|
|
||||||
**Networks:**
|
## Inventories
|
||||||
- VPC: 10.10.0.0/20
|
|
||||||
- WireGuard: 10.20.0.0/24
|
|
||||||
|
|
||||||
**S3 Buckets (mounted via s3fs):**
|
- `inventory/hosts.yaml`: default. Connects as root to the WireGuard
|
||||||
- asxpio-terraform-main → /mnt/s3/terraform
|
addresses.
|
||||||
- asxpio-backups-main → /mnt/s3/backups
|
- `inventory/ci.yaml`: used by the pipeline. Connects as `deploy` over
|
||||||
- asxpio-immich-main → /mnt/s3/immich
|
the private network, reaching asxpio-services through asxpio-ops.
|
||||||
|
|
||||||
## Playbook Workflow
|
`known_hosts` pins both servers' SSH host keys for the pipeline.
|
||||||
|
|
||||||
1. **WireGuard Setup** - Deploy VPN, generate client configs
|
## Variables
|
||||||
2. **Full Deployment** - SSH hardening (no lockout), Traefik, Forgejo, Runner
|
|
||||||
3. **S3 Storage** - Mount S3 buckets with automatic cache management
|
|
||||||
|
|
||||||
All playbooks are idempotent - safe to rerun without breaking existing setup.
|
|
||||||
|
|
||||||
## Configuration Files
|
|
||||||
|
|
||||||
```
|
```
|
||||||
inventory/
|
inventory/
|
||||||
├── hosts.yaml # Server inventory, IPs, groups
|
├── group_vars/all/
|
||||||
├── group_vars/all/ # Variables applied to all hosts
|
│ ├── network.yaml # VPC and WireGuard addresses
|
||||||
│ ├── main.yaml # Forgejo version, ports, runner config
|
│ ├── paths.yaml # /opt
|
||||||
│ ├── network.yaml # VPC (10.10.0.0/20), WireGuard (10.20.0.0/24) IPs
|
│ ├── s3.yaml # object storage buckets and s3fs mounts
|
||||||
│ ├── services.yaml # Service paths (/opt/*), ports, Docker networks
|
│ ├── wireguard.yaml # WireGuard peers
|
||||||
│ ├── s3.yaml # S3 buckets, mount points, credentials reference
|
│ └── vault.yaml # encrypted secrets
|
||||||
│ └── vault.yaml # ENCRYPTED: S3 keys, Forgejo secrets, runner token
|
├── group_vars/ops_servers/
|
||||||
└── host_vars/ # Host-specific overrides
|
│ ├── traefik.yaml # traefik_version
|
||||||
├── asxpio-ops.yaml # WireGuard server, clients list, S3 cache: 10GB
|
│ └── forgejo.yaml # forgejo_version, domain and aliases
|
||||||
└── asxpio-services.yaml # S3 cache: 80GB
|
├── group_vars/services_servers/
|
||||||
|
│ ├── forgejo_runner.yaml # runner version, labels' nix image
|
||||||
|
│ └── minecraft.yaml
|
||||||
|
└── host_vars/ # per-host addresses, SSH and s3fs settings
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Compose files without variables are plain files under
|
||||||
|
`roles/<role>/files/`; edit the image tag there.
|
||||||
|
|
||||||
|
## Roles in site.yaml
|
||||||
|
|
||||||
|
`base`, `traefik`, `crawler_alert`, `forgejo`, `forgejo_runner`,
|
||||||
|
`sillytavern`, `minecraft`, `minecraft_fossils`. Service roles go
|
||||||
|
through `docker_service`, which writes the files, runs
|
||||||
|
`docker compose up -d`, restarts on a config change and waits for the
|
||||||
|
service to answer.
|
||||||
|
|
||||||
|
`crawler_alert` installs a copy of
|
||||||
|
[traefik-crawlers-statistics](https://git.asxp.io/asxpi/traefik-crawlers-statistics);
|
||||||
|
update `roles/crawler_alert/files/` from there.
|
||||||
|
|
||||||
|
## Bootstrap playbooks
|
||||||
|
|
||||||
|
| Playbook | Purpose |
|
||||||
|
|----------|---------|
|
||||||
|
| `01-wireguard-setup.yaml` | WireGuard hub on asxpio-ops, peer on asxpio-services, client configs |
|
||||||
|
| `02-ssh-hardening.yaml` | restrict SSH to WireGuard and the private network |
|
||||||
|
| `03-setup-s3fs.yaml` | s3fs mounts on asxpio-services (off on asxpio-ops) |
|
||||||
|
| `04-add-user-zerrtika.yaml` | add a user to running servers |
|
||||||
|
| `05-swap.yaml` | 2 GB swapfile on asxpio-ops |
|
||||||
|
|
||||||
|
Always dry-run these first with `--check --diff`.
|
||||||
|
|
||||||
|
- `03-setup-s3fs.yaml` rewrites `/etc/s3fs/.passwd-s3fs`, which the
|
||||||
|
backup script on asxpio-ops also reads.
|
||||||
|
- On asxpio-services the WireGuard interface is named
|
||||||
|
`asxpio-services`, not `wg0`. Check `01-wireguard-setup.yaml` against
|
||||||
|
the host before re-running it there.
|
||||||
|
|
||||||
|
## Runner notes
|
||||||
|
|
||||||
|
The `nix-latest` label runs `nixos/nix:<forgejo_runner_nix_version>`
|
||||||
|
with a `nix-store-<version>` volume over `/nix`. To bump it: change the
|
||||||
|
variable, let the pipeline apply it, then remove the old volume.
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue