Sergei Poljanski
0b331bc5f1
The first run failed with "home directory /homeless-shelter exists": job containers are unprivileged, Nix cannot sandbox builds in them, and unsandboxed builds trip over each other's leftovers. Evaluate the system in the job instead, which still catches broken options before main, and let nixos-rebuild build on the target host, where the Nix daemon has a proper sandbox. Assisted-by: Claude:fable-5-2
68 lines
2.6 KiB
Markdown
68 lines
2.6 KiB
Markdown
# asxp.io infrastructure
|
|
|
|
Two Hetzner Cloud servers, described by Terraform. asxpio-services runs
|
|
NixOS (`nixos/`); asxpio-ops runs Debian configured by Ansible
|
|
(`ansible/`). Pushing to `main` deploys both through Forgejo Actions.
|
|
|
|
## Servers
|
|
|
|
| Server | Size | Runs |
|
|
|--------|------|------|
|
|
| asxpio-ops | 2 vCPU, 4 GB | Traefik, Forgejo, crawler-alert; app stacks deployed from their own repos |
|
|
| asxpio-services | 8 vCPU, 16 GB | Forgejo Actions runner, Minecraft (Paper and Fossils), SillyTavern |
|
|
|
|
Both are reachable for administration only over WireGuard
|
|
(10.20.0.0/24) or the private network (10.10.0.0/20).
|
|
|
|
## Layout
|
|
|
|
- `terraform/`: servers, network, firewalls, object storage buckets.
|
|
- `nixos/`: the whole asxpio-services system as a flake.
|
|
- `ansible/site.yaml`: everything managed on asxpio-ops.
|
|
- `ansible/playbooks/`: asxpio-ops bootstrap, run by hand.
|
|
- `.forgejo/workflows/`: `nixos.yaml` and `deploy.yaml`, the pipelines.
|
|
|
|
## What is managed where
|
|
|
|
| What | Host | Where |
|
|
|------|------|-------|
|
|
| OS, users, SSH, WireGuard, firewall | services | `nixos/` |
|
|
| CI runner, Minecraft (Paper, Fossils), SillyTavern | services | `nixos/hosts/asxpio-services/` |
|
|
| baseline (upgrades, fail2ban, journal, Docker config) | ops | role `base` |
|
|
| Traefik, Forgejo, crawler-alert | ops | roles `traefik`, `forgejo`, `crawler_alert` |
|
|
|
|
Not managed here: the app stacks on asxpio-ops (asxpio, datacenterbingo,
|
|
blog, czsk, vhpbot, storage). Each deploys from its own repository into
|
|
`/opt/<app>`.
|
|
|
|
## Changing something
|
|
|
|
Edit, push a branch, merge to `main`.
|
|
|
|
- A branch push evaluates the NixOS system and dry-runs Ansible against
|
|
asxpio-ops.
|
|
- `main` then switches asxpio-services to the new system and applies
|
|
Ansible to asxpio-ops.
|
|
|
|
A run with nothing to change restarts nothing. A change to the CI
|
|
runner restarts the runner under its own job: the change lands, that
|
|
job is reported as failed, and the next run confirms it. Details are in
|
|
`nixos/README.md` and `ansible/README.md`.
|
|
|
|
## Secrets
|
|
|
|
Nothing secret is stored in the clear. asxpio-services secrets are in
|
|
`nixos/secrets/`, encrypted with sops; asxpio-ops secrets are in
|
|
`ansible/inventory/group_vars/all/vault.yaml`, encrypted with Ansible
|
|
Vault. The pipelines need two repository secrets:
|
|
|
|
- `ANSIBLE_VAULT_PASSWORD`: decrypts the vault.
|
|
- `SSH_DEPLOY_KEY`: private key of user `deploy` on both servers.
|
|
|
|
Locally, put the vault password in `ansible/.vault_pass` (ignored by git).
|
|
|
|
## Bootstrap (by hand)
|
|
|
|
`ansible/playbooks/` holds what is done once on asxpio-ops or can lock
|
|
you out: WireGuard, SSH hardening, swap, user creation. Installing or
|
|
reinstalling asxpio-services is in `nixos/README.md`.
|