# Ansible

`site.yaml` is the desired state of both servers and is applied by the
deploy pipeline on every push to `main`. The playbooks under
`playbooks/` are host bootstrap and are run by hand.

## Running

```sh
echo '<vault password>' > .vault_pass && chmod 600 .vault_pass

ansible-playbook site.yaml --check --diff       # dry run, over WireGuard
ansible-playbook site.yaml                      # apply
ansible-playbook site.yaml --tags forgejo       # one role
```

On NixOS: `nix shell nixpkgs#ansible -c ansible-playbook …`.

Templates that carry secrets are never shown in `--diff` output.

## Inventories

- `inventory/hosts.yaml`: default. Connects as root to the WireGuard
  addresses.
- `inventory/ci.yaml`: used by the pipeline. Connects as `deploy` over
  the private network, reaching asxpio-services through asxpio-ops.

`known_hosts` pins both servers' SSH host keys for the pipeline.

## Variables

```
inventory/
├── group_vars/all/
│   ├── network.yaml         # VPC and WireGuard addresses
│   ├── paths.yaml           # /opt
│   ├── s3.yaml              # object storage buckets and s3fs mounts
│   ├── wireguard.yaml       # WireGuard peers
│   └── vault.yaml           # encrypted secrets
├── group_vars/ops_servers/
│   ├── traefik.yaml         # traefik_version
│   └── forgejo.yaml         # forgejo_version, domain and aliases
├── group_vars/services_servers/
│   ├── forgejo_runner.yaml  # runner version, labels' nix image
│   └── minecraft.yaml
└── host_vars/               # per-host addresses, SSH and s3fs settings
```

Compose files without variables are plain files under
`roles/<role>/files/`; edit the image tag there.

## Roles in site.yaml

`base`, `traefik`, `crawler_alert`, `forgejo`, `forgejo_runner`,
`sillytavern`, `minecraft`, `xmpp`, `xray`, `restic_backup`. Service roles go
through `docker_service`, which writes the files, runs
`docker compose up -d`, restarts on a config change and waits for the
service to answer.

`crawler_alert` installs a copy of
[traefik-crawlers-statistics](https://git.asxp.io/asxpi/traefik-crawlers-statistics);
update `roles/crawler_alert/files/` from there.

## Bootstrap playbooks

| Playbook | Purpose |
|----------|---------|
| `01-wireguard-setup.yaml` | WireGuard hub on asxpio-ops, peer on asxpio-services, client configs |
| `02-ssh-hardening.yaml` | restrict SSH to WireGuard and the private network |
| `03-setup-s3fs.yaml` | s3fs mounts on asxpio-services (off on asxpio-ops) |
| `04-add-user-zerrtika.yaml` | add a user to running servers |
| `05-swap.yaml` | 2 GB swapfile on asxpio-ops |

Always dry-run these first with `--check --diff`.

- `03-setup-s3fs.yaml` rewrites `/etc/s3fs/.passwd-s3fs`, which the
  backup script on asxpio-ops also reads.
- On asxpio-services the WireGuard interface is named
  `asxpio-services`, not `wg0`. Check `01-wireguard-setup.yaml` against
  the host before re-running it there.

## Runner notes

The `nix-latest` label runs `nixos/nix:<forgejo_runner_nix_version>`
with a `nix-store-<version>` volume over `/nix`. To bump it: change the
variable, let the pipeline apply it, then remove the old volume.

## Runner and nix image upgrades

`forgejo_runner_version` and `forgejo_runner_nix_version` are bumped by
Renovate PRs. Applying either one recreates the runner while it runs
the very job that applies it:

- that run ends as *failed* (runner 13; *cancelled* on older runners)
  after everything else has been applied, its log stopping in the
  forgejo_runner role,
  and its job container stays behind on asxpio-services
  (`docker ps | grep FORGEJO-ACTIONS-TASK`); remove it;
- the next run confirms the result. The workflow only triggers on
  changes under `ansible/` (or `workflow_dispatch` in the web UI), so an
  empty commit starts nothing;
- after a nix bump, remove the old `nix-store-<old version>` volume.