Sergei Poljanski
e20f5a7d07
All checks were successful
deploy / ansible (push) Successful in 7m23s
Nothing watched the servers: tonight's asxpio crash loop was found by hand. On asxpio-ops, Prometheus (30 days), Loki (14 days) and Grafana at gf.asxp.io (WireGuard only), with blackbox_exporter probing every service from outside and postgres_exporter. On both hosts node_exporter and cAdvisor on the VPC address, and Alloy shipping container logs, the journal and Traefik's access log to Loki over the VPC. Traefik and Forgejo expose their metrics; restic and the bucket replica record their last success for node_exporter. Dashboards: an overview as home, plus hosts, containers, Traefik, probes, Postgres and logs. Alerts by email to me@asxp.io: host or service down, certificate under 14 days, disk or memory nearly full, a container restart-looping, Postgres down, backup or replica stale. Assisted-by: Claude:opus-5-5
72 lines
1.9 KiB
YAML
72 lines
1.9 KiB
YAML
---
|
|
# Everything the deploy pipeline manages. Idempotent: a run with nothing
|
|
# to change touches nothing and restarts nothing.
|
|
#
|
|
# ansible-playbook site.yaml --check --diff # dry run
|
|
# ansible-playbook site.yaml --tags traefik # one service
|
|
#
|
|
# Host bootstrap (WireGuard, SSH hardening, s3fs, swap) is in playbooks/
|
|
# and is run by hand.
|
|
|
|
- name: Host baseline
|
|
hosts: asxpio
|
|
become: true
|
|
roles:
|
|
- { role: base, tags: [base] }
|
|
|
|
- name: Monitoring agents
|
|
hosts: asxpio
|
|
become: true
|
|
roles:
|
|
- { role: monitoring_agent, tags: [monitoring_agent, monitoring] }
|
|
|
|
- name: asxpio-ops services
|
|
hosts: ops_servers
|
|
become: true
|
|
roles:
|
|
- { role: traefik, tags: [traefik] }
|
|
- { role: crawler_alert, tags: [crawler_alert] }
|
|
- { role: monitoring, tags: [monitoring] }
|
|
|
|
- name: asxpio-services services
|
|
hosts: services_servers
|
|
become: true
|
|
roles:
|
|
- { role: sillytavern, tags: [sillytavern] }
|
|
- { role: minecraft, tags: [minecraft] }
|
|
- { role: xmpp, tags: [xmpp] }
|
|
- { role: xray, tags: [xray] }
|
|
|
|
# Late: restarting Forgejo briefly cuts the runner off from its job log
|
|
- name: Forgejo
|
|
hosts: ops_servers
|
|
become: true
|
|
roles:
|
|
- { role: forgejo, tags: [forgejo] }
|
|
|
|
- name: Backups
|
|
hosts: asxpio
|
|
become: true
|
|
roles:
|
|
- { role: restic_backup, tags: [restic] }
|
|
|
|
- name: another-im/interop runner
|
|
hosts: services_servers
|
|
become: true
|
|
roles:
|
|
- { role: forgejo_runner_interop, tags: [forgejo_runner_interop] }
|
|
|
|
- name: Bucket replica
|
|
hosts: services_servers
|
|
become: true
|
|
roles:
|
|
- { role: s3_replicate, tags: [s3_replicate] }
|
|
|
|
# Last: when the pipeline changes the runner it recreates the container
|
|
# that is running the pipeline. Everything else is done by then; that job
|
|
# ends as failed and the next run confirms the result.
|
|
- name: Forgejo runner
|
|
hosts: services_servers
|
|
become: true
|
|
roles:
|
|
- { role: forgejo_runner, tags: [forgejo_runner] }
|