ansible: monitoring with Prometheus, Loki and Grafana

Nothing watched the servers: tonight's asxpio crash loop was found by
hand. On asxpio-ops, Prometheus (30 days), Loki (14 days) and Grafana
at gf.asxp.io (WireGuard only), with blackbox_exporter probing every
service from outside and postgres_exporter. On both hosts node_exporter
and cAdvisor on the VPC address, and Alloy shipping container logs, the
journal and Traefik's access log to Loki over the VPC.

Traefik and Forgejo expose their metrics; restic and the bucket replica
record their last success for node_exporter. Dashboards: an overview as
home, plus hosts, containers, Traefik, probes, Postgres and logs.
Alerts by email to me@asxp.io: host or service down, certificate under
14 days, disk or memory nearly full, a container restart-looping,
Postgres down, backup or replica stale.

Assisted-by: Claude:opus-5-5

ansible/site.yaml

@@ -14,12 +14,19 @@
   roles:
     - { role: base, tags: [base] }
 
+- name: Monitoring agents
+  hosts: asxpio
+  become: true
+  roles:
+    - { role: monitoring_agent, tags: [monitoring_agent, monitoring] }
+
 - name: asxpio-ops services
   hosts: ops_servers
   become: true
   roles:
     - { role: traefik, tags: [traefik] }
     - { role: crawler_alert, tags: [crawler_alert] }
+    - { role: monitoring, tags: [monitoring] }
 
 - name: asxpio-services services
   hosts: services_servers