ansible: monitoring with Prometheus, Loki and Grafana

Nothing watched the servers: tonight's asxpio crash loop was found by
hand. On asxpio-ops, Prometheus (30 days), Loki (14 days) and Grafana
at gf.asxp.io (WireGuard only), with blackbox_exporter probing every
service from outside and postgres_exporter. On both hosts node_exporter
and cAdvisor on the VPC address, and Alloy shipping container logs, the
journal and Traefik's access log to Loki over the VPC.

Traefik and Forgejo expose their metrics; restic and the bucket replica
record their last success for node_exporter. Dashboards: an overview as
home, plus hosts, containers, Traefik, probes, Postgres and logs.
Alerts by email to me@asxp.io: host or service down, certificate under
14 days, disk or memory nearly full, a container restart-looping,
Postgres down, backup or replica stale.

Assisted-by: Claude:opus-5-5

ansible/roles/monitoring/templates/grafana.env.j2

@@ -0,0 +1,17 @@
+GF_SERVER_ROOT_URL=https://{{ monitoring_grafana_host }}/
+GF_SECURITY_ADMIN_USER=admin
+GF_SECURITY_ADMIN_PASSWORD={{ monitoring_grafana_admin_password }}
+GF_SECURITY_COOKIE_SECURE=true
+GF_USERS_ALLOW_SIGN_UP=false
+GF_AUTH_ANONYMOUS_ENABLED=false
+GF_ANALYTICS_REPORTING_ENABLED=false
+GF_ANALYTICS_CHECK_FOR_UPDATES=false
+GF_NEWS_NEWS_FEED_ENABLED=false
+GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/overview.json
+GF_SMTP_ENABLED=true
+GF_SMTP_HOST=smtp.fastmail.com:587
+GF_SMTP_USER={{ vault_forgejo_smtp_user }}
+GF_SMTP_PASSWORD={{ vault_forgejo_smtp_password }}
+GF_SMTP_FROM_ADDRESS={{ vault_forgejo_smtp_user }}
+GF_SMTP_FROM_NAME=asxpio monitoring
+GF_SMTP_STARTTLS_POLICY=MandatoryStartTLS