ansible: monitoring with Prometheus, Loki and Grafana

Nothing watched the servers: tonight's asxpio crash loop was found by
hand. On asxpio-ops, Prometheus (30 days), Loki (14 days) and Grafana
at gf.asxp.io (WireGuard only), with blackbox_exporter probing every
service from outside and postgres_exporter. On both hosts node_exporter
and cAdvisor on the VPC address, and Alloy shipping container logs, the
journal and Traefik's access log to Loki over the VPC.

Traefik and Forgejo expose their metrics; restic and the bucket replica
record their last success for node_exporter. Dashboards: an overview as
home, plus hosts, containers, Traefik, probes, Postgres and logs.
Alerts by email to me@asxp.io: host or service down, certificate under
14 days, disk or memory nearly full, a container restart-looping,
Postgres down, backup or replica stale.

Assisted-by: Claude:opus-5-5

ansible/roles/restic_backup/templates/restic-backup.j2

@@ -22,6 +22,12 @@ declare -A REPOS=(
 
 log() { echo "restic-backup: $*"; }
 
+# Last good backup per repository, for Prometheus (node_exporter textfile)
+metric() {
+  local f={{ monitoring_textfile_dir }}/restic_$1.prom
+  printf '# HELP restic_backup_last_success_timestamp_seconds Last successful restic backup.\n# TYPE restic_backup_last_success_timestamp_seconds gauge\nrestic_backup_last_success_timestamp_seconds{repo="%s"} %s\n' "$1" "$(date +%s)" > "$f.tmp" && mv "$f.tmp" "$f"
+}
+
 case "${1:-}" in
 init)
   for name in "${!REPOS[@]}"; do
@@ -56,6 +62,7 @@ run)
     [ "$(date +%a)" = "{{ restic_prune_weekday }}" ] && prune="--prune"
     log "$name: forget $prune"
     restic forget --host {{ inventory_hostname }} {{ restic_keep }} $prune || failed=1
+    metric "$name"
   done
 
   rm -rf "$STAGE"