ansible: monitoring with Prometheus, Loki and Grafana

Nothing watched the servers: tonight's asxpio crash loop was found by
hand. On asxpio-ops, Prometheus (30 days), Loki (14 days) and Grafana
at gf.asxp.io (WireGuard only), with blackbox_exporter probing every
service from outside and postgres_exporter. On both hosts node_exporter
and cAdvisor on the VPC address, and Alloy shipping container logs, the
journal and Traefik's access log to Loki over the VPC.

Traefik and Forgejo expose their metrics; restic and the bucket replica
record their last success for node_exporter. Dashboards: an overview as
home, plus hosts, containers, Traefik, probes, Postgres and logs.
Alerts by email to me@asxp.io: host or service down, certificate under
14 days, disk or memory nearly full, a container restart-looping,
Postgres down, backup or replica stale.

Assisted-by: Claude:opus-5-5

ansible/roles/monitoring/tasks/main.yaml

@@ -0,0 +1,37 @@
+---
+# Prometheus, Loki, Grafana, blackbox_exporter and postgres_exporter on
+# asxpio-ops. Grafana: https://gf.asxp.io, WireGuard only.
+- name: Create the provisioning directories
+  ansible.builtin.file:
+    path: "{{ monitoring_base_path }}/{{ item }}"
+    state: directory
+    mode: "0755"
+  loop:
+    - grafana/provisioning/datasources
+    - grafana/provisioning/dashboards
+    - grafana/provisioning/alerting
+    - grafana/dashboards
+
+- name: Deploy the dashboards
+  ansible.builtin.copy:
+    src: dashboards/
+    dest: "{{ monitoring_base_path }}/grafana/dashboards/"
+    mode: "0644"
+
+- name: Deploy the monitoring stack
+  ansible.builtin.include_role:
+    name: docker_service
+  vars:
+    service_name: monitoring
+    service_base_path: "{{ monitoring_base_path }}"
+    service_templates:
+      - { src: docker-compose.yml.j2, dest: docker-compose.yml }
+      - { src: prometheus.yml.j2, dest: prometheus.yml }
+      - { src: blackbox.yml.j2, dest: blackbox.yml }
+      - { src: loki.yml.j2, dest: loki.yml }
+      - { src: grafana.env.j2, dest: grafana.env, mode: "0600" }
+      - { src: datasources.yaml.j2, dest: grafana/provisioning/datasources/datasources.yaml }
+      - { src: dashboards.yaml.j2, dest: grafana/provisioning/dashboards/dashboards.yaml }
+      - { src: alerting.yaml.j2, dest: grafana/provisioning/alerting/alerting.yaml }
+      - { src: alert-rules.yaml.j2, dest: grafana/provisioning/alerting/rules.yaml }
+    health_check_url: "http://{{ asxpio_ops_vpc_ip }}:{{ monitoring_loki_port }}/ready"