ansible: monitoring with Prometheus, Loki and Grafana

Nothing watched the servers: tonight's asxpio crash loop was found by
hand. On asxpio-ops, Prometheus (30 days), Loki (14 days) and Grafana
at gf.asxp.io (WireGuard only), with blackbox_exporter probing every
service from outside and postgres_exporter. On both hosts node_exporter
and cAdvisor on the VPC address, and Alloy shipping container logs, the
journal and Traefik's access log to Loki over the VPC.

Traefik and Forgejo expose their metrics; restic and the bucket replica
record their last success for node_exporter. Dashboards: an overview as
home, plus hosts, containers, Traefik, probes, Postgres and logs.
Alerts by email to me@asxp.io: host or service down, certificate under
14 days, disk or memory nearly full, a container restart-looping,
Postgres down, backup or replica stale.

Assisted-by: Claude:opus-5-5

ansible/roles/monitoring/templates/loki.yml.j2

@@ -0,0 +1,37 @@
+# Managed by Ansible (role monitoring).
+auth_enabled: false
+server:
+  http_listen_port: 3100
+  log_level: warn
+common:
+  path_prefix: /loki
+  replication_factor: 1
+  ring:
+    kvstore:
+      store: inmemory
+  storage:
+    filesystem:
+      chunks_directory: /loki/chunks
+      rules_directory: /loki/rules
+schema_config:
+  configs:
+    - from: "2026-11-01"
+      store: tsdb
+      object_store: filesystem
+      schema: v13
+      index:
+        prefix: index_
+        period: 24h
+limits_config:
+  retention_period: {{ monitoring_logs_retention }}
+  reject_old_samples: true
+  reject_old_samples_max_age: 168h
+  ingestion_rate_mb: 8
+  ingestion_burst_size_mb: 16
+  volume_enabled: true
+compactor:
+  working_directory: /loki/compactor
+  retention_enabled: true
+  delete_request_store: filesystem
+analytics:
+  reporting_enabled: false