Sergei Poljanski
e20f5a7d07
All checks were successful
deploy / ansible (push) Successful in 7m23s
Nothing watched the servers: tonight's asxpio crash loop was found by hand. On asxpio-ops, Prometheus (30 days), Loki (14 days) and Grafana at gf.asxp.io (WireGuard only), with blackbox_exporter probing every service from outside and postgres_exporter. On both hosts node_exporter and cAdvisor on the VPC address, and Alloy shipping container logs, the journal and Traefik's access log to Loki over the VPC. Traefik and Forgejo expose their metrics; restic and the bucket replica record their last success for node_exporter. Dashboards: an overview as home, plus hosts, containers, Traefik, probes, Postgres and logs. Alerts by email to me@asxp.io: host or service down, certificate under 14 days, disk or memory nearly full, a container restart-looping, Postgres down, backup or replica stale. Assisted-by: Claude:opus-5-5
23 lines
993 B
Django/Jinja
23 lines
993 B
Django/Jinja
#!/bin/bash
|
|
# Managed by Ansible (role s3_replicate). hel1 -> fsn1, one bucket at a time;
|
|
# a failing pair does not stop the others, but fails the run.
|
|
set -uo pipefail
|
|
failed=0
|
|
rclone() {
|
|
docker run --rm --env-file /etc/s3-replicate/env \
|
|
rclone/rclone:{{ s3_replicate_rclone_version }} "$@"
|
|
}
|
|
{% for p in s3_replicate_pairs %}
|
|
echo "s3-replicate: {{ p.src }} -> {{ p.dst }}"
|
|
rclone sync hel1:{{ p.src }} fsn1:{{ p.dst }} \
|
|
--fast-list --checkers 16 --transfers 8 --max-delete {{ s3_replicate_max_delete }} \
|
|
{% for e in p.exclude | default([]) %}
|
|
--exclude '{{ e }}' \
|
|
{% endfor %}
|
|
--stats-one-line -v || failed=1
|
|
{% endfor %}
|
|
if [ $failed = 0 ]; then
|
|
f={{ monitoring_textfile_dir }}/s3_replicate.prom
|
|
printf '# HELP s3_replicate_last_success_timestamp_seconds Last fully successful hel1 to fsn1 replica.\n# TYPE s3_replicate_last_success_timestamp_seconds gauge\ns3_replicate_last_success_timestamp_seconds %s\n' "$(date +%s)" > "$f.tmp" && mv "$f.tmp" "$f"
|
|
fi
|
|
exit $failed
|