Sergei Poljanski
59b5b4468a
Hetzner's server backups cost 20% of each server's price and only restore whole disks. Back up what matters with restic instead, into the backup buckets we already pay for: one repository per host in hel1 (asxpio-backups-main) and one in fsn1 (asxpio-backups-repl), written independently so either region alone is a full copy. Postgres goes in as a dump, Forgejo's and vhpbot's SQLite databases through the online backup API, Garage with a fresh metadata snapshot, ejabberd through its own backup command. Keeps 7 daily, 4 weekly and 6 monthly snapshots, prunes on Sundays, and mails the journal when a run fails. restic is the pinned upstream release, so its version is tracked in the repo like everything else. The vault gains the restic password and the Xray config for the role that follows. Assisted-by: Claude:opus-5-5
20 lines
796 B
Python
20 lines
796 B
Python
#!/usr/bin/python3
|
|
# Mail the tail of a failed unit's journal. Usage: restic-alert <unit>
|
|
import os, smtplib, socket, subprocess, sys
|
|
from email.message import EmailMessage
|
|
|
|
unit = sys.argv[1]
|
|
env = dict(l.rstrip("\n").split("=", 1) for l in open("/etc/restic/env") if "=" in l)
|
|
log = subprocess.run(["journalctl", "-u", unit, "-n", "60", "--no-pager", "-o", "cat"],
|
|
capture_output=True, text=True).stdout
|
|
|
|
msg = EmailMessage()
|
|
msg["Subject"] = f"[{socket.gethostname()}] {unit} failed"
|
|
msg["From"] = env["ALERT_SMTP_USER"]
|
|
msg["To"] = env["ALERT_RECIPIENT"]
|
|
msg.set_content(log)
|
|
|
|
with smtplib.SMTP(env["ALERT_SMTP_HOST"], int(env["ALERT_SMTP_PORT"]), timeout=30) as s:
|
|
s.starttls()
|
|
s.login(env["ALERT_SMTP_USER"], env["ALERT_SMTP_PASSWORD"])
|
|
s.send_message(msg)
|