ansible: nightly restic backups to both object storage regions

Hetzner's server backups cost 20% of each server's price and only
restore whole disks. Back up what matters with restic instead, into the
backup buckets we already pay for: one repository per host in hel1
(asxpio-backups-main) and one in fsn1 (asxpio-backups-repl), written
independently so either region alone is a full copy.

Postgres goes in as a dump, Forgejo's and vhpbot's SQLite databases
through the online backup API, Garage with a fresh metadata snapshot,
ejabberd through its own backup command. Keeps 7 daily, 4 weekly and
6 monthly snapshots, prunes on Sundays, and mails the journal when a
run fails.

restic is the pinned upstream release, so its version is tracked in
the repo like everything else. The vault gains the restic password
and the Xray config for the role that follows.

Assisted-by: Claude:opus-5-5

ansible/roles/restic_backup/tasks/main.yaml

@@ -0,0 +1,88 @@
+---
+- name: Install sqlite3 for consistent copies of live databases
+  ansible.builtin.apt:
+    name: [sqlite3, bzip2]
+    state: present
+
+- name: Check the installed restic version
+  ansible.builtin.command: /usr/local/bin/restic version
+  register: restic_installed
+  changed_when: false
+  failed_when: false
+  check_mode: false
+
+- name: Install restic {{ restic_version }}
+  when: ("restic " ~ restic_version ~ " ") not in (restic_installed.stdout | default(''))
+  block:
+    - name: Download the release
+      ansible.builtin.get_url:
+        url: "https://github.com/restic/restic/releases/download/v{{ restic_version }}/restic_{{ restic_version }}_linux_amd64.bz2"
+        dest: "/tmp/restic_{{ restic_version }}.bz2"
+        checksum: "sha256:https://github.com/restic/restic/releases/download/v{{ restic_version }}/SHA256SUMS"
+        mode: "0600"
+
+    - name: Unpack it
+      ansible.builtin.shell: >
+        bunzip2 -c /tmp/restic_{{ restic_version }}.bz2 > /usr/local/bin/restic.new &&
+        chmod 755 /usr/local/bin/restic.new &&
+        mv /usr/local/bin/restic.new /usr/local/bin/restic &&
+        rm /tmp/restic_{{ restic_version }}.bz2
+      changed_when: true
+
+- name: Create config and cache directories
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    mode: "0700"
+  loop: [/etc/restic, /var/cache/restic]
+
+- name: Write credentials
+  ansible.builtin.template:
+    src: env.j2
+    dest: /etc/restic/env
+    mode: "0600"
+  diff: false
+
+- name: Install the scripts
+  ansible.builtin.template:
+    src: restic-backup.j2
+    dest: /usr/local/sbin/restic-backup
+    mode: "0700"
+
+- name: Install the alert script
+  ansible.builtin.copy:
+    src: restic-alert
+    dest: /usr/local/sbin/restic-alert
+    mode: "0700"
+
+- name: Install the systemd units
+  ansible.builtin.template:
+    src: "{{ item }}.j2"
+    dest: "/etc/systemd/system/{{ item }}"
+    mode: "0644"
+  loop: [restic-backup.service, restic-backup.timer]
+  notify: Reload systemd
+
+- name: Install the alert unit
+  ansible.builtin.copy:
+    src: restic-alert@.service
+    dest: /etc/systemd/system/restic-alert@.service
+    mode: "0644"
+  notify: Reload systemd
+
+- name: Flush handlers
+  ansible.builtin.meta: flush_handlers
+
+- name: Create the repositories
+  ansible.builtin.command: /usr/local/sbin/restic-backup init
+  register: restic_init
+  changed_when: "'created' in restic_init.stdout"
+  when: not ansible_check_mode
+
+- name: Enable the nightly timer
+  ansible.builtin.systemd:
+    name: restic-backup.timer
+    enabled: true
+    state: started
+  # On a first dry run the unit is not installed yet
+  ignore_errors: "{{ ansible_check_mode }}"