ansible: describe what runs on the servers

The roles had drifted far from production: Forgejo was pinned at 12
while 15.0.11 runs, its config lacked the mailer, packages and the real
database path, and several services existed only on the hosts. Running
a role would have rolled production back. Take the servers as the truth
and make the repository match them.

site.yaml now holds the desired state of both hosts:

 - base: unattended upgrades, fail2ban, Docker daemon config and the
   weekly image prune, as found on the hosts. New: a 500M journal cap,
   and the Docker systemd override is kept absent. That override once
   opened dockerd on tcp/2375 without authentication.
 - traefik: templates as deployed, version in a variable (v3.7.18).
   The Coraza WAF files are removed; no router ever used them.
 - forgejo: 15.0.11, app.ini generated from the live file with the
   secrets taken from the vault.
 - crawler_alert, sillytavern, minecraft, minecraft_fossils: added
   from the hosts.

Forgejo's canonical name becomes git.asxp.io; git.czsk.it stays as an
alias on the same router.

docker_service restarts a stack when its config changes, no longer
forces ownership on existing data directories, and hides template
diffs because they carry secrets. Containers listed in
service_graceful_stop are saved and stopped with a long timeout before
compose recreates them: Docker's 10 s default is too short for a
Minecraft world. Both game servers get stop_grace_period for the same
reason.

Removed: the full-deployment and runner playbooks with their prompts,
the Forgejo backup and restore roles for the old bare-metal host, and
inventory entries for services that were never deployed. SSH hardening
stays a manual playbook, since a mistake there locks everyone out.

New secrets (SMTP, RCON, CurseForge key) went into the vault.

Assisted-by: Claude:fable-5-2

ansible/roles/base/defaults/main.yaml

@@ -0,0 +1,4 @@
+---
+journald_max_use: 500M
+# Weekly cleanup of images no container uses; every deploy leaves one behind
+docker_prune_schedule: "0 3 * * 1"

ansible/roles/base/files/20auto-upgrades

@@ -0,0 +1,2 @@
+APT::Periodic::Update-Package-Lists "1";
+APT::Periodic::Unattended-Upgrade "1";

ansible/roles/base/files/daemon.json

@@ -0,0 +1,7 @@
+{
+  "live-restore": true,
+  "storage-driver": "overlay2",
+  "log-opts": {
+    "max-size": "20m"
+  }
+}

ansible/roles/base/handlers/main.yaml

@@ -0,0 +1,12 @@
+---
+- name: Restart journald
+  ansible.builtin.systemd:
+    name: systemd-journald
+    state: restarted
+
+# live-restore is on, so containers keep running across this restart
+- name: Restart docker
+  ansible.builtin.systemd:
+    name: docker
+    state: restarted
+    daemon_reload: true

ansible/roles/base/tasks/main.yaml

@@ -0,0 +1,76 @@
+---
+# Host baseline shared by both servers. Users, Docker itself and the
+# first-boot packages come from terraform/cloud-init; SSH and WireGuard
+# have their own bootstrap playbooks.
+
+- name: Install baseline packages
+  ansible.builtin.apt:
+    name:
+      - unattended-upgrades
+      - fail2ban
+    state: present
+
+- name: Enable unattended security upgrades
+  ansible.builtin.copy:
+    src: 20auto-upgrades
+    dest: /etc/apt/apt.conf.d/20auto-upgrades
+    mode: "0644"
+
+- name: Keep fail2ban running (sshd jail from the Debian defaults)
+  ansible.builtin.systemd:
+    name: fail2ban
+    enabled: true
+    state: started
+
+- name: Create journald drop-in directory
+  ansible.builtin.file:
+    path: /etc/systemd/journald.conf.d
+    state: directory
+    mode: "0755"
+
+- name: Cap the journal size
+  ansible.builtin.copy:
+    dest: /etc/systemd/journald.conf.d/size.conf
+    content: |
+      [Journal]
+      SystemMaxUse={{ journald_max_use }}
+    mode: "0644"
+  notify: Restart journald
+
+- name: Configure the Docker daemon
+  ansible.builtin.copy:
+    src: daemon.json
+    dest: /etc/docker/daemon.json
+    mode: "0644"
+  notify: Restart docker
+
+# dockerd must only listen on its unix socket. A drop-in once added
+# -H tcp://<vpc ip>:2375, which gave every container on the private
+# network unauthenticated root on this host.
+- name: Remove any Docker systemd override
+  ansible.builtin.file:
+    path: /etc/systemd/system/docker.service.d/override.conf
+    state: absent
+  notify: Restart docker
+
+- name: Prune unused Docker images weekly
+  ansible.builtin.copy:
+    dest: /etc/cron.d/docker-image-prune
+    content: |
+      {{ docker_prune_schedule }} root docker image prune -af > /var/log/docker-prune.log 2>&1
+    mode: "0644"
+
+- name: Check for the old prune entry in root's crontab
+  ansible.builtin.command: crontab -l
+  register: root_crontab
+  changed_when: false
+  failed_when: false
+  check_mode: false
+
+- name: Drop the old prune entry from root's crontab
+  ansible.builtin.shell: |
+    set -o pipefail
+    crontab -l | grep -v 'docker image prune' | crontab -
+  args:
+    executable: /bin/bash
+  when: "'docker image prune' in root_crontab.stdout"

ansible/roles/crawler_alert/defaults/main.yaml

@@ -0,0 +1,8 @@
+---
+# Daily/weekly AI-crawler report for the Forgejo router, mailed through
+# Forgejo's own SMTP settings. Source and docs:
+# https://git.asxp.io/asxpi/traefik-crawlers-statistics
+crawler_recipient: "{{ vault_crawler_recipient }}"
+crawler_site: git.asxp.io
+crawler_threshold: 100000
+crawler_robots_url: https://git.czsk.it/robots.txt

ansible/roles/crawler_alert/files/crawler-alert.py

@@ -0,0 +1,707 @@
+#!/usr/bin/env python3
+"""Crawler watch for a Traefik-fronted Forgejo (or any Traefik site).
+
+Separates real AI crawlers from traffic that only claims to be, by
+checking the source IP against each vendor's published range list.
+
+Modes:
+  (default)  daily threshold alert: mail only if yesterday exceeded
+             CRAWLER_THRESHOLD requests
+  --weekly   summary: always mail stats for the past 7 days
+  --dry-run  print instead of mailing (combinable with --weekly)
+
+Configuration is by environment variable; see README.md. The only
+value with no sensible default is CRAWLER_RECIPIENT.
+"""
+import configparser, glob, gzip, ipaddress, json, os, re, smtplib, socket, subprocess, sys, urllib.request
+from collections import Counter
+from datetime import date, timedelta
+from email.mime.text import MIMEText
+
+# --- configuration ------------------------------------------------------
+THRESHOLD = int(os.environ.get("CRAWLER_THRESHOLD", "100000"))
+RECIPIENT = os.environ.get("CRAWLER_RECIPIENT", "")
+SITE      = os.environ.get("CRAWLER_SITE", "this site")
+LOG_GLOB  = os.environ.get("CRAWLER_LOG", "/var/log/traefik/access.log*")
+# Traefik router name to filter on, e.g. "forgejo@docker". Empty = all
+# traffic reaching Traefik, which is what you want for a single-site host.
+ROUTER    = os.environ.get("CRAWLER_ROUTER", "")
+# Container to read SMTP settings from (Forgejo/Gitea app.ini [mailer]).
+# Set CRAWLER_SMTP_* instead to configure SMTP directly.
+MAIL_CONTAINER = os.environ.get("CRAWLER_MAIL_CONTAINER", "forgejo")
+MAIL_INI  = os.environ.get("CRAWLER_MAIL_INI", "/data/gitea/conf/app.ini")
+
+WEEKLY = "--weekly" in sys.argv
+DRY = "--dry-run" in sys.argv
+
+if not RECIPIENT and not DRY:
+    sys.exit("CRAWLER_RECIPIENT is not set (no --dry-run either); refusing "
+             "to run. See README.md.")
+
+# --- verified-crawler support -------------------------------------------
+# Vendors publishing machine-readable IP ranges. A UA token alone proves
+# nothing: on the instance this was written for, ~83% of AI-bot-labelled
+# traffic came from hosts outside these ranges. Only IP-verified hits count.
+RANGE_SOURCES = {
+    "openai":    ["https://openai.com/chatgpt-user.json",
+                  "https://openai.com/gptbot.json",
+                  "https://openai.com/searchbot.json"],
+    "anthropic": ["https://claude.com/crawling/bots.json"],
+    "perplexity":["https://www.perplexity.ai/perplexitybot.json"],
+    "apple":     ["https://search.developer.apple.com/applebot.json"],
+    "google":    ["https://developers.google.com/static/crawling/ipranges/special-crawlers.json",
+                  "https://developers.google.com/static/crawling/ipranges/common-crawlers.json",
+                  "https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json",
+                  "https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json"],
+}
+# UA token -> vendor whose ranges must contain the source IP
+BOT_VENDOR = {
+    "ChatGPT-User": "openai", "GPTBot": "openai", "OAI-SearchBot": "openai",
+    "ClaudeBot": "anthropic", "Claude-User": "anthropic", "Claude-SearchBot": "anthropic",
+    "PerplexityBot": "perplexity", "Perplexity-User": "perplexity",
+    "Applebot-Extended": "apple", "Applebot": "apple",
+    "Googlebot": "google", "Google-Extended": "google", "GoogleOther": "google",
+}
+
+# --- reverse-DNS verification -------------------------------------------
+# For vendors that publish no IP ranges. The check is forward-confirmed
+# (FCrDNS): PTR of the address must end in one of the vendor's domains,
+# AND that hostname must resolve back to the same address. The forward
+# step is what makes it meaningful - a PTR record alone is set by
+# whoever controls the address block, so it proves nothing on its own.
+BOT_RDNS = {
+    "Amazonbot":            (".crawl.amazonbot.amazon",),
+    "meta-externalagent":   (".facebook.com", ".fbsv.net"),
+    "meta-externalfetcher": (".facebook.com", ".fbsv.net"),
+    "FacebookBot":          (".facebook.com", ".fbsv.net"),
+    "YouBot":               (".search.you.com",),
+    "Bytespider":           (".bytedance.com", ".byteoversea.com"),
+    "PetalBot":             (".petalsearch.com", ".aspiegel.com"),
+    "DuckAssistBot":        (".duckduckgo.com",),
+    # Apple and Google publish ranges, but also answer rDNS; listed so a
+    # range miss can still be resolved when the published file is stale.
+    "Applebot":             (".applebot.apple.com",),
+    "Googlebot":            (".googlebot.com", ".google.com"),
+}
+# rDNS is slow (two lookups per address), so only unique IPs are probed
+# and results are memoised for the run.
+RDNS_MAX = int(os.environ.get("CRAWLER_RDNS_MAX", "400"))
+RDNS_ENABLE = os.environ.get("CRAWLER_RDNS", "1") != "0"
+
+# --- ASN verification, for single-tenant networks only -------------------
+# An ASN match usually proves nothing: genuine YouBot and genuine
+# Amazonbot both live in AS14618, which is also every EC2 instance a
+# spoofer could rent. It is only evidence when the ASN belongs to the
+# vendor exclusively and sells no compute in it. Meta qualifies -
+# AS32934 is Meta's own network and cannot be rented - and Meta
+# publishes no IP range file (their docs say to email webmasters@meta.com),
+# so without this there is nothing to check them against at all.
+BOT_ASN = {
+    "meta-externalagent":   {"32934"},
+    "meta-externalfetcher": {"32934"},
+    "FacebookBot":          {"32934"},
+    "facebookexternalhit":  {"32934"},
+}
+ASN_ENABLE = os.environ.get("CRAWLER_ASN", "1") != "0"
+# Team Cymru's DNS interface: <reversed-addr>.origin[6].asn.cymru.com TXT
+CYMRU4 = "origin.asn.cymru.com"
+CYMRU6 = "origin6.asn.cymru.com"
+# robots.txt to audit verified crawlers against. Fetched once per run;
+# empty or unreachable simply skips the compliance check.
+ROBOTS_URL = os.environ.get("CRAWLER_ROBOTS", "")
+CACHE = os.environ.get("CRAWLER_CACHE", "/var/cache/crawler-alert/ranges.json")
+CF_URL = "https://www.cloudflare.com/ips-v4"
+# claude.com and cloudflare.com sit behind Cloudflare, which 403s the
+# default Python-urllib UA. Send a real one.
+UA = "Mozilla/5.0 (compatible; crawler-alert/1.0; +https://codeberg.org/)"
+
+
+def fetch(url, timeout=20):
+    req = urllib.request.Request(url, headers={"User-Agent": UA})
+    with urllib.request.urlopen(req, timeout=timeout) as r:
+        return r.read()
+
+
+def load_ranges():
+    """Fetch vendor prefixes; fall back to cache so a vendor outage
+    degrades the report instead of breaking the mail."""
+    out, stale = {}, []
+    for vendor, urls in RANGE_SOURCES.items():
+        nets = []
+        for u in urls:
+            try:
+                for p in json.loads(fetch(u)).get("prefixes", []):
+                    for k, v in p.items():
+                        if k.startswith("ipv"):
+                            nets.append(v)
+            except Exception:
+                stale.append(vendor)
+        if nets:
+            out[vendor] = nets
+    cf = []
+    try:
+        cf = [l.strip() for l in fetch(CF_URL).decode().splitlines() if l.strip()]
+    except Exception:
+        stale.append("cloudflare")
+    try:
+        cached = json.load(open(CACHE))
+    except Exception:
+        cached = {}
+    for vendor in RANGE_SOURCES:
+        if vendor not in out and vendor in cached.get("vendors", {}):
+            out[vendor] = cached["vendors"][vendor]
+    if not cf:
+        cf = cached.get("cloudflare", [])
+    if out:
+        try:
+            os.makedirs(os.path.dirname(CACHE), exist_ok=True)
+            json.dump({"vendors": out, "cloudflare": cf}, open(CACHE, "w"))
+        except Exception:
+            pass
+    # bucket by first octet so lookups stay cheap over ~1425 prefixes
+    idx = {}
+    for vendor, nets in out.items():
+        b = {}
+        for s in nets:
+            n = ipaddress.ip_network(s)
+            b.setdefault(int(n.network_address) >> 24 if n.version == 4 else -1, []).append(n)
+        idx[vendor] = b
+    cfnets = [ipaddress.ip_network(s) for s in cf]
+    return idx, cfnets, sorted(set(stale))
+
+
+def in_index(idx_v, addr):
+    if idx_v is None:
+        return False
+    key = int(addr) >> 24 if addr.version == 4 else -1
+    return any(addr in n for n in idx_v.get(key, []))
+
+
+def load_robots():
+    """Disallow patterns from robots.txt, as (prefix, contains) pairs.
+
+    Only the `User-agent: *` group is read: the point is to audit what
+    well-behaved crawlers were asked not to fetch, not to model every
+    per-agent rule. Returns [] when unset or unreachable, which disables
+    the compliance check rather than reporting false violations.
+    """
+    if not ROBOTS_URL:
+        return []
+    try:
+        txt = fetch(ROBOTS_URL, timeout=15).decode("utf-8", "replace")
+    except Exception:
+        return []
+    rules, in_star = [], False
+    for line in txt.splitlines():
+        line = line.split("#")[0].strip()
+        if not line:
+            continue
+        k, _, v = line.partition(":")
+        k, v = k.strip().lower(), v.strip()
+        if k == "user-agent":
+            in_star = (v == "*")
+        elif k == "disallow" and in_star and v:
+            # Forgejo-style /*/*/thing/ means "contains /thing/".
+            if v.startswith("/*"):
+                rules.append(("", v.lstrip("/*")))
+            else:
+                rules.append((v.rstrip("*"), ""))
+    return rules
+
+
+def robots_blocked(path, rules):
+    for prefix, contains in rules:
+        if prefix and path.startswith(prefix):
+            return True
+        if contains and contains in path:
+            return True
+    return False
+
+
+_asn_cache = {}
+
+
+def asn_of(ip):
+    """Origin ASN for an address via Team Cymru's DNS interface.
+
+    Returns the ASN as a string, or "" if it cannot be determined.
+    Needs `dig`; absence degrades to no ASN evidence, never an error.
+    """
+    if ip in _asn_cache:
+        return _asn_cache[ip]
+    asn = ""
+    try:
+        a = ipaddress.ip_address(ip)
+        if a.version == 4:
+            q = ".".join(reversed(ip.split("."))) + "." + CYMRU4
+        else:
+            nibbles = a.exploded.replace(":", "")
+            q = ".".join(reversed(nibbles)) + "." + CYMRU6
+        out = subprocess.run(["dig", "+short", "+time=3", "+tries=1", q, "TXT"],
+                             capture_output=True, text=True, timeout=12).stdout
+        # "32934 | 2a03:2880:f814::/48 | IE | ripencc | 2011-08-22"
+        first = out.strip().strip('"').split("|")[0].strip()
+        if first.split(" ")[0].isdigit():
+            asn = first.split(" ")[0]
+    except (OSError, ValueError, subprocess.SubprocessError):
+        pass
+    _asn_cache[ip] = asn
+    return asn
+
+
+_rdns_cache = {}
+
+
+def fcrdns(ip, suffixes):
+    """Forward-confirmed reverse DNS.
+
+    PTR must end in one of `suffixes`, and the name it gives must resolve
+    back to `ip`. Returns (ok, hostname). Without the forward step this
+    would only prove the address owner can write their own PTR record.
+    """
+    key = (ip, suffixes)
+    if key in _rdns_cache:
+        return _rdns_cache[key]
+    result = (False, "", "mismatch")
+    try:
+        host = socket.gethostbyaddr(ip)[0].rstrip(".").lower()
+        if any(host.endswith(s) or host == s.lstrip(".") for s in suffixes):
+            _, _, addrs = socket.gethostbyname_ex(host)
+            try:
+                v6 = socket.getaddrinfo(host, None, socket.AF_INET6)
+                addrs = addrs + [a[4][0] for a in v6]
+            except OSError:
+                pass
+            if ip in addrs:
+                result = (True, host, "ok")
+            else:
+                result = (False, host, "mismatch")
+        else:
+            result = (False, host, "mismatch")
+    except OSError:
+        # No PTR at all. This is absence of evidence, not evidence of
+        # forgery: Meta's IPv6 crawler space, for one, publishes no PTR
+        # records. Kept distinct so a real mismatch stays visible.
+        result = (False, "", "no-ptr")
+    _rdns_cache[key] = result
+    return result
+
+days = [date.today() - timedelta(days=i) for i in range(7 if WEEKLY else 1, 0, -1)]
+stamps = [d.strftime("%d/%b/%Y") for d in days]
+
+RANGES, CFNETS, STALE = (load_ranges() if WEEKLY else ({}, [], []))
+
+per_day = {s: [0, set()] for s in stamps}
+repos = Counter()
+uas = Counter()
+verified = Counter()      # bot -> IP-verified hits
+spoofed = Counter()       # bot -> UA claimed, IP outside vendor ranges
+proxied = Counter()       # bot -> arrived via Cloudflare, unverifiable
+ver_paths = Counter()     # paths fetched by verified bots
+ver_ips = {}              # bot -> set of verified source IPs
+# What the verified crawlers actually did, once confirmed: status mix,
+# bytes served, and any path robots.txt asked them not to fetch.
+ver_status = Counter()
+ver_bytes = [0]
+ver_violations = Counter()   # path -> hits, robots.txt disallowed
+ver_bot_bytes = Counter()    # bot -> bytes served
+# Repository *content* fetched by verified crawlers, as opposed to
+# policy files and chrome. This is the question a permissive robots.txt
+# is asking: now that they may read the code, do they? Counting top
+# paths cannot answer it - robots.txt and sitemap.xml drown everything.
+ver_content = Counter()      # bot -> content hits
+ver_content_paths = Counter()
+CONTENT_RE = re.compile(r"/(src|raw|commit|commits|media|blame|archive|compare)/")
+# Which repositories each vendor is actually interested in.
+ver_repos = {}               # bot -> Counter(owner/repo)
+ver_pkgs = Counter()         # "type/name" -> hits, registry artifacts
+ver_pkg_index = Counter()    # owner -> hits on the bare package index
+# Forge paths are /owner/repo/...; these first segments are not owners,
+# and a two-segment path like /pix/photo.jpg on another vhost is a file,
+# not a repository - hence the extension check.
+NOT_OWNER = {"assets", "fonts", "explore", "user", "api", "avatars",
+             "repo-avatars", "swagger.v1.json", "attachments", "notifications",
+             "issues", "pulls", "milestones", "admin", "org", "repo", "login",
+             "-", "favicon.ico", "robots.txt", "sitemap.xml", "pix", "static"}
+
+
+# Container/package registry: /{owner}/-/packages/{type}/{name}[/{version}]
+# Worth counting separately from repositories - it says which published
+# artifacts crawlers are indexing, not which source they are reading.
+PKG_RE = re.compile(r"^/([^/]+)/-/packages(?:/([^/?]+)/([^/?]+))?")
+
+
+def package_of(path):
+    """(owner, 'type/name') for a registry path, else None. Bare
+    /owner/-/packages (the index) returns name None."""
+    m = PKG_RE.match(path.split("?")[0])
+    if not m:
+        return None
+    owner, ptype, name = m.group(1), m.group(2), m.group(3)
+    return (owner, f"{ptype}/{name}" if ptype and name else None)
+
+
+def repo_of(path):
+    """owner/repo for a forge path, else None."""
+    seg = path.lstrip("/").split("?")[0].split("/")
+    if len(seg) < 2 or not seg[0] or not seg[1]:
+        return None
+    if seg[0].lower() in NOT_OWNER or seg[1].lower() in NOT_OWNER:
+        return None
+    if "." in seg[1] and len(seg) == 2:      # /pix/photo.jpg -> a file
+        return None
+    return f"{seg[0]}/{seg[1]}"
+# rDNS-verified vendors: {bot: {ip: hits}}, resolved after the log pass
+rdns_hits = {b: Counter() for b in BOT_RDNS}
+rdns_proxied = Counter()
+# {bot: {ip: Counter(path)}} - kept so that once an address is confirmed
+# by DNS/ASN its paths can join the verified-path breakdown, which
+# otherwise only ever showed IP-range-verified vendors.
+rdns_paths = {b: {} for b in BOT_RDNS}
+ROBOTS_RULES = load_robots() if WEEKLY else []
+
+
+def record_verified(bot, path, status, nbytes):
+    """Log what a confirmed crawler did. Called only after verification,
+    so these counters describe legitimate traffic and nothing else."""
+    ver_paths[path] += 1
+    ver_status[status] += 1
+    ver_bytes[0] += nbytes
+    ver_bot_bytes[bot] += nbytes
+    if CONTENT_RE.search(path):
+        ver_content[bot] += 1
+        ver_content_paths[path] += 1
+    pkg = package_of(path)
+    if pkg:
+        owner, name = pkg
+        if name:
+            ver_pkgs[f"{owner}/{name}"] += 1
+        else:
+            ver_pkg_index[owner] += 1
+    r = repo_of(path)
+    if r:
+        ver_repos.setdefault(bot, Counter())[r] += 1
+    if ROBOTS_RULES and robots_blocked(path, ROBOTS_RULES):
+        ver_violations[path] += 1
+
+
+for path in sorted(glob.glob(LOG_GLOB)):
+    opener = gzip.open if path.endswith(".gz") else open
+    with opener(path, "rt", errors="replace") as f:
+        for line in f:
+            if ROUTER and ROUTER not in line:
+                continue
+            for s in stamps:
+                if s in line:
+                    per_day[s][0] += 1
+                    per_day[s][1].add(line.split(" ", 1)[0])
+                    parts = line.split()
+                    if len(parts) > 6 and parts[6].count("/") >= 2:
+                        repos["/".join(parts[6].split("/")[1:3])] += 1
+                    # Traefik CLF: ... "<referer>" "<user-agent>" <n> "<router>" ...
+                    # Anchor on the request counter so this works whatever
+                    # the router is called.
+                    ua = re.search(r"\"[^\"]*\" \"([^\"]*)\" \d+ \"", line)
+                    if ua:
+                        u = ua.group(1)
+                        m = re.search(r"(bot|crawler|spider|scrapy|externalagent|gpt|claude|perplexity)[\w./-]*", u, re.I)
+                        uas[m.group(0) if m else ("(none)" if u == "-" else "(browser-like)")] += 1
+                        if WEEKLY:
+                            # longest token first so Applebot-Extended wins over Applebot
+                            bot = next((b for b in sorted(BOT_VENDOR, key=len, reverse=True) if b in u), None)
+                            if bot:
+                                try:
+                                    addr = ipaddress.ip_address(parts[0])
+                                except ValueError:
+                                    addr = None
+                                if addr is None:
+                                    pass
+                                elif any(addr in n for n in CFNETS):
+                                    proxied[bot] += 1
+                                elif in_index(RANGES.get(BOT_VENDOR[bot]), addr):
+                                    verified[bot] += 1
+                                    ver_ips.setdefault(bot, set()).add(parts[0])
+                                    if len(parts) > 9:
+                                        record_verified(
+                                            bot, parts[6][:60], parts[8],
+                                            int(parts[9]) if parts[9].isdigit() else 0)
+                                else:
+                                    spoofed[bot] += 1
+                        if WEEKLY and RDNS_ENABLE:
+                            rb = next((b for b in sorted(BOT_RDNS, key=len, reverse=True)
+                                       if b in u), None)
+                            # skip if already counted by IP-range check
+                            if rb and not (rb in BOT_VENDOR and rb == bot):
+                                try:
+                                    a2 = ipaddress.ip_address(parts[0])
+                                except ValueError:
+                                    a2 = None
+                                if a2 is None:
+                                    pass
+                                elif any(a2 in n for n in CFNETS):
+                                    rdns_proxied[rb] += 1
+                                else:
+                                    rdns_hits[rb][parts[0]] += 1
+                                    if len(parts) > 9 and not a2.is_private:
+                                        rdns_paths[rb].setdefault(parts[0], []).append(
+                                            (parts[6][:60], parts[8],
+                                             int(parts[9]) if parts[9].isdigit() else 0))
+                    break
+
+total = sum(v[0] for v in per_day.values())
+all_ips = set().union(*(v[1] for v in per_day.values()))
+
+# --- resolve rDNS candidates ---------------------------------------------
+# One pass over unique addresses, busiest first, capped by RDNS_MAX so a
+# flood of distinct forgeries cannot stall the report on DNS timeouts.
+rdns_ok = Counter()       # bot -> forward-confirmed by PTR
+rdns_mismatch = Counter() # bot -> PTR exists but points elsewhere: forgery
+rdns_noptr = Counter()    # bot -> no PTR at all: cannot be judged by DNS
+rdns_asn = Counter()      # bot -> no PTR, but origin ASN is the vendor's
+rdns_skipped = Counter()  # bot -> hits left unresolved by the cap
+rdns_private = Counter()  # bot -> hits from RFC1918/loopback (see below)
+rdns_ok_ips = {}
+if WEEKLY and RDNS_ENABLE:
+    socket.setdefaulttimeout(3)
+    for bot, counter in rdns_hits.items():
+        budget = RDNS_MAX
+        for ip, hits in counter.most_common():
+            # A private source address means the real client IP never
+            # reached the log: something in front (Docker's bridge
+            # gateway, a local proxy) is rewriting it. Neither verified
+            # nor spoofed - the evidence simply is not there.
+            if ipaddress.ip_address(ip).is_private:
+                rdns_private[bot] += hits
+                continue
+            if budget <= 0:
+                rdns_skipped[bot] += hits
+                continue
+            budget -= 1
+            ok, _host, why = fcrdns(ip, BOT_RDNS[bot])
+            confirmed = False
+            if ok:
+                rdns_ok[bot] += hits
+                rdns_ok_ips.setdefault(bot, set()).add(ip)
+                confirmed = True
+            elif why == "no-ptr":
+                # Fall back to ASN, but only for vendors whose network is
+                # single-tenant (see BOT_ASN). Otherwise leave it unknown.
+                if ASN_ENABLE and bot in BOT_ASN and asn_of(ip) in BOT_ASN[bot]:
+                    rdns_asn[bot] += hits
+                    rdns_ok_ips.setdefault(bot, set()).add(ip)
+                    confirmed = True
+                else:
+                    rdns_noptr[bot] += hits
+            else:
+                rdns_mismatch[bot] += hits
+            if confirmed:
+                for p, st, nb in rdns_paths.get(bot, {}).get(ip, ()):
+                    record_verified(bot, p, st, nb)
+    socket.setdefaulttimeout(None)
+
+if not WEEKLY:
+    s0 = stamps[0]
+    count, ips = per_day[s0][0], len(per_day[s0][1])
+    print(f"{s0}: {count} requests, {ips} unique IPs (threshold {THRESHOLD})")
+    if DRY or count <= THRESHOLD:
+        sys.exit(0)
+    subject = f"{SITE}: {count} requests on {s0} (over {THRESHOLD}/day threshold)"
+    body = (f"Forgejo served {count} requests from {ips} unique IPs on {s0} - "
+            f"above the {THRESHOLD}/day watch threshold.\n\n"
+            "Time to consider Anubis (PoW challenge) in front of Forgejo.\n")
+else:
+    lines = [f"{SITE} weekly crawler report ({stamps[0]} - {stamps[-1]})", ""]
+    lines.append(f"Total: {total} requests, {len(all_ips)} unique IPs")
+    lines.append(f"Daily threshold alert fires above {THRESHOLD} req/day (none = quiet week)")
+    lines.append("")
+    lines.append("Per day:")
+    for s in stamps:
+        lines.append(f"  {s}: {per_day[s][0]:>7} requests, {len(per_day[s][1]):>6} unique IPs")
+    lines.append("")
+    lines.append("Top repos:")
+    for r, c in repos.most_common(5):
+        lines.append(f"  {c:>7}  {r}")
+    lines.append("")
+    lines.append("User-agent classes (browser-like = mostly the spoofing swarm):")
+    for u, c in uas.most_common(8):
+        lines.append(f"  {c:>7}  {u}")
+    lines.append("")
+
+    # --- legitimate (IP-verified) crawlers -------------------------------
+    lines.append("=" * 62)
+    lines.append("LEGITIMATE CRAWLERS (source IP inside vendor's published ranges)")
+    lines.append("=" * 62)
+    if STALE:
+        lines.append(f"NOTE: range fetch failed for {', '.join(STALE)} - used cache")
+    tv, ts, tp = sum(verified.values()), sum(spoofed.values()), sum(proxied.values())
+    claimed = tv + ts + tp
+    if not claimed:
+        lines.append("  No AI-bot user-agents seen this week.")
+    else:
+        lines.append(f"  {'bot':<20}{'verified':>9}{'spoofed':>9}{'via CF':>8}  {'IPs':>4}")
+        for bot in sorted(set(verified) | set(spoofed) | set(proxied),
+                          key=lambda b: -(verified[b] + spoofed[b] + proxied[b])):
+            lines.append(f"  {bot:<20}{verified[bot]:>9}{spoofed[bot]:>9}"
+                         f"{proxied[bot]:>8}  {len(ver_ips.get(bot, ())):>4}")
+        pct = tv / claimed * 100
+        lines.append("")
+        lines.append(f"  {claimed} requests claimed an AI-bot identity; {tv} verified ({pct:.1f}%).")
+        lines.append(f"  {ts} failed IP verification (spoofed). {tp} arrived via Cloudflare")
+        lines.append("  and cannot be verified by IP - not counted either way.")
+    # --- vendors verified by reverse DNS ---------------------------------
+    if RDNS_ENABLE and (rdns_ok or rdns_mismatch or rdns_noptr or rdns_asn
+                        or rdns_proxied or rdns_private):
+        lines.append("")
+        lines.append("-" * 62)
+        lines.append("VERIFIED BY DNS / ASN (vendors publishing no IP ranges)")
+        lines.append("-" * 62)
+        lines.append(f"  {'bot':<20}{'PTR ok':>7}{'ASN ok':>7}{'FORGED':>8}"
+                     f"{'no-PTR':>7}{'CF':>6}{'priv':>7}  {'IPs':>4}")
+        allbots = (set(rdns_ok) | set(rdns_mismatch) | set(rdns_noptr)
+                   | set(rdns_asn) | set(rdns_proxied) | set(rdns_private))
+        for bot in sorted(allbots, key=lambda b: -(rdns_ok[b] + rdns_mismatch[b]
+                                                   + rdns_noptr[b] + rdns_asn[b]
+                                                   + rdns_proxied[b] + rdns_private[b])):
+            lines.append(f"  {bot:<20}{rdns_ok[bot]:>7}{rdns_asn[bot]:>7}"
+                         f"{rdns_mismatch[bot]:>8}{rdns_noptr[bot]:>7}"
+                         f"{rdns_proxied[bot]:>6}{rdns_private[bot]:>7}"
+                         f"  {len(rdns_ok_ips.get(bot, ())):>4}")
+        if rdns_skipped:
+            sk = ", ".join(f"{b} {c}" for b, c in rdns_skipped.most_common(4))
+            lines.append(f"  (unresolved, over CRAWLER_RDNS_MAX={RDNS_MAX}: {sk})")
+        lines.append("")
+        lines.append("  PTR ok  forward-confirmed: PTR ends in a vendor domain AND")
+        lines.append("          that hostname resolves back to the same address.")
+        lines.append("  ASN ok  no PTR, but the address sits in an ASN belonging")
+        lines.append("          solely to the vendor (Meta's AS32934). Only used")
+        lines.append("          where the ASN sells no compute - never for clouds.")
+        lines.append("  FORGED  a PTR exists and points somewhere else. This is")
+        lines.append("          the only column that is evidence of forgery.")
+        lines.append("  no-PTR  no PTR record and no usable ASN: unknown, not bad.")
+        if sum(rdns_private.values()):
+            lines.append("  priv    logged source was RFC1918: a proxy or Docker")
+            lines.append("          bridge masked the client. Unjudgeable. See README.")
+
+    # --- what the verified crawlers actually fetched ---------------------
+    # Last, because it spans both tables above: an address only counts here
+    # once it has been confirmed, whether by IP range, PTR or ASN.
+    if ver_paths:
+        lines.append("")
+        lines.append("=" * 62)
+        lines.append("WHAT THE VERIFIED CRAWLERS DID (all methods)")
+        lines.append("=" * 62)
+        tot_v = sum(ver_paths.values())
+        mb = ver_bytes[0] / 1048576
+        lines.append(f"  {tot_v} requests, {mb:.1f} MB served.")
+        if ver_status:
+            mix = "  ".join(f"{s}:{c}" for s, c in ver_status.most_common(6))
+            lines.append(f"  Status: {mix}")
+        if ver_bot_bytes:
+            top = "  ".join(f"{b} {v/1048576:.1f}MB"
+                            for b, v in ver_bot_bytes.most_common(4))
+            lines.append(f"  Bandwidth: {top}")
+        lines.append("")
+        if not ROBOTS_RULES:
+            lines.append("  robots.txt compliance: not checked "
+                         "(set CRAWLER_ROBOTS to enable)")
+        elif ver_violations:
+            n = sum(ver_violations.values())
+            lines.append(f"  ROBOTS.TXT VIOLATIONS: {n} requests to disallowed paths")
+            for p, c in ver_violations.most_common(6):
+                lines.append(f"    {c:>6}  {p}")
+            lines.append("  A verified crawler ignoring robots.txt is a policy")
+            lines.append("  problem, not a security one - the identity is genuine.")
+        else:
+            lines.append(f"  robots.txt compliance: clean, 0 violations in "
+                         f"{tot_v} requests")
+        lines.append("")
+        # Repository content vs everything else. If robots.txt permits
+        # /src/ and /raw/, this is what says whether anyone took it up.
+        nc = sum(ver_content.values())
+        pct_c = nc / tot_v * 100 if tot_v else 0
+        lines.append(f"  Repository content fetched: {nc} of {tot_v} requests "
+                     f"({pct_c:.1f}%)")
+        if nc:
+            by_bot = "  ".join(f"{b} {c}" for b, c in ver_content.most_common(5))
+            lines.append(f"    by: {by_bot}")
+            for p, c in ver_content_paths.most_common(8):
+                lines.append(f"    {c:>6}  {p}")
+        else:
+            lines.append("    none - crawlers indexed the site but read no code")
+        # Which repositories each vendor cares about. Grouped by vendor
+        # rather than by UA token: OpenAI alone sends three tokens, and
+        # "what is OpenAI reading" is the useful question.
+        if ver_repos:
+            groups = {}
+            for b, c in ver_repos.items():
+                if b in BOT_VENDOR:
+                    v = BOT_VENDOR[b]                  # openai, anthropic, ...
+                elif b in BOT_ASN:
+                    v = "meta"
+                elif b in BOT_RDNS:
+                    v = b.lower()                      # amazonbot, youbot, ...
+                else:
+                    v = b.lower()
+                groups.setdefault(v, Counter()).update(c)
+            lines.append("")
+            lines.append("  Repositories by vendor:")
+            for v in sorted(groups, key=lambda x: -sum(groups[x].values())):
+                tot = sum(groups[v].values())
+                top = ", ".join(f"{r} ({n})" for r, n in groups[v].most_common(3))
+                lines.append(f"    {v:<12} {tot:>5} hits  {top}")
+        if ver_pkgs or ver_pkg_index:
+            np = sum(ver_pkgs.values()) + sum(ver_pkg_index.values())
+            lines.append("")
+            lines.append(f"  Package registry: {np} requests")
+            for p, c in ver_pkgs.most_common(6):
+                lines.append(f"    {c:>6}  {p}")
+            if ver_pkg_index:
+                idx = ", ".join(f"{o} ({n})" for o, n in ver_pkg_index.most_common(4))
+                lines.append(f"    index pages: {idx}")
+        lines.append("")
+        lines.append("  Top paths overall:")
+        for p, c in ver_paths.most_common(12):
+            lines.append(f"    {c:>6}  {p}")
+    body = "\n".join(lines) + "\n"
+    subject = f"{SITE} weekly crawler report: {total} requests, {len(all_ips)} IPs"
+    if DRY:
+        print(subject); print(); print(body)
+        sys.exit(0)
+
+# SMTP settings: use CRAWLER_SMTP_* if given, otherwise borrow them from
+# a Forgejo/Gitea app.ini [mailer] section so there is no second copy of
+# the credentials to keep in sync.
+if os.environ.get("CRAWLER_SMTP_ADDR"):
+    addr = os.environ["CRAWLER_SMTP_ADDR"]
+    port = int(os.environ.get("CRAWLER_SMTP_PORT", "587"))
+    user = os.environ.get("CRAWLER_SMTP_USER", "")
+    passwd = os.environ.get("CRAWLER_SMTP_PASSWORD", "")
+    sender = os.environ.get("CRAWLER_SMTP_FROM", user or RECIPIENT)
+    proto = os.environ.get("CRAWLER_SMTP_PROTOCOL", "smtp+starttls")
+else:
+    ini = subprocess.run(["docker", "exec", MAIL_CONTAINER, "cat", MAIL_INI],
+                         capture_output=True, text=True).stdout
+    if not ini.strip():
+        sys.exit(f"could not read {MAIL_INI} from container "
+                 f"'{MAIL_CONTAINER}' and CRAWLER_SMTP_ADDR is unset")
+    cp = configparser.ConfigParser(interpolation=None, strict=False)
+    cp.read_string("[DEFAULT]\n" + ini)
+    m = cp["mailer"]
+    proto = m.get("PROTOCOL", "smtps").strip()
+    addr, port = m.get("SMTP_ADDR").strip(), int(m.get("SMTP_PORT", "465").strip())
+    user, passwd = m.get("USER", "").strip(), m.get("PASSWD", "").strip().strip("`\"")
+    sender = m.get("FROM", user).strip()
+
+msg = MIMEText(body)
+msg["Subject"], msg["From"], msg["To"] = subject, sender, RECIPIENT
+if proto == "smtps":
+    s = smtplib.SMTP_SSL(addr, port, timeout=30)
+else:
+    s = smtplib.SMTP(addr, port, timeout=30)
+    if proto != "smtp":          # plain smtp = no TLS (local relay)
+        s.starttls()
+if user:                          # unauthenticated local relays exist
+    s.login(user, passwd)
+s.sendmail(sender, [RECIPIENT], msg.as_string())
+s.quit()
+print(f"mailed to {RECIPIENT}")

ansible/roles/crawler_alert/files/crawler-alert.service

@@ -0,0 +1,8 @@
+[Unit]
+Description=Daily crawler threshold alert
+After=docker.service
+
+[Service]
+Type=oneshot
+EnvironmentFile=/etc/crawler-alert.env
+ExecStart=/usr/local/bin/crawler-alert.py

ansible/roles/crawler_alert/files/crawler-alert.timer

@@ -0,0 +1,9 @@
+[Unit]
+Description=Daily Forgejo crawler volume check
+
+[Timer]
+OnCalendar=*-*-* 00:15:00
+Persistent=true
+
+[Install]
+WantedBy=timers.target

ansible/roles/crawler_alert/files/crawler-weekly.service

@@ -0,0 +1,8 @@
+[Unit]
+Description=Weekly crawler report mail
+After=docker.service
+
+[Service]
+Type=oneshot
+EnvironmentFile=/etc/crawler-alert.env
+ExecStart=/usr/local/bin/crawler-alert.py --weekly

ansible/roles/crawler_alert/files/crawler-weekly.timer

@@ -0,0 +1,9 @@
+[Unit]
+Description=Weekly Forgejo crawler report, Mondays 12:00 UTC
+
+[Timer]
+OnCalendar=Mon *-*-* 12:00:00
+Persistent=true
+
+[Install]
+WantedBy=timers.target

ansible/roles/crawler_alert/handlers/main.yaml

@@ -0,0 +1,4 @@
+---
+- name: Reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true

ansible/roles/crawler_alert/tasks/main.yaml

@@ -0,0 +1,37 @@
+---
+- name: Install crawler-alert script
+  ansible.builtin.copy:
+    src: crawler-alert.py
+    dest: /usr/local/bin/crawler-alert.py
+    mode: "0755"
+
+- name: Configure crawler-alert
+  ansible.builtin.template:
+    src: crawler-alert.env.j2
+    dest: /etc/crawler-alert.env
+    mode: "0600"
+  diff: false
+
+- name: Install crawler-alert units
+  ansible.builtin.copy:
+    src: "{{ item }}"
+    dest: "/etc/systemd/system/{{ item }}"
+    mode: "0644"
+  loop:
+    - crawler-alert.service
+    - crawler-alert.timer
+    - crawler-weekly.service
+    - crawler-weekly.timer
+  notify: Reload systemd
+
+- name: Flush handlers
+  ansible.builtin.meta: flush_handlers
+
+- name: Enable crawler-alert timers
+  ansible.builtin.systemd:
+    name: "{{ item }}"
+    enabled: true
+    state: started
+  loop:
+    - crawler-alert.timer
+    - crawler-weekly.timer

ansible/roles/crawler_alert/templates/crawler-alert.env.j2

@@ -0,0 +1,8 @@
+CRAWLER_RECIPIENT={{ crawler_recipient }}
+CRAWLER_SITE={{ crawler_site }}
+CRAWLER_LOG=/opt/traefik/logs/access.log*
+CRAWLER_ROUTER=forgejo@docker
+CRAWLER_THRESHOLD={{ crawler_threshold }}
+CRAWLER_MAIL_CONTAINER=forgejo
+CRAWLER_MAIL_INI=/data/gitea/conf/app.ini
+CRAWLER_ROBOTS={{ crawler_robots_url }}

ansible/roles/docker_service/defaults/main.yaml

@@ -1,34 +1,28 @@
 ---
-# Default variables for docker_service role
-# These can be overridden when including this role
+service_name: ""        # required, also the compose project name
+service_base_path: ""   # required, e.g. /opt/traefik
 
-# Service configuration
-service_name: ""  # REQUIRED: Name of the service (e.g., "traefik", "forgejo")
-service_base_path: ""  # REQUIRED: Base installation path (e.g., "/opt/traefik")
-
-# Directories to create under service_base_path
-# Format: list of dicts with 'path' and 'mode' keys
+# Each list item: path/dest plus optional owner, group, mode. Ownership and
+# mode are only enforced when given, so existing data directories keep theirs.
 service_directories: []
-# Example:
-#   - { path: "data", mode: "0755" }
-#   - { path: "config", mode: "0644" }
+service_templates: []   # src (template), dest
+service_files: []       # src (plain file, copied verbatim), dest
+service_absent: []      # paths under the base path that must not exist
 
-# Template files to deploy
-# Format: list of dicts with 'src', 'dest', and optional 'mode' keys
-service_templates: []
-# Example:
-#   - { src: "docker-compose.yml.j2", dest: "docker-compose.yml", mode: "0644" }
-#   - { src: "config.yaml.j2", dest: "config/config.yaml", mode: "0600" }
+# Containers to save and stop properly before compose recreates them:
+# - { container: name, timeout: 120, pre_stop: "docker exec ..." }
+service_graceful_stop: []
 
-# Docker compose configuration
-docker_compose_file: "docker-compose.yml"  # Name of the compose file
-docker_compose_project_name: "{{ service_name }}"  # Docker compose project name
+docker_compose_file: "docker-compose.yml"
+docker_compose_project_name: "{{ service_name }}"
 
-# Health check configuration
-health_check_enabled: true  # Whether to perform health check after deployment
-health_check_url: ""  # URL to check (empty = skip HTTP check)
-health_check_host: "localhost"  # Host to check port on (default: localhost)
-health_check_port: ""  # Port to check (empty = skip port check)
-health_check_delay: 10  # Seconds to wait before health check
-health_check_retries: 30  # Number of retries for health check
-health_check_retry_delay: 2  # Seconds between retries
+# A changed config file restarts the stack unless compose already recreated it
+service_restart_on_change: true
+
+health_check_enabled: true
+health_check_url: ""
+health_check_host: "localhost"
+health_check_port: ""
+health_check_delay: 10
+health_check_retries: 30
+health_check_retry_delay: 2

ansible/roles/docker_service/tasks/main.yaml

@@ -1,46 +1,94 @@
 ---
-# Shared tasks for deploying Docker-based services
-# This role provides a standardized pattern for service deployment
-
 - name: Validate required variables
   ansible.builtin.assert:
     that:
-      - service_name is defined
       - service_name | length > 0
-      - service_base_path is defined
       - service_base_path | length > 0
-    fail_msg: "service_name and service_base_path are required variables"
-    success_msg: "Required variables validated for {{ service_name }}"
+    fail_msg: "service_name and service_base_path are required"
+    quiet: true
 
 - name: Create base service directory
   ansible.builtin.file:
     path: "{{ service_base_path }}"
     state: directory
-    mode: '0755'
-    owner: root
-    group: root
 
 - name: Create service subdirectories
   ansible.builtin.file:
     path: "{{ service_base_path }}/{{ item.path }}"
     state: directory
-    mode: "{{ item.mode | default('0755') }}"
-    owner: root
-    group: root
+    owner: "{{ item.owner | default(omit) }}"
+    group: "{{ item.group | default(omit) }}"
+    mode: "{{ item.mode | default(omit) }}"
   loop: "{{ service_directories }}"
-  when: service_directories | length > 0
+  loop_control:
+    label: "{{ item.path }}"
 
-- name: Deploy service template files
+- name: Deploy service templates
   ansible.builtin.template:
     src: "{{ item.src }}"
     dest: "{{ service_base_path }}/{{ item.dest }}"
+    owner: "{{ item.owner | default('root') }}"
+    group: "{{ item.group | default('root') }}"
     mode: "{{ item.mode | default('0644') }}"
-    owner: root
-    group: root
   loop: "{{ service_templates }}"
-  when: service_templates | length > 0
+  loop_control:
+    label: "{{ item.dest }}"
+  # Templates may carry secrets; never print their contents
+  diff: false
   register: template_result
 
+- name: Deploy service files
+  ansible.builtin.copy:
+    src: "{{ item.src }}"
+    dest: "{{ service_base_path }}/{{ item.dest }}"
+    owner: "{{ item.owner | default('root') }}"
+    group: "{{ item.group | default('root') }}"
+    mode: "{{ item.mode | default('0644') }}"
+  loop: "{{ service_files }}"
+  loop_control:
+    label: "{{ item.dest }}"
+  register: file_result
+
+- name: Remove files that are no longer part of the service
+  ansible.builtin.file:
+    path: "{{ service_base_path }}/{{ item }}"
+    state: absent
+  loop: "{{ service_absent }}"
+
+# Game servers need time to save: Docker's default stop timeout is 10 s.
+# When compose is about to recreate one of the listed containers, save
+# and stop it properly first; compose then replaces the stopped container.
+- name: Ask compose what it would do
+  ansible.builtin.command:
+    cmd: docker compose -p {{ docker_compose_project_name }} -f {{ docker_compose_file }} up -d --dry-run
+    chdir: "{{ service_base_path }}"
+  register: compose_plan
+  changed_when: false
+  # The dry run exits non-zero when an image still has to be pulled; the
+  # plan it prints up to that point is all that is needed here
+  failed_when: false
+  check_mode: false
+  when: service_graceful_stop | length > 0
+
+- name: Run pre-stop command before recreation
+  ansible.builtin.command:
+    cmd: "{{ item.pre_stop }}"
+  loop: "{{ service_graceful_stop }}"
+  loop_control:
+    label: "{{ item.container }}"
+  when:
+    - item.pre_stop is defined
+    - (compose_plan.stdout ~ compose_plan.stderr) is search('Container ' ~ item.container ~ ' Recreate')
+  failed_when: false
+
+- name: Stop gracefully before recreation
+  ansible.builtin.command:
+    cmd: "docker stop -t {{ item.timeout | default(120) }} {{ item.container }}"
+  loop: "{{ service_graceful_stop }}"
+  loop_control:
+    label: "{{ item.container }}"
+  when: (compose_plan.stdout ~ compose_plan.stderr) is search('Container ' ~ item.container ~ ' Recreate')
+
 - name: Start service with Docker Compose
   community.docker.docker_compose_v2:
     project_src: "{{ service_base_path }}"
@@ -49,33 +97,45 @@
       - "{{ docker_compose_file }}"
     state: present
   register: compose_result
+  # In check mode compose cannot pull, so a missing image looks like an error
+  ignore_errors: "{{ ansible_check_mode }}"
+
+- name: Restart service after a config change
+  community.docker.docker_compose_v2:
+    project_src: "{{ service_base_path }}"
+    project_name: "{{ docker_compose_project_name }}"
+    files:
+      - "{{ docker_compose_file }}"
+    state: restarted
+  when:
+    - service_restart_on_change
+    - template_result is changed or file_result is changed
+    - compose_result is not changed
 
 - name: Wait before health check
   ansible.builtin.pause:
     seconds: "{{ health_check_delay }}"
-  when: health_check_enabled and (health_check_url != "" or health_check_port != "")
+  when:
+    - health_check_enabled
+    - health_check_url != "" or health_check_port != ""
+    - compose_result is changed or template_result is changed or file_result is changed
 
 - name: Wait for service HTTP endpoint
   ansible.builtin.uri:
     url: "{{ health_check_url }}"
-    status_code: [200, 301, 302, 401, 404]  # Accept various codes (service-dependent)
+    status_code: [200, 301, 302, 303, 401, 404]
+    follow_redirects: none
     timeout: 5
   register: health_result
   until: health_result is not failed
   retries: "{{ health_check_retries }}"
   delay: "{{ health_check_retry_delay }}"
   when: health_check_enabled and health_check_url != ""
-  failed_when: false
+  check_mode: false
 
 - name: Wait for service port
   ansible.builtin.wait_for:
     host: "{{ health_check_host }}"
     port: "{{ health_check_port }}"
-    delay: 0
     timeout: "{{ health_check_retries * health_check_retry_delay }}"
-    state: started
   when: health_check_enabled and health_check_port != ""
-
-- name: Display deployment status
-  ansible.builtin.debug:
-    msg: "{{ service_name }} deployment completed successfully"

ansible/roles/forgejo/tasks/main.yaml

@@ -5,19 +5,22 @@
 - name: Configure SSH passthrough
   ansible.builtin.import_tasks: ssh-passthrough.yaml
 
-- name: Deploy Forgejo using shared docker_service role
+- name: Deploy Forgejo
   ansible.builtin.include_role:
     name: docker_service
   vars:
     service_name: "forgejo"
     service_base_path: "{{ forgejo_base_path }}"
     service_directories:
-      - { path: "data", mode: "0755" }
-      - { path: "config", mode: "0755" }
+      - { path: "data" }
+      - { path: "config" }
     service_templates:
-      - { src: "app.ini.j2", dest: "config/app.ini", mode: "0644" }
-      - { src: "docker-compose.yml.j2", dest: "docker-compose.yml", mode: "0644" }
-    health_check_enabled: true
-    health_check_url: "http://localhost:{{ forgejo_http_port }}/"
+      - src: "app.ini.j2"
+        dest: "config/app.ini"
+        owner: "{{ forgejo_uid }}"
+        group: "{{ forgejo_gid }}"
+        mode: "0600"
+      - { src: "docker-compose.yml.j2", dest: "docker-compose.yml" }
+    health_check_url: "http://localhost:{{ forgejo_http_port }}/api/v1/version"
     health_check_delay: 5
-    health_check_retries: 15
+    health_check_retries: 30

ansible/roles/forgejo/tasks/ssh-passthrough.yaml

@@ -1,45 +1,32 @@
 ---
+# git@<host> lands on the host's sshd, which hands the session to the
+# Forgejo container. Keys are looked up through the container too
+# (AuthorizedKeysCommand), so the host keeps no authorized_keys for git.
+
 - name: Create git user on host
-  user:
+  ansible.builtin.user:
     name: git
     comment: "Forgejo Git User"
-    home: /var/lib/git
+    home: "{{ forgejo_base_path }}/data/git"
+    create_home: false
     shell: /usr/local/bin/forgejo-shell
-    system: yes
-    create_home: yes
+    system: true
+    groups: docker
+    append: true
 
 - name: Deploy forgejo-shell wrapper script
-  template:
+  ansible.builtin.template:
     src: forgejo-shell.j2
     dest: /usr/local/bin/forgejo-shell
     owner: root
     group: root
-    mode: '0755'
-
-- name: Create .ssh directory for git user
-  file:
-    path: /var/lib/git/.ssh
-    state: directory
-    owner: git
-    group: git
-    mode: '0700'
+    mode: "0755"
 
 - name: Deploy SSH daemon configuration for Forgejo passthrough
-  template:
+  ansible.builtin.template:
     src: sshd-forgejo.conf.j2
     dest: /etc/ssh/sshd_config.d/forgejo.conf
     owner: root
     group: root
-    mode: '0644'
+    mode: "0644"
   notify: Reload SSH daemon
-
-- name: Ensure docker group exists
-  group:
-    name: docker
-    state: present
-
-- name: Add git user to docker group
-  user:
-    name: git
-    groups: docker
-    append: yes

ansible/roles/forgejo/templates/app.ini.j2

@@ -1,5 +1,6 @@
-APP_NAME = Gitea: Git with a cup of tea together with asxpi
+APP_NAME = Git with a cup of tea
 RUN_MODE = prod
+WORK_PATH = /data/gitea
 
 [server]
 PROTOCOL = http
@@ -13,10 +14,12 @@ SSH_LISTEN_PORT = 22
 START_SSH_SERVER = false
 LFS_START_SERVER = true
 OFFLINE_MODE = false
+LFS_JWT_SECRET = {{ forgejo_server_lfs_jwt_secret }}
+LANDING_PAGE = explore
 
 [database]
 DB_TYPE = sqlite3
-PATH = /data/gitea.db
+PATH = /data/gitea/gitea.db
 LOG_SQL = false
 
 [repository]
@@ -26,7 +29,7 @@ DEFAULT_BRANCH = main
 DEFAULT_PRIVATE = private
 ENABLE_PUSH_CREATE_USER = true
 ENABLE_PUSH_CREATE_ORG = false
-DISABLED_REPO_UNITS =
+DISABLED_REPO_UNITS = 
 DEFAULT_REPO_UNITS = repo.code,repo.releases,repo.issues,repo.pulls,repo.wiki,repo.projects,repo.packages,repo.actions
 
 [ui]
@@ -35,15 +38,15 @@ DEFAULT_THEME = forgejo-auto
 
 [security]
 INSTALL_LOCK = true
-SECRET_KEY = {{ forgejo_secret_key | default('CHANGE-ME-PLEASE-GENERATE-NEW-SECRET') }}
-INTERNAL_TOKEN = {{ forgejo_internal_token | default('CHANGE-ME-PLEASE-GENERATE-NEW-TOKEN') }}
+SECRET_KEY = {{ forgejo_secret_key }}
+INTERNAL_TOKEN = {{ forgejo_internal_token }}
 PASSWORD_HASH_ALGO = pbkdf2
 
 [service]
 DISABLE_REGISTRATION = true
 REQUIRE_SIGNIN_VIEW = false
 REGISTER_EMAIL_CONFIRM = false
-ENABLE_NOTIFY_MAIL = false
+ENABLE_NOTIFY_MAIL = true
 DEFAULT_KEEP_EMAIL_PRIVATE = true
 DEFAULT_ALLOW_CREATE_ORGANIZATION = true
 DEFAULT_ENABLE_TIMETRACKING = true
@@ -51,10 +54,16 @@ NO_REPLY_ADDRESS = noreply.{{ forgejo_domain }}
 
 [oauth2]
 ENABLED = true
-JWT_SECRET = {{ forgejo_jwt_secret | default('CHANGE-ME-PLEASE-GENERATE-NEW-JWT-SECRET') }}
+JWT_SECRET = {{ forgejo_jwt_secret }}
 
 [mailer]
-ENABLED = false
+ENABLED = true
+FROM = Czechoslovakia Git <noreply@asxp.io>
+PROTOCOL = smtp+starttls
+SMTP_ADDR = smtp.fastmail.com
+SMTP_PORT = 587
+USER = {{ forgejo_smtp_user }}
+PASSWD = {{ forgejo_smtp_password }}
 
 [session]
 PROVIDER = file
@@ -80,4 +89,13 @@ DEFAULT_TRUST_MODEL = collaborator
 
 [lfs]
 PATH = /data/git/lfs
-LFS_JWT_SECRET = {{ forgejo_lfs_jwt_secret | default('CHANGE-ME-PLEASE-GENERATE-NEW-LFS-JWT-SECRET') }}
+LFS_JWT_SECRET = {{ forgejo_lfs_jwt_secret }}
+
+[packages]
+ENABLED = true
+
+[other]
+SHOW_FOOTER_VERSION = false
+SHOW_FOOTER_TEMPLATE_LOAD_TIME = false
+SHOW_FOOTER_POWERED_BY = false
+

ansible/roles/forgejo/templates/docker-compose.yml.j2

@@ -19,7 +19,7 @@ services:
       - {{ traefik_network }}
     labels:
       - "traefik.enable=true"
-      - "traefik.http.routers.forgejo.rule=Host(`{{ forgejo_domain }}`)"
+      - "traefik.http.routers.forgejo.rule={{ ([forgejo_domain] + forgejo_alias_domains) | map('regex_replace', '^(.*)$', 'Host(`\\1`)') | join(' || ') }}"
       - "traefik.http.routers.forgejo.entrypoints=websecure"
       - "traefik.http.routers.forgejo.tls=true"
       - "traefik.http.routers.forgejo.tls.certresolver=letsencrypt"

ansible/roles/forgejo_backup/tasks/main.yaml

@@ -1,102 +0,0 @@
----
-- name: Create backup directory on ops server
-  file:
-    path: "{{ backup_dest_dir }}"
-    state: directory
-    mode: '0755'
-    owner: asxpi
-    group: asxpi
-  delegate_to: asxpio-ops
-  become: true
-
-- name: Find existing Forgejo dump files on old server
-  find:
-    paths: /srv/gitea
-    patterns: "*.zip"
-    file_type: file
-  register: existing_dumps
-
-- name: Create temporary staging directory for backup
-  file:
-    path: /tmp/forgejo-backup-staging
-    state: directory
-    mode: '0755'
-  become: true
-
-- name: Copy database to staging
-  copy:
-    src: /srv/gitea/gitea.db
-    dest: /tmp/forgejo-backup-staging/gitea.db
-    remote_src: true
-  become: true
-
-- name: Create data directory in staging
-  file:
-    path: /tmp/forgejo-backup-staging/data
-    state: directory
-    mode: '0755'
-  become: true
-
-- name: Copy data directories to staging
-  copy:
-    src: "/srv/gitea/data/{{ item }}/"
-    dest: "/tmp/forgejo-backup-staging/data/{{ item }}/"
-    remote_src: true
-  loop:
-    - avatars
-    - repo-avatars
-    - repo-archive
-    - attachments
-    - gitea-repositories
-  become: true
-  ignore_errors: true
-
-- name: Create backup archive from staging
-  archive:
-    path: /tmp/forgejo-backup-staging
-    dest: /tmp/forgejo-full-backup-{{ ansible_date_time.date }}.tar.gz
-    format: gz
-  become: true
-
-- name: Clean up staging directory
-  file:
-    path: /tmp/forgejo-backup-staging
-    state: absent
-  become: true
-
-- name: Display found backup files
-  debug:
-    msg: |
-      Found {{ existing_dumps.files | length }} dump files
-      Created full backup archive: /tmp/forgejo-full-backup-{{ ansible_date_time.date }}.tar.gz
-
-- name: Fetch dump file to ops server (if exists)
-  synchronize:
-    mode: pull
-    src: "{{ (existing_dumps.files | sort(attribute='mtime', reverse=true) | first).path }}"
-    dest: /opt/forgejo-backup/
-  delegate_to: asxpio-ops
-  when: existing_dumps.files | length > 0
-  ignore_errors: true
-
-- name: Fetch full backup archive to ops server
-  synchronize:
-    mode: pull
-    src: /tmp/forgejo-full-backup-{{ ansible_date_time.date }}.tar.gz
-    dest: /opt/forgejo-backup/
-  delegate_to: asxpio-ops
-  become: true
-
-- name: Display backup location
-  debug:
-    msg: |
-      Backup files transferred to asxpio-ops:/opt/forgejo-backup/
-
-      Files:
-      - forgejo-full-backup-{{ ansible_date_time.date }}.tar.gz (complete backup)
-      {% if existing_dumps.files | length > 0 %}
-      - {{ (existing_dumps.files | sort(attribute='mtime', reverse=true) | first).path | basename }} (dump file)
-      {% endif %}
-
-      Next step: Run restore playbook
-      ansible-playbook playbooks/restore-forgejo.yaml

ansible/roles/forgejo_restore/tasks/main.yaml

@@ -1,142 +0,0 @@
----
-- name: Find backup files on server
-  find:
-    paths: "{{ backup_dest_dir }}"
-    patterns: "forgejo-full-backup-*.tar.gz"
-  register: server_backup_files
-
-- name: Fail if no backup files found
-  fail:
-    msg: "No backup files found in {{ backup_dest_dir }}. Please run backup-forgejo.yaml first."
-  when: server_backup_files.files | length == 0
-
-- name: Select most recent backup file
-  set_fact:
-    backup_file: "{{ (server_backup_files.files | sort(attribute='mtime', reverse=true) | first).path }}"
-
-- name: Display backup file to be restored
-  debug:
-    msg: "Restoring from: {{ backup_file }}"
-
-- name: Stop Forgejo container
-  community.docker.docker_compose_v2:
-    project_src: /opt/forgejo
-    state: stopped
-  ignore_errors: true
-
-- name: Backup current Forgejo data (if exists)
-  archive:
-    path: /opt/forgejo/data
-    dest: "/opt/forgejo/data-backup-{{ ansible_date_time.epoch }}.tar.gz"
-    format: gz
-  ignore_errors: true
-
-- name: Remove old data directory
-  file:
-    path: /opt/forgejo/data
-    state: absent
-
-- name: Create fresh data directory structure
-  file:
-    path: "{{ item.path }}"
-    state: directory
-    owner: 1000
-    group: 1000
-    mode: "{{ item.mode | default('0755') }}"
-  loop:
-    - { path: '/opt/forgejo/data' }
-    - { path: '/opt/forgejo/data/gitea' }
-    - { path: '/opt/forgejo/data/gitea/data' }
-    - { path: '/opt/forgejo/data/git' }
-    - { path: '/opt/forgejo/data/git/repositories' }
-    - { path: '/opt/forgejo/data/git/.ssh', mode: '0700' }
-
-- name: Create temporary restore directory
-  file:
-    path: /tmp/forgejo-restore
-    state: directory
-    mode: '0755'
-
-- name: Extract backup to temporary location
-  unarchive:
-    src: "{{ backup_file }}"
-    dest: /tmp/forgejo-restore/
-    remote_src: true
-
-- name: Set backup source path
-  set_fact:
-    backup_source: "/tmp/forgejo-restore/forgejo-backup-staging"
-
-- name: Copy database from backup
-  copy:
-    src: "{{ backup_source }}/gitea.db"
-    dest: /opt/forgejo/data/gitea/gitea.db
-    remote_src: true
-    owner: 1000
-    group: 1000
-    mode: '0644'
-
-- name: Copy data directories from backup
-  ansible.builtin.copy:
-    src: "{{ backup_source }}/data/{{ item.src }}/"
-    dest: "{{ forgejo_base_path | default('/opt/forgejo') }}/data/{{ item.dest }}/"
-    remote_src: true
-    owner: 1000
-    group: 1000
-    mode: preserve
-  loop:
-    - { src: "avatars", dest: "gitea/data/avatars" }
-    - { src: "repo-avatars", dest: "gitea/data/repo-avatars" }
-    - { src: "repo-archive", dest: "gitea/data/repo-archive" }
-    - { src: "attachments", dest: "gitea/data/attachments" }
-    - { src: "gitea-repositories", dest: "git/repositories" }
-  ignore_errors: "{{ item.src != 'gitea-repositories' }}"  # Only fail on repositories copy failure
-
-- name: Set correct ownership for all Forgejo data
-  ansible.builtin.file:
-    path: "{{ forgejo_base_path | default('/opt/forgejo') }}/data"
-    owner: 1000
-    group: 1000
-    recurse: true
-
-- name: Start Forgejo container
-  community.docker.docker_compose_v2:
-    project_src: "{{ forgejo_base_path | default('/opt/forgejo') }}"
-    state: present
-
-- name: Wait for Forgejo to be healthy
-  wait_for:
-    host: localhost
-    port: "{{ forgejo_http_port }}"
-    timeout: 60
-    delay: 5
-
-- name: Clean up temporary files
-  file:
-    path: /tmp/forgejo-restore
-    state: absent
-
-- name: Verify users in database
-  ansible.builtin.command: docker exec -u git forgejo forgejo admin user list
-  register: user_list
-  changed_when: false
-  ignore_errors: true
-
-- name: Display restore information
-  debug:
-    msg: |
-      Forgejo restore completed successfully!
-
-      Users:
-      {{ user_list.stdout_lines | join('\n      ') }}
-
-      Forgejo is now running at:
-      - Internal: http://{{ asxpio_ops_wireguard_ip }}:{{ forgejo_http_port }}
-      - External: https://{{ forgejo_domain }}
-
-      Next steps:
-      1. Test web interface: https://{{ forgejo_domain }}
-      2. Verify avatars and images are visible
-      3. Test login with your credentials
-      4. Test git clone: git clone https://{{ forgejo_domain }}/user/repo.git
-      5. Check logs: docker logs forgejo

ansible/roles/forgejo_runner/tasks/main.yaml

@@ -9,12 +9,12 @@
     service_name: "forgejo-runner"
     service_base_path: "{{ forgejo_runner_base_path }}"
     service_directories:
-      - { path: "data", mode: "0755" }
-      - { path: "config", mode: "0755" }
+      - { path: "data" }
+      - { path: "config" }
     service_templates:
-      - { src: "config.yaml.j2", dest: "config/config.yaml", mode: "0644" }
-      - { src: "docker-compose.yml.j2", dest: "docker-compose.yml", mode: "0644" }
-    health_check_enabled: false  # Runner doesn't expose HTTP port
+      - { src: "config.yaml.j2", dest: "config/config.yaml" }
+      - { src: "docker-compose.yml.j2", dest: "docker-compose.yml" }
+    health_check_enabled: false  # the runner exposes no port
 
 - name: Check if runner is already registered
   ansible.builtin.stat:

ansible/roles/minecraft/files/docker-compose.yml

@@ -0,0 +1,15 @@
+services:
+  minecraft:
+    image: eclipse-temurin:21-jre
+    container_name: minecraft
+    working_dir: /data
+    command: java -Xmx4G -Xms2G -jar paper-1.21.11-54.jar nogui
+    stdin_open: true
+    tty: true
+    restart: unless-stopped
+    stop_grace_period: 2m
+    ports:
+      - "25565:25565"
+      - "19132:19132/udp"
+    volumes:
+      - /opt/minecraft:/data

ansible/roles/minecraft/tasks/main.yaml

@@ -0,0 +1,15 @@
+---
+# Paper server; the world and the server jar live next to the compose
+# file in /opt/minecraft and are not managed here. A compose change
+# recreates the container, so it is stopped with time to save first.
+- name: Deploy Minecraft
+  ansible.builtin.include_role:
+    name: docker_service
+  vars:
+    service_name: "minecraft"
+    service_base_path: "{{ services_base_path }}/minecraft"
+    service_files:
+      - { src: "docker-compose.yml", dest: "docker-compose.yml" }
+    service_graceful_stop:
+      - { container: minecraft, timeout: 120 }
+    health_check_enabled: false

ansible/roles/minecraft_fossils/files/docker-compose.yml

@@ -0,0 +1,87 @@
+services:
+  minecraft-fossils:
+    image: itzg/minecraft-server:java21
+    container_name: minecraft-fossils
+    restart: unless-stopped
+    stop_grace_period: 2m
+    stdin_open: true
+    tty: true
+    ports:
+      - "25566:25565"
+    env_file:
+      - secrets.env
+    environment:
+      EULA: "TRUE"
+      TYPE: FABRIC
+      VERSION: "1.20.1"
+      FABRIC_LOADER_VERSION: "0.18.4"
+      MEMORY: "8G"
+      ONLINE_MODE: "FALSE"
+      ENFORCE_WHITELIST: "TRUE"
+      ENFORCE_SECURE_PROFILE: "FALSE"
+      WHITELIST_FILE: /data/whitelist.json
+      OPS_FILE: /data/ops.json
+      ENABLE_RCON: "TRUE"
+      MOTD: "[mc.czsk.dev] Fossils Archaeology Revival: Genesis"
+      USE_AIKAR_FLAGS: "TRUE"
+    volumes:
+      - /opt/minecraft-fossils/data:/data
+    networks:
+      - fossils
+
+  whitelist-sync:
+    image: alpine:3
+    container_name: minecraft-fossils-wlsync
+    restart: unless-stopped
+    depends_on:
+      - minecraft-fossils
+    volumes:
+      - /opt/minecraft:/src:ro
+      - /opt/minecraft-fossils/data:/dst
+    env_file:
+      - secrets.env
+    environment:
+      RCON_HOST: minecraft-fossils
+      INTERVAL: "15"
+    command:
+      - sh
+      - -c
+      - |
+        apk add --no-cache python3 >/dev/null
+        cat > /rcon.py <<'PY'
+        import socket, struct, sys, os
+        host, port, pw, cmd = os.environ["RCON_HOST"], 25575, os.environ["RCON_PASSWORD"], sys.argv[1]
+        def pkt(i, t, b): p = struct.pack("<ii", i, t) + b.encode() + b"\x00\x00"; return struct.pack("<i", len(p)) + p
+        def recv(s):
+            ln = struct.unpack("<i", s.recv(4))[0]; d = b""
+            while len(d) < ln: d += s.recv(ln - len(d))
+            return struct.unpack("<ii", d[:8])[0], d[8:-2].decode(errors="replace")
+        s = socket.create_connection((host, port), timeout=5)
+        s.send(pkt(1, 3, pw))
+        rid, _ = recv(s)
+        if rid == -1: print("auth failed"); sys.exit(1)
+        s.send(pkt(2, 2, cmd))
+        _, out = recv(s); print(out)
+        PY
+        echo "[wlsync] started, interval=$${INTERVAL}s"
+        while true; do
+          for f in whitelist.json ops.json banned-players.json banned-ips.json; do
+            if [ -f /src/$$f ] && ! cmp -s /src/$$f /dst/$$f 2>/dev/null; then
+              cp /src/$$f /dst/$$f
+              echo "[wlsync] copied $$f"
+              case $$f in
+                whitelist.json) CMD="whitelist reload" ;;
+                ops.json)       CMD="reload" ;;
+                banned-players.json|banned-ips.json) CMD="kick @a Ban list updated" ;;
+              esac
+              python3 /rcon.py "$$CMD" 2>&1 | sed 's/^/[wlsync] rcon: /' || echo "[wlsync] rcon failed"
+            fi
+          done
+          sleep $$INTERVAL
+        done
+    networks:
+      - fossils
+
+networks:
+  fossils:
+    driver: bridge

ansible/roles/minecraft_fossils/files/minecraft-fossils-restart.service

@@ -0,0 +1,8 @@
+[Unit]
+Description=Daily restart of minecraft-fossils container
+Requires=docker.service
+After=docker.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/docker restart minecraft-fossils

ansible/roles/minecraft_fossils/files/minecraft-fossils-restart.timer

@@ -0,0 +1,10 @@
+[Unit]
+Description=Restart minecraft-fossils daily at 08:00 Europe/Tallinn
+
+[Timer]
+OnCalendar=*-*-* 08:00:00 Europe/Tallinn
+Persistent=true
+Unit=minecraft-fossils-restart.service
+
+[Install]
+WantedBy=timers.target

ansible/roles/minecraft_fossils/handlers/main.yaml

@@ -0,0 +1,4 @@
+---
+- name: Reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true

ansible/roles/minecraft_fossils/tasks/main.yaml

@@ -0,0 +1,49 @@
+---
+# Fabric server plus a sidecar that mirrors the whitelist, ops and ban
+# lists from the Paper server. World data in /opt/minecraft-fossils/data
+# is not managed here. A compose change recreates the container, so it
+# is saved and stopped with time to finish first.
+- name: Deploy Minecraft Fossils
+  ansible.builtin.include_role:
+    name: docker_service
+  vars:
+    service_name: "minecraft-fossils"
+    service_base_path: "{{ services_base_path }}/minecraft-fossils"
+    service_templates:
+      - src: "secrets.env.j2"
+        dest: "secrets.env"
+        owner: "{{ minecraft_fossils_owner }}"
+        group: "{{ minecraft_fossils_owner }}"
+        mode: "0600"
+    service_files:
+      - src: "docker-compose.yml"
+        dest: "docker-compose.yml"
+        owner: "{{ minecraft_fossils_owner }}"
+        group: "{{ minecraft_fossils_owner }}"
+    # secrets.env only matters at container start; never restart the
+    # game server as a side effect of a deploy
+    service_restart_on_change: false
+    service_graceful_stop:
+      - container: minecraft-fossils
+        timeout: 120
+        pre_stop: docker exec minecraft-fossils rcon-cli save-all flush
+    health_check_enabled: false
+
+- name: Install the daily restart units
+  ansible.builtin.copy:
+    src: "{{ item }}"
+    dest: "/etc/systemd/system/{{ item }}"
+    mode: "0644"
+  loop:
+    - minecraft-fossils-restart.service
+    - minecraft-fossils-restart.timer
+  notify: Reload systemd
+
+- name: Flush handlers
+  ansible.builtin.meta: flush_handlers
+
+- name: Enable the daily restart timer
+  ansible.builtin.systemd:
+    name: minecraft-fossils-restart.timer
+    enabled: true
+    state: started

ansible/roles/minecraft_fossils/templates/secrets.env.j2

@@ -0,0 +1,2 @@
+RCON_PASSWORD={{ minecraft_fossils_rcon_password }}
+CF_API_KEY={{ minecraft_fossils_cf_api_key }}

ansible/roles/sillytavern/files/docker-compose.yml

@@ -0,0 +1,11 @@
+services:
+  sillytavern:
+    image: ghcr.io/sillytavern/sillytavern:1.19.0
+    container_name: sillytavern
+    restart: unless-stopped
+    ports:
+      - "10.20.0.20:8000:8000"
+    volumes:
+      - ./config:/home/node/app/config
+      - ./data:/home/node/app/data
+      - ./plugins:/home/node/app/plugins

ansible/roles/sillytavern/tasks/main.yaml

@@ -0,0 +1,16 @@
+---
+# Reachable on the WireGuard address only. Bump the image tag in
+# files/docker-compose.yml to upgrade.
+- name: Deploy SillyTavern
+  ansible.builtin.include_role:
+    name: docker_service
+  vars:
+    service_name: "sillytavern"
+    service_base_path: "{{ services_base_path }}/sillytavern"
+    service_directories:
+      - { path: "config" }
+      - { path: "data" }
+      - { path: "plugins" }
+    service_files:
+      - { src: "docker-compose.yml", dest: "docker-compose.yml" }
+    health_check_enabled: false

ansible/roles/traefik/tasks/main.yaml

@@ -8,21 +8,25 @@
     driver: bridge
     state: present
 
-- name: Deploy Traefik using shared docker_service role
+- name: Deploy Traefik
   ansible.builtin.include_role:
     name: docker_service
   vars:
     service_name: "traefik"
     service_base_path: "{{ traefik_base_path }}"
     service_directories:
-      - { path: "config", mode: "0755" }
-      - { path: "letsencrypt", mode: "0755" }
-      - { path: "logs", mode: "0755" }
+      - { path: "config" }
+      - { path: "letsencrypt" }
+      - { path: "logs" }
     service_templates:
-      - { src: "traefik.yml.j2", dest: "traefik.yml", mode: "0644" }
-      - { src: "dynamic.yml.j2", dest: "config/dynamic.yml", mode: "0644" }
-      - { src: "docker-compose.yml.j2", dest: "docker-compose.yml", mode: "0644" }
-    health_check_enabled: true
-    health_check_host: "{{ wireguard_ip }}"  # Dashboard only accessible on WireGuard IP
+      - { src: "traefik.yml.j2", dest: "traefik.yml" }
+      - { src: "dynamic.yml.j2", dest: "config/dynamic.yml" }
+      - { src: "env.j2", dest: ".env" }
+      - { src: "docker-compose.yml.j2", dest: "docker-compose.yml" }
+    # The Coraza WAF middlewares were never wired to a router
+    service_absent:
+      - config/waf-middleware.yml
+      - config/waf-wp.yml
+    health_check_host: "{{ wireguard_ip }}"  # dashboard listens on WireGuard only
     health_check_port: "{{ traefik_dashboard_port }}"
     health_check_delay: 3

ansible/roles/traefik/templates/docker-compose.yml.j2

@@ -1,18 +1,16 @@
+name: traefik
+
 services:
   traefik:
-    image: traefik:v3.2
     container_name: traefik
+    image: traefik:{{ traefik_version }}
     restart: unless-stopped
     security_opt:
       - no-new-privileges:true
-    networks:
-      - {{ traefik_network }}
     ports:
       - "80:80"
       - "443:443"
-      - "{{ wireguard_ipv4 }}:8080:8080"  # Dashboard only on WireGuard IP
-    environment:
-      - TZ=UTC
+      - "{{ wireguard_ip }}:{{ traefik_dashboard_port }}:8080"  # Dashboard on WireGuard IP only
     volumes:
       - /etc/localtime:/etc/localtime:ro
       - /var/run/docker.sock:/var/run/docker.sock:ro
@@ -20,6 +18,26 @@ services:
       - ./config:/config:ro
       - ./letsencrypt:/letsencrypt
       - ./logs:/var/log/traefik
+    networks:
+      - {{ traefik_network }}
+    env_file:
+      - .env
+    environment:
+      - TZ=UTC
+    labels:
+      - "traefik.enable=true"
+      - "traefik.docker.network={{ traefik_network }}"
+
+      # Dashboard Router with IP Whitelist
+      - "traefik.http.routers.traefik-dashboard.rule=Host(`traefik.asxp.io`)"
+      - "traefik.http.routers.traefik-dashboard.entrypoints=websecure"
+      - "traefik.http.routers.traefik-dashboard.tls=true"
+      - "traefik.http.routers.traefik-dashboard.tls.certresolver=letsencrypt"
+      - "traefik.http.routers.traefik-dashboard.service=api@internal"
+
+      # IP Whitelist Middleware (VPC + WireGuard)
+      - "traefik.http.middlewares.traefik-ipwhitelist.ipwhitelist.sourcerange=10.10.0.0/20,10.20.0.0/24"
+      - "traefik.http.routers.traefik-dashboard.middlewares=traefik-ipwhitelist"
 
 networks:
   {{ traefik_network }}:

ansible/roles/traefik/templates/dynamic.yml.j2

@@ -1,15 +1,20 @@
+# Traefik Dynamic Configuration
+
 http:
   middlewares:
-    # Security headers
+    # Security Headers Middleware
     secure-headers:
       headers:
         sslRedirect: true
-        forceSTSHeader: true
+        stsSeconds: 31536000
         stsIncludeSubdomains: true
         stsPreload: true
-        stsSeconds: 31536000
+        forceSTSHeader: true
+        frameDeny: true
+        contentTypeNosniff: true
+        browserXssFilter: true
 
-    # Rate limiting
+    # Rate Limiting Middleware
     rate-limit:
       rateLimit:
         average: 100

ansible/roles/traefik/templates/env.j2

@@ -0,0 +1,8 @@
+# Traefik Configuration
+
+# Let's Encrypt Email (required for SSL certificates)
+LETSENCRYPT_EMAIL=
+
+# Timezone
+TZ=UTC
+LETSENCRYPT_EMAIL={{ letsencrypt_email }}

ansible/roles/traefik/templates/traefik.yml.j2

@@ -1,7 +1,11 @@
+# Traefik Static Configuration
+
+# API and Dashboard
 api:
   dashboard: true
-  insecure: true  # Dashboard on port 8080 (bound to WireGuard IP only)
+  insecure: true  # Dashboard accessible on :8080 without auth (restricted by network)
 
+# Entry Points
 entryPoints:
   web:
     address: ":80"
@@ -10,10 +14,10 @@ entryPoints:
         entryPoint:
           to: websecure
           scheme: https
-          permanent: true
   websecure:
     address: ":443"
 
+# Docker Provider
 providers:
   docker:
     endpoint: "unix:///var/run/docker.sock"
@@ -23,6 +27,7 @@ providers:
     directory: /config
     watch: true
 
+# Let's Encrypt Certificate Resolver
 certificatesResolvers:
   letsencrypt:
     acme:
@@ -31,6 +36,7 @@ certificatesResolvers:
       httpChallenge:
         entryPoint: web
 
+# Logging
 log:
   level: INFO
   filePath: /var/log/traefik/traefik.log
@@ -38,3 +44,9 @@ log:
 accessLog:
   filePath: /var/log/traefik/access.log
   bufferingSize: 100
+  fields:
+    headers:
+      defaultMode: drop
+      names:
+        User-Agent: keep
+        Referer: keep