Make the tool reusable on other instances

Everything was hardcoded for one deployment: hostname in mail subjects,
recipient address, log path, and a router name of "forgejo@docker" that
also appeared inside the user-agent regex. That last one fails silently
rather than loudly - on any instance whose Traefik router is named
something else, the regex matches nothing, every user-agent is dropped,
and the report claims zero bot traffic instead of erroring.

Move configuration to environment variables read from
/etc/crawler-alert.env, with defaults that suit a plain Traefik host.
Only CRAWLER_RECIPIENT is required, and the script now refuses to run
without it rather than mailing into the void. Anchor the user-agent
regex on the request counter instead of the router name.

SMTP can now be configured directly via CRAWLER_SMTP_*, so the tool no
longer requires Forgejo in Docker; borrowing credentials from app.ini
stays the default since it avoids a second copy of the password.
Unauthenticated and non-TLS local relays are handled.

Document the Traefik access log configuration in
traefik-accesslog.yml. This is the one real prerequisite: Traefik drops
all headers by default, so without an explicit User-Agent: keep there
is nothing to analyse. Includes the field layout the parser expects,
logrotate config, and the forwardedHeaders setup needed to recover real
client IPs from behind Cloudflare.

Rewrite README for someone arriving without context, and add a 0BSD
LICENSE so the code can actually be reused.

Assisted-by: Claude:opus-5

LICENSE

@@ -0,0 +1,12 @@
+Copyright (C) 2026 by Sergei Poljanski <me@asxp.io>
+
+Permission to use, copy, modify, and/or distribute this software for any
+purpose with or without fee is hereby granted.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

README.md

@@ -1,16 +1,21 @@
-# forgejo-crawlers-statistics
+# crawler-alert
 
-Crawler monitoring for the Forgejo instance at `git.asxp.io` / `git.czsk.it`.
+Tells you which AI crawlers are really visiting your site, and which are
-Reads the Traefik access log, mails a threshold alert daily and a summary
+lying about it.
-weekly, and separates real AI crawlers from the ones only claiming to be.
 
-## Why IP verification
+Reads a Traefik access log, mails a threshold alert daily and a summary
+weekly. Every bot is checked against its vendor's published IP ranges,
+so a forged `User-Agent` does not get counted as the real thing.
 
-A user-agent header is self-declared and free to forge. Measured over
+Python 3.9+, standard library only. No database, no daemon, no agent.
-2.27M requests (Nov 2025 - Aug 2026), most traffic labelled as an AI
-crawler did not come from the vendor it named:
 
-| Bot | Claimed | Verified | Spoofed |
+## Why user-agent alone is not enough
+
+A `User-Agent` header is a string the client chooses. Anyone can send
+`ClaudeBot/1.0`. On the Forgejo instance this was written for, measured
+over 2.27M requests between November 2025 and August 2026:
+
+| Bot | Claimed | Verified | Forged |
 |---|---:|---:|---:|
 | Claude-User | 5,000 | 0 | 4,992 |
 | ChatGPT-User | 2,670 | 13 | 2,657 |
@@ -19,17 +24,28 @@ crawler did not come from the vendor it named:
 | Google-Extended | 879 | 0 | 747 |
 | GPTBot | 1,864 | 766 | 1,098 |
 
-The forgeries share infrastructure: a pool of ~100 Google Cloud hosts
+About 83% of AI-labelled traffic was not from the vendor it named.
-rotates through six vendor identities, sometimes within the same hour.
-Their requests target `/.env`, `/terraform.tfstate`,
-`/serviceAccountKey.json` — credential harvesting wearing a crawler
-badge. Verified bots, by contrast, made zero robots.txt violations
-across 2,682 requests and never fetched repository source.
 
-So the report counts a request as legitimate only when the source IP
+The forgeries shared infrastructure: roughly 100 Google Cloud hosts
-falls inside the vendor's published range list.
+rotating through six vendor identities, sometimes within the same hour.
+Their requests went to `/.env`, `/terraform.tfstate`,
+`/serviceAccountKey.json`, `/.env.prod.bak` — credential harvesting
+wearing a crawler's name.
 
-## Verification sources
+The genuine crawlers behaved completely differently: 2,682 requests,
+**zero** `robots.txt` violations, no repository source fetched, ~125 MB
+total. Mostly they re-read `robots.txt` and `sitemap.xml`.
+
+That gap is the point. Counting by user-agent tells you AI crawlers are
+hammering your server. Counting by verified IP tells you they are not,
+and that something else is — which is a different problem with a
+different fix.
+
+## How verification works
+
+Most major vendors publish their crawler IP ranges as JSON. A request
+counts as legitimate only when its source address falls inside the
+range list belonging to the vendor its user-agent names.
 
 | Vendor | Endpoint |
 |---|---|
@@ -39,54 +55,116 @@ falls inside the vendor's published range list.
 | Perplexity | `perplexity.ai/perplexitybot.json` |
 | Apple | `search.developer.apple.com/applebot.json` |
 
-Meta, ByteDance, Amazon, Diffbot and Cohere publish no range file;
+Meta, ByteDance, Amazon, Diffbot, Cohere and Common Crawl publish no
-their traffic is reported but cannot be verified by IP.
+range file. Their traffic is reported but cannot be verified this way;
+Amazon supports reverse-DNS verification instead, which this script
+does not currently implement.
 
-Ranges are fetched at run time and cached at
+Ranges are fetched at run time and cached. If a vendor endpoint is
-`/var/cache/crawler-alert/ranges.json`. A failed fetch falls back to
+unreachable the script falls back to the cache and says so in the mail,
-the cache and the mail says so — a vendor outage degrades the report
+so an outage degrades the report instead of breaking it.
-rather than breaking it.
 
-## Modes
+## Requirements
 
-    crawler-alert.py              # daily: mail only if yesterday > 100k requests
+- Traefik writing an access log **with the User-Agent header kept** —
-    crawler-alert.py --weekly     # Monday: 7-day summary, always mails
+  it is dropped by default. See `traefik-accesslog.yml`; without it
-    crawler-alert.py --dry-run    # print instead of mailing
+  nothing here works.
-
+- Python 3.9+
-Only `--weekly` fetches ranges; the daily threshold check stays fast.
+- An SMTP account, or a Forgejo/Gitea container whose `app.ini`
-
+  `[mailer]` section the script can borrow credentials from.
-## Caveats
-
-**Cloudflare-proxied requests cannot be verified.** For traffic arriving
-through Cloudflare the logged IP is the CF edge, not the client. Those
-hits are counted in a separate `via CF` column rather than being called
-real or fake. Enabling a real-IP header in Traefik would resolve them.
-
-**Perplexity's list is stale** (8 prefixes, last updated 2025-02) and
-Apple's dates to 2023, so some genuine traffic from those two may fail
-verification.
 
 ## Install
 
     sudo install -m755 crawler-alert.py /usr/local/bin/
-    sudo cp crawler-*.{service,timer} /etc/systemd/system/
+    sudo cp crawler-alert.env.example /etc/crawler-alert.env
+    sudo editor /etc/crawler-alert.env          # set CRAWLER_RECIPIENT
+    sudo cp crawler-*.service crawler-*.timer /etc/systemd/system/
     sudo systemctl daemon-reload
     sudo systemctl enable --now crawler-alert.timer crawler-weekly.timer
 
-SMTP credentials are read from Forgejo's own `app.ini` `[mailer]`
+Check it against your own log before enabling the timers:
-section via `docker exec`, so there is no second copy to maintain.
-Recipient and threshold are constants at the top of the script.
 
-## Requirements
+    CRAWLER_LOG='/var/log/traefik/access.log*' crawler-alert.py --weekly --dry-run
 
-Python 3.9+ (stdlib only), read access to `/opt/traefik/logs/access.log`,
+`--dry-run` prints the mail instead of sending it and needs no SMTP
-and `docker exec` on the `forgejo` container. Traefik must keep the
+config, so it is safe to run repeatedly while you tune things.
-`User-Agent` header in its access log:
 
-    accessLog:
+## Configuration
-      filePath: /var/log/traefik/access.log
+
-      fields:
+All settings are environment variables, read from
-        headers:
+`/etc/crawler-alert.env` by the systemd units. Only `CRAWLER_RECIPIENT`
-          defaultMode: drop
+has no default.
-          names:
+
-            User-Agent: keep
+| Variable | Default | Meaning |
+|---|---|---|
+| `CRAWLER_RECIPIENT` | *(required)* | Where reports are mailed |
+| `CRAWLER_SITE` | `this site` | Name shown in subject/header |
+| `CRAWLER_LOG` | `/var/log/traefik/access.log*` | Log glob |
+| `CRAWLER_ROUTER` | *(empty)* | Traefik router to count, e.g. `forgejo@docker`. Empty counts everything |
+| `CRAWLER_THRESHOLD` | `100000` | Daily alert fires above this |
+| `CRAWLER_CACHE` | `/var/cache/crawler-alert/ranges.json` | Cached ranges |
+| `CRAWLER_MAIL_CONTAINER` | `forgejo` | Container to read SMTP from |
+| `CRAWLER_MAIL_INI` | `/data/gitea/conf/app.ini` | Path inside it |
+| `CRAWLER_SMTP_*` | *(unset)* | Configure SMTP directly instead |
+
+Set `CRAWLER_THRESHOLD` near your normal peak. Far above it, the alert
+never fires and is not a tripwire — run `--dry-run` for a few days and
+pick a number a busy day would actually reach.
+
+## Modes
+
+    crawler-alert.py              # daily: mail only if over threshold
+    crawler-alert.py --weekly     # 7-day summary, always mails
+    crawler-alert.py --dry-run    # print instead of mailing
+
+Only `--weekly` fetches ranges; the daily check needs no network.
+
+## Sample output
+
+    ==============================================================
+    LEGITIMATE CRAWLERS (source IP inside vendor's published ranges)
+    ==============================================================
+      bot                  verified  spoofed  via CF   IPs
+      ChatGPT-User                0      561       0     0
+      ClaudeBot                 222      161       0    14
+      OAI-SearchBot              23      209       0    14
+      GPTBot                     34      196       0     2
+      PerplexityBot               0      196       0     0
+
+      1909 requests claimed an AI-bot identity; 323 verified (16.9%).
+      1586 failed IP verification (spoofed). 0 arrived via Cloudflare
+      and cannot be verified by IP - not counted either way.
+
+      Top paths fetched by verified crawlers:
+         117  /robots.txt
+          73  /sitemap.xml
+          13  /explore/repos
+
+## Limitations
+
+**Cloudflare-proxied requests cannot be verified.** If traffic reaches
+Traefik through Cloudflare, the logged address is Cloudflare's edge and
+not the client. Those hits go in a separate `via CF` column rather than
+being guessed at. `traefik-accesslog.yml` shows how to recover real
+client IPs with `forwardedHeaders.trustedIPs`.
+
+**Some vendor lists are stale.** Perplexity's has 8 prefixes and was
+last updated in February 2025; Apple's dates to 2023. Genuine traffic
+from those two may fail verification. A 0% verified rate for a small
+vendor is weaker evidence than it looks.
+
+**No reverse-DNS verification.** Amazonbot and Meta's crawlers can only
+be checked by rDNS or ASN, neither of which is implemented. Their
+traffic appears in the user-agent tally but not the verified table.
+
+**IPv6 is verified only where vendors publish v6 prefixes**, which most
+do; addresses outside those are treated as unverified.
+
+**This is a reporting tool, not an enforcement one.** It tells you what
+happened. Blocking is a separate decision, and `robots.txt` only works
+on the bots that already respect it — which, per the numbers above, are
+exactly the ones not causing the load.
+
+## License
+
+0BSD — see LICENSE. Do what you like with it.

crawler-alert.env.example

@@ -0,0 +1,40 @@
+# Copy to /etc/crawler-alert.env and edit.
+# Only CRAWLER_RECIPIENT is required.
+
+# Where the reports are mailed.
+CRAWLER_RECIPIENT=you@example.com
+
+# Name used in mail subjects and the report header.
+CRAWLER_SITE=git.example.com
+
+# Traefik access log. Glob so rotated/gzipped files are included.
+CRAWLER_LOG=/var/log/traefik/access.log*
+
+# Traefik router to count, e.g. forgejo@docker. Leave empty to count
+# everything that reaches Traefik (right for a single-site host).
+CRAWLER_ROUTER=
+
+# Daily alert fires when yesterday exceeds this many requests.
+# Set it near your normal peak, not far above it: a threshold that never
+# fires is not a tripwire. Check a few days of real traffic first with
+#   crawler-alert.py --dry-run
+CRAWLER_THRESHOLD=100000
+
+# Cached vendor IP ranges; used when a vendor endpoint is unreachable.
+CRAWLER_CACHE=/var/cache/crawler-alert/ranges.json
+
+# --- SMTP -------------------------------------------------------------
+# By default the script reads SMTP settings from a Forgejo/Gitea
+# app.ini [mailer] section, so credentials live in one place only:
+CRAWLER_MAIL_CONTAINER=forgejo
+CRAWLER_MAIL_INI=/data/gitea/conf/app.ini
+
+# Or configure SMTP directly and ignore the container entirely.
+# Setting CRAWLER_SMTP_ADDR switches to this path.
+#CRAWLER_SMTP_ADDR=smtp.example.com
+#CRAWLER_SMTP_PORT=587
+#CRAWLER_SMTP_USER=alerts@example.com
+#CRAWLER_SMTP_PASSWORD=
+#CRAWLER_SMTP_FROM=alerts@example.com
+# smtp+starttls (default) | smtps | smtp (no TLS, local relay)
+#CRAWLER_SMTP_PROTOCOL=smtp+starttls

crawler-alert.py

@@ -1,23 +1,47 @@
 #!/usr/bin/env python3
-"""Forgejo crawler watch on git.asxp.io. Two modes:
+"""Crawler watch for a Traefik-fronted Forgejo (or any Traefik site).
-  (default)  daily threshold alert: mail only if yesterday > 100k requests
+
-  --weekly   Monday summary: always mail stats for the past 7 days
+Separates real AI crawlers from traffic that only claims to be, by
+checking the source IP against each vendor's published range list.
+
+Modes:
+  (default)  daily threshold alert: mail only if yesterday exceeded
+             CRAWLER_THRESHOLD requests
+  --weekly   summary: always mail stats for the past 7 days
   --dry-run  print instead of mailing (combinable with --weekly)
-Set up 2026-07-15; timers: crawler-alert.timer, crawler-weekly.timer."""
+
+Configuration is by environment variable; see README.md. The only
+value with no sensible default is CRAWLER_RECIPIENT.
+"""
 import configparser, glob, gzip, ipaddress, json, os, re, smtplib, subprocess, sys, urllib.request
 from collections import Counter
 from datetime import date, timedelta
 from email.mime.text import MIMEText
 
-THRESHOLD = 100_000
+# --- configuration ------------------------------------------------------
-RECIPIENT = "me@asxp.io"
+THRESHOLD = int(os.environ.get("CRAWLER_THRESHOLD", "100000"))
+RECIPIENT = os.environ.get("CRAWLER_RECIPIENT", "")
+SITE      = os.environ.get("CRAWLER_SITE", "this site")
+LOG_GLOB  = os.environ.get("CRAWLER_LOG", "/var/log/traefik/access.log*")
+# Traefik router name to filter on, e.g. "forgejo@docker". Empty = all
+# traffic reaching Traefik, which is what you want for a single-site host.
+ROUTER    = os.environ.get("CRAWLER_ROUTER", "")
+# Container to read SMTP settings from (Forgejo/Gitea app.ini [mailer]).
+# Set CRAWLER_SMTP_* instead to configure SMTP directly.
+MAIL_CONTAINER = os.environ.get("CRAWLER_MAIL_CONTAINER", "forgejo")
+MAIL_INI  = os.environ.get("CRAWLER_MAIL_INI", "/data/gitea/conf/app.ini")
+
 WEEKLY = "--weekly" in sys.argv
 DRY = "--dry-run" in sys.argv
 
+if not RECIPIENT and not DRY:
+    sys.exit("CRAWLER_RECIPIENT is not set (no --dry-run either); refusing "
+             "to run. See README.md.")
+
 # --- verified-crawler support -------------------------------------------
 # Vendors publishing machine-readable IP ranges. A UA token alone proves
-# nothing: as of Aug 2026 ~77% of AI-bot-labelled traffic here came from
+# nothing: on the instance this was written for, ~83% of AI-bot-labelled
-# hosts outside these ranges (mostly GCP). Only IP-verified hits count.
+# traffic came from hosts outside these ranges. Only IP-verified hits count.
 RANGE_SOURCES = {
     "openai":    ["https://openai.com/chatgpt-user.json",
                   "https://openai.com/gptbot.json",
@@ -38,11 +62,11 @@ BOT_VENDOR = {
     "Applebot-Extended": "apple", "Applebot": "apple",
     "Googlebot": "google", "Google-Extended": "google", "GoogleOther": "google",
 }
-CACHE = "/var/cache/crawler-alert/ranges.json"
+CACHE = os.environ.get("CRAWLER_CACHE", "/var/cache/crawler-alert/ranges.json")
 CF_URL = "https://www.cloudflare.com/ips-v4"
 # claude.com and cloudflare.com sit behind Cloudflare, which 403s the
 # default Python-urllib UA. Send a real one.
-UA = "Mozilla/5.0 (compatible; crawler-alert/1.0; +https://git.asxp.io)"
+UA = "Mozilla/5.0 (compatible; crawler-alert/1.0; +https://codeberg.org/)"
 
 
 def fetch(url, timeout=20):
@@ -118,11 +142,11 @@ spoofed = Counter()       # bot -> UA claimed, IP outside vendor ranges
 proxied = Counter()       # bot -> arrived via Cloudflare, unverifiable
 ver_paths = Counter()     # paths fetched by verified bots
 ver_ips = {}              # bot -> set of verified source IPs
-for path in sorted(glob.glob("/opt/traefik/logs/access.log*")):
+for path in sorted(glob.glob(LOG_GLOB)):
     opener = gzip.open if path.endswith(".gz") else open
     with opener(path, "rt", errors="replace") as f:
         for line in f:
-            if "forgejo@docker" not in line:
+            if ROUTER and ROUTER not in line:
                 continue
             for s in stamps:
                 if s in line:
@@ -131,7 +155,10 @@ for path in sorted(glob.glob("/opt/traefik/logs/access.log*")):
                     parts = line.split()
                     if len(parts) > 6 and parts[6].count("/") >= 2:
                         repos["/".join(parts[6].split("/")[1:3])] += 1
-                    ua = re.search(r"\" \"([^\"]*)\" \d+ \"forgejo", line)
+                    # Traefik CLF: ... "<referer>" "<user-agent>" <n> "<router>" ...
+                    # Anchor on the request counter so this works whatever
+                    # the router is called.
+                    ua = re.search(r"\"[^\"]*\" \"([^\"]*)\" \d+ \"", line)
                     if ua:
                         u = ua.group(1)
                         m = re.search(r"(bot|crawler|spider|scrapy|externalagent|gpt|claude|perplexity)[\w./-]*", u, re.I)
@@ -163,15 +190,15 @@ all_ips = set().union(*(v[1] for v in per_day.values()))
 if not WEEKLY:
     s0 = stamps[0]
     count, ips = per_day[s0][0], len(per_day[s0][1])
-    print(f"{s0}: {count} forgejo requests, {ips} unique IPs (threshold {THRESHOLD})")
+    print(f"{s0}: {count} requests, {ips} unique IPs (threshold {THRESHOLD})")
     if DRY or count <= THRESHOLD:
         sys.exit(0)
-    subject = f"git.asxp.io: {count} requests on {s0} (>100k/day threshold)"
+    subject = f"{SITE}: {count} requests on {s0} (over {THRESHOLD}/day threshold)"
     body = (f"Forgejo served {count} requests from {ips} unique IPs on {s0} - "
             f"above the {THRESHOLD}/day watch threshold.\n\n"
             "Time to consider Anubis (PoW challenge) in front of Forgejo.\n")
 else:
-    lines = [f"git.asxp.io weekly crawler report ({stamps[0]} - {stamps[-1]})", ""]
+    lines = [f"{SITE} weekly crawler report ({stamps[0]} - {stamps[-1]})", ""]
     lines.append(f"Total: {total} requests, {len(all_ips)} unique IPs")
     lines.append(f"Daily threshold alert fires above {THRESHOLD} req/day (none = quiet week)")
     lines.append("")
@@ -215,20 +242,34 @@ else:
             for p, c in ver_paths.most_common(8):
                 lines.append(f"    {c:>6}  {p}")
     body = "\n".join(lines) + "\n"
-    subject = f"git.asxp.io weekly crawler report: {total} requests, {len(all_ips)} IPs"
+    subject = f"{SITE} weekly crawler report: {total} requests, {len(all_ips)} IPs"
     if DRY:
         print(subject); print(); print(body)
         sys.exit(0)
 
-ini = subprocess.run(["docker","exec","forgejo","cat","/data/gitea/conf/app.ini"],
+# SMTP settings: use CRAWLER_SMTP_* if given, otherwise borrow them from
+# a Forgejo/Gitea app.ini [mailer] section so there is no second copy of
+# the credentials to keep in sync.
+if os.environ.get("CRAWLER_SMTP_ADDR"):
+    addr = os.environ["CRAWLER_SMTP_ADDR"]
+    port = int(os.environ.get("CRAWLER_SMTP_PORT", "587"))
+    user = os.environ.get("CRAWLER_SMTP_USER", "")
+    passwd = os.environ.get("CRAWLER_SMTP_PASSWORD", "")
+    sender = os.environ.get("CRAWLER_SMTP_FROM", user or RECIPIENT)
+    proto = os.environ.get("CRAWLER_SMTP_PROTOCOL", "smtp+starttls")
+else:
+    ini = subprocess.run(["docker", "exec", MAIL_CONTAINER, "cat", MAIL_INI],
                          capture_output=True, text=True).stdout
-cp = configparser.ConfigParser(interpolation=None, strict=False)
+    if not ini.strip():
-cp.read_string("[DEFAULT]\n" + ini)
+        sys.exit(f"could not read {MAIL_INI} from container "
-m = cp["mailer"]
+                 f"'{MAIL_CONTAINER}' and CRAWLER_SMTP_ADDR is unset")
-proto = m.get("PROTOCOL", "smtps").strip()
+    cp = configparser.ConfigParser(interpolation=None, strict=False)
-addr, port = m.get("SMTP_ADDR").strip(), int(m.get("SMTP_PORT", "465").strip())
+    cp.read_string("[DEFAULT]\n" + ini)
-user, passwd = m.get("USER").strip(), m.get("PASSWD").strip().strip("`\"")
+    m = cp["mailer"]
-sender = m.get("FROM", user).strip()
+    proto = m.get("PROTOCOL", "smtps").strip()
+    addr, port = m.get("SMTP_ADDR").strip(), int(m.get("SMTP_PORT", "465").strip())
+    user, passwd = m.get("USER", "").strip(), m.get("PASSWD", "").strip().strip("`\"")
+    sender = m.get("FROM", user).strip()
 
 msg = MIMEText(body)
 msg["Subject"], msg["From"], msg["To"] = subject, sender, RECIPIENT
@@ -236,8 +277,10 @@ if proto == "smtps":
     s = smtplib.SMTP_SSL(addr, port, timeout=30)
 else:
     s = smtplib.SMTP(addr, port, timeout=30)
+    if proto != "smtp":          # plain smtp = no TLS (local relay)
         s.starttls()
-s.login(user, passwd)
+if user:                          # unauthenticated local relays exist
+    s.login(user, passwd)
 s.sendmail(sender, [RECIPIENT], msg.as_string())
 s.quit()
 print(f"mailed to {RECIPIENT}")

crawler-alert.service

@@ -1,7 +1,8 @@
 [Unit]
-Description=Mail alert when Forgejo exceeds 100k requests/day
+Description=Daily crawler threshold alert
 After=docker.service
 
 [Service]
 Type=oneshot
+EnvironmentFile=/etc/crawler-alert.env
 ExecStart=/usr/local/bin/crawler-alert.py

crawler-weekly.service

@@ -1,7 +1,8 @@
 [Unit]
-Description=Weekly Forgejo crawler report mail
+Description=Weekly crawler report mail
 After=docker.service
 
 [Service]
 Type=oneshot
+EnvironmentFile=/etc/crawler-alert.env
 ExecStart=/usr/local/bin/crawler-alert.py --weekly

traefik-accesslog.yml

@@ -0,0 +1,83 @@
+# Traefik static configuration: the access log this script needs.
+# Merge into your existing traefik.yml (or docker-compose command flags).
+#
+# The critical part is keeping the User-Agent header. Traefik drops all
+# headers by default, and without User-Agent every request is anonymous
+# and no bot analysis is possible at all.
+
+accessLog:
+  filePath: /var/log/traefik/access.log
+  # Traefik's default. The script parses this, not JSON.
+  format: common
+  bufferingSize: 100
+  fields:
+    # Keep the standard CLF fields.
+    defaultMode: keep
+    headers:
+      # Drop every header except the two named below: an access log
+      # holding Cookie or Authorization is a liability, not telemetry.
+      defaultMode: drop
+      names:
+        User-Agent: keep
+        Referer: keep
+
+# Resulting line (one line, wrapped here for readability):
+#
+#   1.2.3.4 - - [10/Aug/2026:22:30:24 +0000] "GET /explore HTTP/2.0"
+#   200 12062 "https://git.example.com/" "Mozilla/5.0 ... ClaudeBot/1.0"
+#   93971 "forgejo@docker" "http://172.18.0.3:3000" 2ms
+#
+# Fields, in order:
+#   1  client IP        <- verified against vendor ranges
+#   2  ident, 3 user    (always "-")
+#   4  timestamp        <- day bucketing
+#   5  request line     <- path, for the per-path breakdown
+#   6  status, 7 bytes
+#   8  Referer          (kept above)
+#   9  User-Agent       <- bot token
+#   10 request counter
+#   11 router name      <- CRAWLER_ROUTER filter, e.g. forgejo@docker
+#   12 backend URL, 13 duration
+#
+# Mount the log directory so both Traefik and the host can see it:
+#
+#   volumes:
+#     - /opt/traefik/logs:/var/log/traefik
+#
+# Rotate it. Traefik reopens on USR1; without rotation this file grows
+# without bound (it reached 533 MB in nine months on the instance this
+# was written for). /etc/logrotate.d/traefik:
+#
+#   /opt/traefik/logs/*.log {
+#     daily
+#     rotate 14
+#     compress
+#     delaycompress
+#     missingok
+#     notifempty
+#     postrotate
+#       docker kill --signal=USR1 traefik 2>/dev/null || true
+#     endscript
+#   }
+#
+# The script reads rotated and gzipped files too, as long as
+# CRAWLER_LOG keeps its trailing glob (access.log*).
+
+# --- If you sit behind Cloudflare -------------------------------------
+# The logged client IP will be Cloudflare's edge, not the real visitor,
+# and no such request can be IP-verified (the report counts these in a
+# separate "via CF" column). To recover real IPs, trust CF's ranges on
+# the entrypoint:
+#
+# entryPoints:
+#   websecure:
+#     address: ":443"
+#     forwardedHeaders:
+#       trustedIPs:
+#         # https://www.cloudflare.com/ips-v4 - keep this list current
+#         - 173.245.48.0/20
+#         - 103.21.244.0/22
+#         # ... remaining CF ranges ...
+#
+# Only list address ranges you actually trust: anyone in trustedIPs can
+# spoof X-Forwarded-For and forge their apparent source address.