first commit

README.md

@@ -0,0 +1,92 @@
+# forgejo-crawlers-statistics
+
+Crawler monitoring for the Forgejo instance at `git.asxp.io` / `git.czsk.it`.
+Reads the Traefik access log, mails a threshold alert daily and a summary
+weekly, and separates real AI crawlers from the ones only claiming to be.
+
+## Why IP verification
+
+A user-agent header is self-declared and free to forge. Measured over
+2.27M requests (Nov 2025 - Aug 2026), most traffic labelled as an AI
+crawler did not come from the vendor it named:
+
+| Bot | Claimed | Verified | Spoofed |
+|---|---:|---:|---:|
+| Claude-User | 5,000 | 0 | 4,992 |
+| ChatGPT-User | 2,670 | 13 | 2,657 |
+| PerplexityBot | 2,038 | 0 | 846 |
+| ClaudeBot | 4,301 | 1,070 | 852 |
+| Google-Extended | 879 | 0 | 747 |
+| GPTBot | 1,864 | 766 | 1,098 |
+
+The forgeries share infrastructure: a pool of ~100 Google Cloud hosts
+rotates through six vendor identities, sometimes within the same hour.
+Their requests target `/.env`, `/terraform.tfstate`,
+`/serviceAccountKey.json` — credential harvesting wearing a crawler
+badge. Verified bots, by contrast, made zero robots.txt violations
+across 2,682 requests and never fetched repository source.
+
+So the report counts a request as legitimate only when the source IP
+falls inside the vendor's published range list.
+
+## Verification sources
+
+| Vendor | Endpoint |
+|---|---|
+| OpenAI | `openai.com/{chatgpt-user,gptbot,searchbot}.json` |
+| Anthropic | `claude.com/crawling/bots.json` |
+| Google | `developers.google.com/static/crawling/ipranges/*.json` |
+| Perplexity | `perplexity.ai/perplexitybot.json` |
+| Apple | `search.developer.apple.com/applebot.json` |
+
+Meta, ByteDance, Amazon, Diffbot and Cohere publish no range file;
+their traffic is reported but cannot be verified by IP.
+
+Ranges are fetched at run time and cached at
+`/var/cache/crawler-alert/ranges.json`. A failed fetch falls back to
+the cache and the mail says so — a vendor outage degrades the report
+rather than breaking it.
+
+## Modes
+
+    crawler-alert.py              # daily: mail only if yesterday > 100k requests
+    crawler-alert.py --weekly     # Monday: 7-day summary, always mails
+    crawler-alert.py --dry-run    # print instead of mailing
+
+Only `--weekly` fetches ranges; the daily threshold check stays fast.
+
+## Caveats
+
+**Cloudflare-proxied requests cannot be verified.** For traffic arriving
+through Cloudflare the logged IP is the CF edge, not the client. Those
+hits are counted in a separate `via CF` column rather than being called
+real or fake. Enabling a real-IP header in Traefik would resolve them.
+
+**Perplexity's list is stale** (8 prefixes, last updated 2025-02) and
+Apple's dates to 2023, so some genuine traffic from those two may fail
+verification.
+
+## Install
+
+    sudo install -m755 crawler-alert.py /usr/local/bin/
+    sudo cp crawler-*.{service,timer} /etc/systemd/system/
+    sudo systemctl daemon-reload
+    sudo systemctl enable --now crawler-alert.timer crawler-weekly.timer
+
+SMTP credentials are read from Forgejo's own `app.ini` `[mailer]`
+section via `docker exec`, so there is no second copy to maintain.
+Recipient and threshold are constants at the top of the script.
+
+## Requirements
+
+Python 3.9+ (stdlib only), read access to `/opt/traefik/logs/access.log`,
+and `docker exec` on the `forgejo` container. Traefik must keep the
+`User-Agent` header in its access log:
+
+    accessLog:
+      filePath: /var/log/traefik/access.log
+      fields:
+        headers:
+          defaultMode: drop
+          names:
+            User-Agent: keep