Sergei Poljanski
dcacd85655
The Traefik access log shows a large volume of requests carrying AI crawler user-agents, but a user-agent header is self-declared and free to forge. Over 2.27M requests (Nov 2025 - Aug 2026), most of that traffic did not originate from the vendor it named: Claude-User was 4992/5000 forged, ChatGPT-User 2657/2670, Google-Extended 747/747. A pool of roughly 100 Google Cloud hosts rotates through six vendor identities and requests /.env, /terraform.tfstate and /serviceAccountKey.json - credential harvesting, not crawling. Counting these by user-agent alone makes the report worse than useless: it attributes scraper load to vendors that never sent it, and hides the fact that genuine crawler traffic is roughly 0.1% of requests and fully robots.txt-compliant. Verify instead against the vendors' published IP range lists (OpenAI, Anthropic, Google, Perplexity, Apple) and split each bot into verified / spoofed / via-Cloudflare buckets. Requests arriving through Cloudflare are reported separately rather than guessed at, since the logged address is the CF edge and not the client. Ranges are fetched at run time and cached, with fallback to the cache when a vendor endpoint is unreachable, so an outage degrades the report rather than breaking the weekly mail. The fetch sends an explicit User-Agent: claude.com and cloudflare.com sit behind Cloudflare, which 403s the default Python-urllib agent. Only --weekly performs the fetch; the daily threshold alert is unchanged and still runs without network access. Assisted-by: Claude:opus-5
9 lines
157 B
SYSTEMD
9 lines
157 B
SYSTEMD
[Unit]
|
|
Description=Weekly Forgejo crawler report, Mondays 12:00 UTC
|
|
|
|
[Timer]
|
|
OnCalendar=Mon *-*-* 12:00:00
|
|
Persistent=true
|
|
|
|
[Install]
|
|
WantedBy=timers.target
|