- Python 100%
Sergei Poljanski
dcacd85655
The Traefik access log shows a large volume of requests carrying AI crawler user-agents, but a user-agent header is self-declared and free to forge. Over 2.27M requests (Nov 2025 - Aug 2026), most of that traffic did not originate from the vendor it named: Claude-User was 4992/5000 forged, ChatGPT-User 2657/2670, Google-Extended 747/747. A pool of roughly 100 Google Cloud hosts rotates through six vendor identities and requests /.env, /terraform.tfstate and /serviceAccountKey.json - credential harvesting, not crawling. Counting these by user-agent alone makes the report worse than useless: it attributes scraper load to vendors that never sent it, and hides the fact that genuine crawler traffic is roughly 0.1% of requests and fully robots.txt-compliant. Verify instead against the vendors' published IP range lists (OpenAI, Anthropic, Google, Perplexity, Apple) and split each bot into verified / spoofed / via-Cloudflare buckets. Requests arriving through Cloudflare are reported separately rather than guessed at, since the logged address is the CF edge and not the client. Ranges are fetched at run time and cached, with fallback to the cache when a vendor endpoint is unreachable, so an outage degrades the report rather than breaking the weekly mail. The fetch sends an explicit User-Agent: claude.com and cloudflare.com sit behind Cloudflare, which 403s the default Python-urllib agent. Only --weekly performs the fetch; the daily threshold alert is unchanged and still runs without network access. Assisted-by: Claude:opus-5 |
||
|---|---|---|
| crawler-alert.py | ||
| crawler-alert.service | ||
| crawler-alert.timer | ||
| crawler-weekly.service | ||
| crawler-weekly.timer | ||
| README.md | ||
forgejo-crawlers-statistics
Crawler monitoring for the Forgejo instance at git.asxp.io / git.czsk.it.
Reads the Traefik access log, mails a threshold alert daily and a summary
weekly, and separates real AI crawlers from the ones only claiming to be.
Why IP verification
A user-agent header is self-declared and free to forge. Measured over 2.27M requests (Nov 2025 - Aug 2026), most traffic labelled as an AI crawler did not come from the vendor it named:
| Bot | Claimed | Verified | Spoofed |
|---|---|---|---|
| Claude-User | 5,000 | 0 | 4,992 |
| ChatGPT-User | 2,670 | 13 | 2,657 |
| PerplexityBot | 2,038 | 0 | 846 |
| ClaudeBot | 4,301 | 1,070 | 852 |
| Google-Extended | 879 | 0 | 747 |
| GPTBot | 1,864 | 766 | 1,098 |
The forgeries share infrastructure: a pool of ~100 Google Cloud hosts
rotates through six vendor identities, sometimes within the same hour.
Their requests target /.env, /terraform.tfstate,
/serviceAccountKey.json — credential harvesting wearing a crawler
badge. Verified bots, by contrast, made zero robots.txt violations
across 2,682 requests and never fetched repository source.
So the report counts a request as legitimate only when the source IP falls inside the vendor's published range list.
Verification sources
| Vendor | Endpoint |
|---|---|
| OpenAI | openai.com/{chatgpt-user,gptbot,searchbot}.json |
| Anthropic | claude.com/crawling/bots.json |
developers.google.com/static/crawling/ipranges/*.json |
|
| Perplexity | perplexity.ai/perplexitybot.json |
| Apple | search.developer.apple.com/applebot.json |
Meta, ByteDance, Amazon, Diffbot and Cohere publish no range file; their traffic is reported but cannot be verified by IP.
Ranges are fetched at run time and cached at
/var/cache/crawler-alert/ranges.json. A failed fetch falls back to
the cache and the mail says so — a vendor outage degrades the report
rather than breaking it.
Modes
crawler-alert.py # daily: mail only if yesterday > 100k requests
crawler-alert.py --weekly # Monday: 7-day summary, always mails
crawler-alert.py --dry-run # print instead of mailing
Only --weekly fetches ranges; the daily threshold check stays fast.
Caveats
Cloudflare-proxied requests cannot be verified. For traffic arriving
through Cloudflare the logged IP is the CF edge, not the client. Those
hits are counted in a separate via CF column rather than being called
real or fake. Enabling a real-IP header in Traefik would resolve them.
Perplexity's list is stale (8 prefixes, last updated 2025-02) and Apple's dates to 2023, so some genuine traffic from those two may fail verification.
Install
sudo install -m755 crawler-alert.py /usr/local/bin/
sudo cp crawler-*.{service,timer} /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now crawler-alert.timer crawler-weekly.timer
SMTP credentials are read from Forgejo's own app.ini [mailer]
section via docker exec, so there is no second copy to maintain.
Recipient and threshold are constants at the top of the script.
Requirements
Python 3.9+ (stdlib only), read access to /opt/traefik/logs/access.log,
and docker exec on the forgejo container. Traefik must keep the
User-Agent header in its access log:
accessLog:
filePath: /var/log/traefik/access.log
fields:
headers:
defaultMode: drop
names:
User-Agent: keep