forgejo-crawlers-statistics

Crawler monitoring for the Forgejo instance at git.asxp.io / git.czsk.it. Reads the Traefik access log, mails a threshold alert daily and a summary weekly, and separates real AI crawlers from the ones only claiming to be.

Why IP verification

A user-agent header is self-declared and free to forge. Measured over 2.27M requests (Nov 2025 - Aug 2026), most traffic labelled as an AI crawler did not come from the vendor it named:

Bot	Claimed	Verified	Spoofed
Claude-User	5,000	0	4,992
ChatGPT-User	2,670	13	2,657
PerplexityBot	2,038	0	846
ClaudeBot	4,301	1,070	852
Google-Extended	879	0	747
GPTBot	1,864	766	1,098

The forgeries share infrastructure: a pool of ~100 Google Cloud hosts rotates through six vendor identities, sometimes within the same hour. Their requests target /.env, /terraform.tfstate, /serviceAccountKey.json — credential harvesting wearing a crawler badge. Verified bots, by contrast, made zero robots.txt violations across 2,682 requests and never fetched repository source.

So the report counts a request as legitimate only when the source IP falls inside the vendor's published range list.

Verification sources

Vendor	Endpoint
OpenAI	openai.com/{chatgpt-user,gptbot,searchbot}.json
Anthropic	claude.com/crawling/bots.json
Google	developers.google.com/static/crawling/ipranges/*.json
Perplexity	perplexity.ai/perplexitybot.json
Apple	search.developer.apple.com/applebot.json

Meta, ByteDance, Amazon, Diffbot and Cohere publish no range file; their traffic is reported but cannot be verified by IP.

Ranges are fetched at run time and cached at /var/cache/crawler-alert/ranges.json. A failed fetch falls back to the cache and the mail says so — a vendor outage degrades the report rather than breaking it.

Modes

crawler-alert.py              # daily: mail only if yesterday > 100k requests
crawler-alert.py --weekly     # Monday: 7-day summary, always mails
crawler-alert.py --dry-run    # print instead of mailing

Only --weekly fetches ranges; the daily threshold check stays fast.

Caveats

Cloudflare-proxied requests cannot be verified. For traffic arriving through Cloudflare the logged IP is the CF edge, not the client. Those hits are counted in a separate via CF column rather than being called real or fake. Enabling a real-IP header in Traefik would resolve them.

Perplexity's list is stale (8 prefixes, last updated 2025-02) and Apple's dates to 2023, so some genuine traffic from those two may fail verification.

Install

sudo install -m755 crawler-alert.py /usr/local/bin/
sudo cp crawler-*.{service,timer} /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now crawler-alert.timer crawler-weekly.timer

SMTP credentials are read from Forgejo's own app.ini [mailer] section via docker exec, so there is no second copy to maintain. Recipient and threshold are constants at the top of the script.

Requirements

Python 3.9+ (stdlib only), read access to /opt/traefik/logs/access.log, and docker exec on the forgejo container. Traefik must keep the User-Agent header in its access log:

accessLog:
  filePath: /var/log/traefik/access.log
  fields:
    headers:
      defaultMode: drop
      names:
        User-Agent: keep