Sergei Poljanski
81716c2fc9
Amazonbot, meta-externalagent, YouBot, Bytespider and PetalBot publish no IP range file, so they were counted in the user-agent tally and nowhere else - the largest single source of traffic on the sample instance was also the least examined. Check them by FCrDNS: the PTR record must end in a vendor domain and that hostname must resolve back to the same address. The forward step is the part that matters. A PTR record alone is written by whoever controls the address block, so without confirming it forward the check would accept anything its owner chose to claim. ASN verification was considered and deliberately left out. Genuine YouBot and genuine Amazonbot both live in AS14618, which is also every EC2 instance a spoofer could rent. An ASN match shows the traffic came from a cloud, not from the vendor; presenting that as verification would be worse than presenting nothing. Report RFC1918 sources separately rather than as failures. On the sample instance 21624 of meta-externalagent's 21651 requests came from 172.18.0.1, the Docker bridge gateway: the real client address was replaced before it reached the log. Those are not spoofed, they are unjudgeable, and calling them spoofed would be a false accusation caused by the reader's own proxy configuration. Resolution is capped at CRAWLER_RDNS_MAX unique addresses per bot (busiest first) so a flood of distinct forgeries cannot stall the weekly mail on DNS timeouts, and can be disabled with CRAWLER_RDNS=0. Assisted-by: Claude:opus-5
203 lines
8.5 KiB
Markdown
203 lines
8.5 KiB
Markdown
# crawler-alert
|
|
|
|
Tells you which AI crawlers are really visiting your site, and which are
|
|
lying about it.
|
|
|
|
Reads a Traefik access log, mails a threshold alert daily and a summary
|
|
weekly. Every bot is checked against its vendor's published IP ranges,
|
|
so a forged `User-Agent` does not get counted as the real thing.
|
|
|
|
Python 3.9+, standard library only. No database, no daemon, no agent.
|
|
|
|
## Why user-agent alone is not enough
|
|
|
|
A `User-Agent` header is a string the client chooses. Anyone can send
|
|
`ClaudeBot/1.0`. On the Forgejo instance this was written for, measured
|
|
over 2.27M requests between November 2025 and August 2026:
|
|
|
|
| Bot | Claimed | Verified | Forged |
|
|
|---|---:|---:|---:|
|
|
| Claude-User | 5,000 | 0 | 4,992 |
|
|
| ChatGPT-User | 2,670 | 13 | 2,657 |
|
|
| PerplexityBot | 2,038 | 0 | 846 |
|
|
| ClaudeBot | 4,301 | 1,070 | 852 |
|
|
| Google-Extended | 879 | 0 | 747 |
|
|
| GPTBot | 1,864 | 766 | 1,098 |
|
|
|
|
About 83% of AI-labelled traffic was not from the vendor it named.
|
|
|
|
The forgeries shared infrastructure: roughly 100 Google Cloud hosts
|
|
rotating through six vendor identities, sometimes within the same hour.
|
|
Their requests went to `/.env`, `/terraform.tfstate`,
|
|
`/serviceAccountKey.json`, `/.env.prod.bak` — credential harvesting
|
|
wearing a crawler's name.
|
|
|
|
The genuine crawlers behaved completely differently: 2,682 requests,
|
|
**zero** `robots.txt` violations, no repository source fetched, ~125 MB
|
|
total. Mostly they re-read `robots.txt` and `sitemap.xml`.
|
|
|
|
That gap is the point. Counting by user-agent tells you AI crawlers are
|
|
hammering your server. Counting by verified IP tells you they are not,
|
|
and that something else is — which is a different problem with a
|
|
different fix.
|
|
|
|
## How verification works
|
|
|
|
Most major vendors publish their crawler IP ranges as JSON. A request
|
|
counts as legitimate only when its source address falls inside the
|
|
range list belonging to the vendor its user-agent names.
|
|
|
|
| Vendor | Endpoint |
|
|
|---|---|
|
|
| OpenAI | `openai.com/{chatgpt-user,gptbot,searchbot}.json` |
|
|
| Anthropic | `claude.com/crawling/bots.json` |
|
|
| Google | `developers.google.com/static/crawling/ipranges/*.json` |
|
|
| Perplexity | `perplexity.ai/perplexitybot.json` |
|
|
| Apple | `search.developer.apple.com/applebot.json` |
|
|
|
|
Meta, ByteDance, Amazon and You.com publish no range file. Those are
|
|
checked by **forward-confirmed reverse DNS** instead: the address's PTR
|
|
record must end in a vendor domain, *and* that hostname must resolve
|
|
back to the same address. The forward step is what makes it evidence —
|
|
a PTR alone is written by whoever controls the address block.
|
|
|
|
| Bot | rDNS suffix |
|
|
|---|---|
|
|
| Amazonbot | `.crawl.amazonbot.amazon` |
|
|
| meta-externalagent, FacebookBot | `.facebook.com`, `.fbsv.net` |
|
|
| YouBot | `.search.you.com` |
|
|
| Bytespider | `.bytedance.com`, `.byteoversea.com` |
|
|
| PetalBot | `.petalsearch.com`, `.aspiegel.com` |
|
|
| DuckAssistBot | `.duckduckgo.com` |
|
|
|
|
Diffbot, Cohere and Common Crawl publish neither ranges nor a
|
|
documented rDNS convention; they appear in the user-agent tally only.
|
|
|
|
### Why there is no ASN verification
|
|
|
|
ASN was considered and rejected. It looks like verification but is not:
|
|
on the sample instance, genuine YouBot (`68.67.112.227`) and genuine
|
|
Amazonbot (`100.24.167.60`) both resolve to **AS14618, Amazon** — the
|
|
same ASN as every EC2 instance on the internet, including any spoofer
|
|
renting one. An ASN match proves the traffic came from a cloud
|
|
provider, not from the vendor. Reporting that as "verified" would be
|
|
worse than reporting nothing.
|
|
|
|
Ranges are fetched at run time and cached. If a vendor endpoint is
|
|
unreachable the script falls back to the cache and says so in the mail,
|
|
so an outage degrades the report instead of breaking it.
|
|
|
|
## Requirements
|
|
|
|
- Traefik writing an access log **with the User-Agent header kept** —
|
|
it is dropped by default. See `traefik-accesslog.yml`; without it
|
|
nothing here works.
|
|
- Python 3.9+
|
|
- An SMTP account, or a Forgejo/Gitea container whose `app.ini`
|
|
`[mailer]` section the script can borrow credentials from.
|
|
|
|
## Install
|
|
|
|
sudo install -m755 crawler-alert.py /usr/local/bin/
|
|
sudo cp crawler-alert.env.example /etc/crawler-alert.env
|
|
sudo editor /etc/crawler-alert.env # set CRAWLER_RECIPIENT
|
|
sudo cp crawler-*.service crawler-*.timer /etc/systemd/system/
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now crawler-alert.timer crawler-weekly.timer
|
|
|
|
Check it against your own log before enabling the timers:
|
|
|
|
CRAWLER_LOG='/var/log/traefik/access.log*' crawler-alert.py --weekly --dry-run
|
|
|
|
`--dry-run` prints the mail instead of sending it and needs no SMTP
|
|
config, so it is safe to run repeatedly while you tune things.
|
|
|
|
## Configuration
|
|
|
|
All settings are environment variables, read from
|
|
`/etc/crawler-alert.env` by the systemd units. Only `CRAWLER_RECIPIENT`
|
|
has no default.
|
|
|
|
| Variable | Default | Meaning |
|
|
|---|---|---|
|
|
| `CRAWLER_RECIPIENT` | *(required)* | Where reports are mailed |
|
|
| `CRAWLER_SITE` | `this site` | Name shown in subject/header |
|
|
| `CRAWLER_LOG` | `/var/log/traefik/access.log*` | Log glob |
|
|
| `CRAWLER_ROUTER` | *(empty)* | Traefik router to count, e.g. `forgejo@docker`. Empty counts everything |
|
|
| `CRAWLER_THRESHOLD` | `100000` | Daily alert fires above this |
|
|
| `CRAWLER_CACHE` | `/var/cache/crawler-alert/ranges.json` | Cached ranges |
|
|
| `CRAWLER_MAIL_CONTAINER` | `forgejo` | Container to read SMTP from |
|
|
| `CRAWLER_MAIL_INI` | `/data/gitea/conf/app.ini` | Path inside it |
|
|
| `CRAWLER_SMTP_*` | *(unset)* | Configure SMTP directly instead |
|
|
|
|
Set `CRAWLER_THRESHOLD` near your normal peak. Far above it, the alert
|
|
never fires and is not a tripwire — run `--dry-run` for a few days and
|
|
pick a number a busy day would actually reach.
|
|
|
|
## Modes
|
|
|
|
crawler-alert.py # daily: mail only if over threshold
|
|
crawler-alert.py --weekly # 7-day summary, always mails
|
|
crawler-alert.py --dry-run # print instead of mailing
|
|
|
|
Only `--weekly` fetches ranges; the daily check needs no network.
|
|
|
|
## Sample output
|
|
|
|
==============================================================
|
|
LEGITIMATE CRAWLERS (source IP inside vendor's published ranges)
|
|
==============================================================
|
|
bot verified spoofed via CF IPs
|
|
ChatGPT-User 0 561 0 0
|
|
ClaudeBot 222 161 0 14
|
|
OAI-SearchBot 23 209 0 14
|
|
GPTBot 34 196 0 2
|
|
PerplexityBot 0 196 0 0
|
|
|
|
1909 requests claimed an AI-bot identity; 323 verified (16.9%).
|
|
1586 failed IP verification (spoofed). 0 arrived via Cloudflare
|
|
and cannot be verified by IP - not counted either way.
|
|
|
|
Top paths fetched by verified crawlers:
|
|
117 /robots.txt
|
|
73 /sitemap.xml
|
|
13 /explore/repos
|
|
|
|
## Limitations
|
|
|
|
**Cloudflare-proxied requests cannot be verified.** If traffic reaches
|
|
Traefik through Cloudflare, the logged address is Cloudflare's edge and
|
|
not the client. Those hits go in a separate `via CF` column rather than
|
|
being guessed at. `traefik-accesslog.yml` shows how to recover real
|
|
client IPs with `forwardedHeaders.trustedIPs`.
|
|
|
|
**Some vendor lists are stale.** Perplexity's has 8 prefixes and was
|
|
last updated in February 2025; Apple's dates to 2023. Genuine traffic
|
|
from those two may fail verification. A 0% verified rate for a small
|
|
vendor is weaker evidence than it looks.
|
|
|
|
**Private source addresses cannot be judged at all.** If the logged
|
|
client IP is RFC1918 (`172.18.0.1`, a Docker bridge gateway, say), then
|
|
something in front of Traefik replaced the real address before it was
|
|
written. Those hits are counted in a `private` column, neither verified
|
|
nor spoofed. On the sample instance this was 21,624 of Meta's 21,651
|
|
requests — the tool cannot tell you whether they were genuine, only
|
|
that the log does not contain the evidence. Fixing it is a Traefik
|
|
`forwardedHeaders` change, not a script change.
|
|
|
|
**rDNS is rate-limited by design.** Each unique address costs two DNS
|
|
lookups, so only the busiest `CRAWLER_RDNS_MAX` (default 400) addresses
|
|
per bot are resolved; the rest are reported as unresolved. Raise it if
|
|
your DNS resolver is fast and local.
|
|
|
|
**IPv6 is verified only where vendors publish v6 prefixes**, which most
|
|
do; addresses outside those are treated as unverified.
|
|
|
|
**This is a reporting tool, not an enforcement one.** It tells you what
|
|
happened. Blocking is a separate decision, and `robots.txt` only works
|
|
on the bots that already respect it — which, per the numbers above, are
|
|
exactly the ones not causing the load.
|
|
|
|
## License
|
|
|
|
0BSD — see LICENSE. Do what you like with it.
|