# crawler-alert

Tells you which AI crawlers are really visiting your site, and which are
lying about it.

Reads a Traefik access log, mails a threshold alert daily and a summary
weekly. Every bot is checked against its vendor's published IP ranges,
so a forged `User-Agent` does not get counted as the real thing.

Python 3.9+, standard library only. No database, no daemon, no agent.

## Why user-agent alone is not enough

A `User-Agent` header is a string the client chooses. Anyone can send
`ClaudeBot/1.0`. On the Forgejo instance this was written for, measured
over 2.27M requests between November 2025 and August 2026:

| Bot | Claimed | Verified | Forged |
|---|---:|---:|---:|
| Claude-User | 5,000 | 0 | 4,992 |
| ChatGPT-User | 2,670 | 13 | 2,657 |
| PerplexityBot | 2,038 | 0 | 846 |
| ClaudeBot | 4,301 | 1,070 | 852 |
| Google-Extended | 879 | 0 | 747 |
| GPTBot | 1,864 | 766 | 1,098 |

About 83% of AI-labelled traffic was not from the vendor it named.

The forgeries shared infrastructure: roughly 100 Google Cloud hosts
rotating through six vendor identities, sometimes within the same hour.
Their requests went to `/.env`, `/terraform.tfstate`,
`/serviceAccountKey.json`, `/.env.prod.bak` — credential harvesting
wearing a crawler's name.

The genuine crawlers behaved completely differently: 2,682 requests,
**zero** `robots.txt` violations, no repository source fetched, ~125 MB
total. Mostly they re-read `robots.txt` and `sitemap.xml`.

That gap is the point. Counting by user-agent tells you AI crawlers are
hammering your server. Counting by verified IP tells you they are not,
and that something else is — which is a different problem with a
different fix.

## How verification works

Most major vendors publish their crawler IP ranges as JSON. A request
counts as legitimate only when its source address falls inside the
range list belonging to the vendor its user-agent names.

| Vendor | Endpoint |
|---|---|
| OpenAI | `openai.com/{chatgpt-user,gptbot,searchbot}.json` |
| Anthropic | `claude.com/crawling/bots.json` |
| Google | `developers.google.com/static/crawling/ipranges/*.json` |
| Perplexity | `perplexity.ai/perplexitybot.json` |
| Apple | `search.developer.apple.com/applebot.json` |

Meta, ByteDance, Amazon, Diffbot, Cohere and Common Crawl publish no
range file. Their traffic is reported but cannot be verified this way;
Amazon supports reverse-DNS verification instead, which this script
does not currently implement.

Ranges are fetched at run time and cached. If a vendor endpoint is
unreachable the script falls back to the cache and says so in the mail,
so an outage degrades the report instead of breaking it.

## Requirements

- Traefik writing an access log **with the User-Agent header kept** —
  it is dropped by default. See `traefik-accesslog.yml`; without it
  nothing here works.
- Python 3.9+
- An SMTP account, or a Forgejo/Gitea container whose `app.ini`
  `[mailer]` section the script can borrow credentials from.

## Install

    sudo install -m755 crawler-alert.py /usr/local/bin/
    sudo cp crawler-alert.env.example /etc/crawler-alert.env
    sudo editor /etc/crawler-alert.env          # set CRAWLER_RECIPIENT
    sudo cp crawler-*.service crawler-*.timer /etc/systemd/system/
    sudo systemctl daemon-reload
    sudo systemctl enable --now crawler-alert.timer crawler-weekly.timer

Check it against your own log before enabling the timers:

    CRAWLER_LOG='/var/log/traefik/access.log*' crawler-alert.py --weekly --dry-run

`--dry-run` prints the mail instead of sending it and needs no SMTP
config, so it is safe to run repeatedly while you tune things.

## Configuration

All settings are environment variables, read from
`/etc/crawler-alert.env` by the systemd units. Only `CRAWLER_RECIPIENT`
has no default.

| Variable | Default | Meaning |
|---|---|---|
| `CRAWLER_RECIPIENT` | *(required)* | Where reports are mailed |
| `CRAWLER_SITE` | `this site` | Name shown in subject/header |
| `CRAWLER_LOG` | `/var/log/traefik/access.log*` | Log glob |
| `CRAWLER_ROUTER` | *(empty)* | Traefik router to count, e.g. `forgejo@docker`. Empty counts everything |
| `CRAWLER_THRESHOLD` | `100000` | Daily alert fires above this |
| `CRAWLER_CACHE` | `/var/cache/crawler-alert/ranges.json` | Cached ranges |
| `CRAWLER_MAIL_CONTAINER` | `forgejo` | Container to read SMTP from |
| `CRAWLER_MAIL_INI` | `/data/gitea/conf/app.ini` | Path inside it |
| `CRAWLER_SMTP_*` | *(unset)* | Configure SMTP directly instead |

Set `CRAWLER_THRESHOLD` near your normal peak. Far above it, the alert
never fires and is not a tripwire — run `--dry-run` for a few days and
pick a number a busy day would actually reach.

## Modes

    crawler-alert.py              # daily: mail only if over threshold
    crawler-alert.py --weekly     # 7-day summary, always mails
    crawler-alert.py --dry-run    # print instead of mailing

Only `--weekly` fetches ranges; the daily check needs no network.

## Sample output

    ==============================================================
    LEGITIMATE CRAWLERS (source IP inside vendor's published ranges)
    ==============================================================
      bot                  verified  spoofed  via CF   IPs
      ChatGPT-User                0      561       0     0
      ClaudeBot                 222      161       0    14
      OAI-SearchBot              23      209       0    14
      GPTBot                     34      196       0     2
      PerplexityBot               0      196       0     0

      1909 requests claimed an AI-bot identity; 323 verified (16.9%).
      1586 failed IP verification (spoofed). 0 arrived via Cloudflare
      and cannot be verified by IP - not counted either way.

      Top paths fetched by verified crawlers:
         117  /robots.txt
          73  /sitemap.xml
          13  /explore/repos

## Limitations

**Cloudflare-proxied requests cannot be verified.** If traffic reaches
Traefik through Cloudflare, the logged address is Cloudflare's edge and
not the client. Those hits go in a separate `via CF` column rather than
being guessed at. `traefik-accesslog.yml` shows how to recover real
client IPs with `forwardedHeaders.trustedIPs`.

**Some vendor lists are stale.** Perplexity's has 8 prefixes and was
last updated in February 2025; Apple's dates to 2023. Genuine traffic
from those two may fail verification. A 0% verified rate for a small
vendor is weaker evidence than it looks.

**No reverse-DNS verification.** Amazonbot and Meta's crawlers can only
be checked by rDNS or ASN, neither of which is implemented. Their
traffic appears in the user-agent tally but not the verified table.

**IPv6 is verified only where vendors publish v6 prefixes**, which most
do; addresses outside those are treated as unverified.

**This is a reporting tool, not an enforcement one.** It tells you what
happened. Blocking is a separate decision, and `robots.txt` only works
on the bots that already respect it — which, per the numbers above, are
exactly the ones not causing the load.

## License

0BSD — see LICENSE. Do what you like with it.