# forgejo-crawlers-statistics

Crawler monitoring for the Forgejo instance at `git.asxp.io` / `git.czsk.it`.
Reads the Traefik access log, mails a threshold alert daily and a summary
weekly, and separates real AI crawlers from the ones only claiming to be.

## Why IP verification

A user-agent header is self-declared and free to forge. Measured over
2.27M requests (Nov 2025 - Aug 2026), most traffic labelled as an AI
crawler did not come from the vendor it named:

| Bot | Claimed | Verified | Spoofed |
|---|---:|---:|---:|
| Claude-User | 5,000 | 0 | 4,992 |
| ChatGPT-User | 2,670 | 13 | 2,657 |
| PerplexityBot | 2,038 | 0 | 846 |
| ClaudeBot | 4,301 | 1,070 | 852 |
| Google-Extended | 879 | 0 | 747 |
| GPTBot | 1,864 | 766 | 1,098 |

The forgeries share infrastructure: a pool of ~100 Google Cloud hosts
rotates through six vendor identities, sometimes within the same hour.
Their requests target `/.env`, `/terraform.tfstate`,
`/serviceAccountKey.json` — credential harvesting wearing a crawler
badge. Verified bots, by contrast, made zero robots.txt violations
across 2,682 requests and never fetched repository source.

So the report counts a request as legitimate only when the source IP
falls inside the vendor's published range list.

## Verification sources

| Vendor | Endpoint |
|---|---|
| OpenAI | `openai.com/{chatgpt-user,gptbot,searchbot}.json` |
| Anthropic | `claude.com/crawling/bots.json` |
| Google | `developers.google.com/static/crawling/ipranges/*.json` |
| Perplexity | `perplexity.ai/perplexitybot.json` |
| Apple | `search.developer.apple.com/applebot.json` |

Meta, ByteDance, Amazon, Diffbot and Cohere publish no range file;
their traffic is reported but cannot be verified by IP.

Ranges are fetched at run time and cached at
`/var/cache/crawler-alert/ranges.json`. A failed fetch falls back to
the cache and the mail says so — a vendor outage degrades the report
rather than breaking it.

## Modes

    crawler-alert.py              # daily: mail only if yesterday > 100k requests
    crawler-alert.py --weekly     # Monday: 7-day summary, always mails
    crawler-alert.py --dry-run    # print instead of mailing

Only `--weekly` fetches ranges; the daily threshold check stays fast.

## Caveats

**Cloudflare-proxied requests cannot be verified.** For traffic arriving
through Cloudflare the logged IP is the CF edge, not the client. Those
hits are counted in a separate `via CF` column rather than being called
real or fake. Enabling a real-IP header in Traefik would resolve them.

**Perplexity's list is stale** (8 prefixes, last updated 2025-02) and
Apple's dates to 2023, so some genuine traffic from those two may fail
verification.

## Install

    sudo install -m755 crawler-alert.py /usr/local/bin/
    sudo cp crawler-*.{service,timer} /etc/systemd/system/
    sudo systemctl daemon-reload
    sudo systemctl enable --now crawler-alert.timer crawler-weekly.timer

SMTP credentials are read from Forgejo's own `app.ini` `[mailer]`
section via `docker exec`, so there is no second copy to maintain.
Recipient and threshold are constants at the top of the script.

## Requirements

Python 3.9+ (stdlib only), read access to `/opt/traefik/logs/access.log`,
and `docker exec` on the `forgejo` container. Traefik must keep the
`User-Agent` header in its access log:

    accessLog:
      filePath: /var/log/traefik/access.log
      fields:
        headers:
          defaultMode: drop
          names:
            User-Agent: keep