Make the tool reusable on other instances
Everything was hardcoded for one deployment: hostname in mail subjects, recipient address, log path, and a router name of "forgejo@docker" that also appeared inside the user-agent regex. That last one fails silently rather than loudly - on any instance whose Traefik router is named something else, the regex matches nothing, every user-agent is dropped, and the report claims zero bot traffic instead of erroring. Move configuration to environment variables read from /etc/crawler-alert.env, with defaults that suit a plain Traefik host. Only CRAWLER_RECIPIENT is required, and the script now refuses to run without it rather than mailing into the void. Anchor the user-agent regex on the request counter instead of the router name. SMTP can now be configured directly via CRAWLER_SMTP_*, so the tool no longer requires Forgejo in Docker; borrowing credentials from app.ini stays the default since it avoids a second copy of the password. Unauthenticated and non-TLS local relays are handled. Document the Traefik access log configuration in traefik-accesslog.yml. This is the one real prerequisite: Traefik drops all headers by default, so without an explicit User-Agent: keep there is nothing to analyse. Includes the field layout the parser expects, logrotate config, and the forwardedHeaders setup needed to recover real client IPs from behind Cloudflare. Rewrite README for someone arriving without context, and add a 0BSD LICENSE so the code can actually be reused. Assisted-by: Claude:opus-5
This commit is contained in:
parent
dcacd85655
commit
5860cbee9a
GPG key ID:
4F8851660FA4121B
7 changed files with 343 additions and 85 deletions
190
README.md
190
README.md
|
|
@ -1,16 +1,21 @@
|
|||
# forgejo-crawlers-statistics
|
||||
# crawler-alert
|
||||
|
||||
Crawler monitoring for the Forgejo instance at `git.asxp.io` / `git.czsk.it`.
|
||||
Reads the Traefik access log, mails a threshold alert daily and a summary
|
||||
weekly, and separates real AI crawlers from the ones only claiming to be.
|
||||
Tells you which AI crawlers are really visiting your site, and which are
|
||||
lying about it.
|
||||
|
||||
## Why IP verification
|
||||
Reads a Traefik access log, mails a threshold alert daily and a summary
|
||||
weekly. Every bot is checked against its vendor's published IP ranges,
|
||||
so a forged `User-Agent` does not get counted as the real thing.
|
||||
|
||||
A user-agent header is self-declared and free to forge. Measured over
|
||||
2.27M requests (Nov 2025 - Aug 2026), most traffic labelled as an AI
|
||||
crawler did not come from the vendor it named:
|
||||
Python 3.9+, standard library only. No database, no daemon, no agent.
|
||||
|
||||
| Bot | Claimed | Verified | Spoofed |
|
||||
## Why user-agent alone is not enough
|
||||
|
||||
A `User-Agent` header is a string the client chooses. Anyone can send
|
||||
`ClaudeBot/1.0`. On the Forgejo instance this was written for, measured
|
||||
over 2.27M requests between November 2025 and August 2026:
|
||||
|
||||
| Bot | Claimed | Verified | Forged |
|
||||
|---|---:|---:|---:|
|
||||
| Claude-User | 5,000 | 0 | 4,992 |
|
||||
| ChatGPT-User | 2,670 | 13 | 2,657 |
|
||||
|
|
@ -19,17 +24,28 @@ crawler did not come from the vendor it named:
|
|||
| Google-Extended | 879 | 0 | 747 |
|
||||
| GPTBot | 1,864 | 766 | 1,098 |
|
||||
|
||||
The forgeries share infrastructure: a pool of ~100 Google Cloud hosts
|
||||
rotates through six vendor identities, sometimes within the same hour.
|
||||
Their requests target `/.env`, `/terraform.tfstate`,
|
||||
`/serviceAccountKey.json` — credential harvesting wearing a crawler
|
||||
badge. Verified bots, by contrast, made zero robots.txt violations
|
||||
across 2,682 requests and never fetched repository source.
|
||||
About 83% of AI-labelled traffic was not from the vendor it named.
|
||||
|
||||
So the report counts a request as legitimate only when the source IP
|
||||
falls inside the vendor's published range list.
|
||||
The forgeries shared infrastructure: roughly 100 Google Cloud hosts
|
||||
rotating through six vendor identities, sometimes within the same hour.
|
||||
Their requests went to `/.env`, `/terraform.tfstate`,
|
||||
`/serviceAccountKey.json`, `/.env.prod.bak` — credential harvesting
|
||||
wearing a crawler's name.
|
||||
|
||||
## Verification sources
|
||||
The genuine crawlers behaved completely differently: 2,682 requests,
|
||||
**zero** `robots.txt` violations, no repository source fetched, ~125 MB
|
||||
total. Mostly they re-read `robots.txt` and `sitemap.xml`.
|
||||
|
||||
That gap is the point. Counting by user-agent tells you AI crawlers are
|
||||
hammering your server. Counting by verified IP tells you they are not,
|
||||
and that something else is — which is a different problem with a
|
||||
different fix.
|
||||
|
||||
## How verification works
|
||||
|
||||
Most major vendors publish their crawler IP ranges as JSON. A request
|
||||
counts as legitimate only when its source address falls inside the
|
||||
range list belonging to the vendor its user-agent names.
|
||||
|
||||
| Vendor | Endpoint |
|
||||
|---|---|
|
||||
|
|
@ -39,54 +55,116 @@ falls inside the vendor's published range list.
|
|||
| Perplexity | `perplexity.ai/perplexitybot.json` |
|
||||
| Apple | `search.developer.apple.com/applebot.json` |
|
||||
|
||||
Meta, ByteDance, Amazon, Diffbot and Cohere publish no range file;
|
||||
their traffic is reported but cannot be verified by IP.
|
||||
Meta, ByteDance, Amazon, Diffbot, Cohere and Common Crawl publish no
|
||||
range file. Their traffic is reported but cannot be verified this way;
|
||||
Amazon supports reverse-DNS verification instead, which this script
|
||||
does not currently implement.
|
||||
|
||||
Ranges are fetched at run time and cached at
|
||||
`/var/cache/crawler-alert/ranges.json`. A failed fetch falls back to
|
||||
the cache and the mail says so — a vendor outage degrades the report
|
||||
rather than breaking it.
|
||||
Ranges are fetched at run time and cached. If a vendor endpoint is
|
||||
unreachable the script falls back to the cache and says so in the mail,
|
||||
so an outage degrades the report instead of breaking it.
|
||||
|
||||
## Modes
|
||||
## Requirements
|
||||
|
||||
crawler-alert.py # daily: mail only if yesterday > 100k requests
|
||||
crawler-alert.py --weekly # Monday: 7-day summary, always mails
|
||||
crawler-alert.py --dry-run # print instead of mailing
|
||||
|
||||
Only `--weekly` fetches ranges; the daily threshold check stays fast.
|
||||
|
||||
## Caveats
|
||||
|
||||
**Cloudflare-proxied requests cannot be verified.** For traffic arriving
|
||||
through Cloudflare the logged IP is the CF edge, not the client. Those
|
||||
hits are counted in a separate `via CF` column rather than being called
|
||||
real or fake. Enabling a real-IP header in Traefik would resolve them.
|
||||
|
||||
**Perplexity's list is stale** (8 prefixes, last updated 2025-02) and
|
||||
Apple's dates to 2023, so some genuine traffic from those two may fail
|
||||
verification.
|
||||
- Traefik writing an access log **with the User-Agent header kept** —
|
||||
it is dropped by default. See `traefik-accesslog.yml`; without it
|
||||
nothing here works.
|
||||
- Python 3.9+
|
||||
- An SMTP account, or a Forgejo/Gitea container whose `app.ini`
|
||||
`[mailer]` section the script can borrow credentials from.
|
||||
|
||||
## Install
|
||||
|
||||
sudo install -m755 crawler-alert.py /usr/local/bin/
|
||||
sudo cp crawler-*.{service,timer} /etc/systemd/system/
|
||||
sudo cp crawler-alert.env.example /etc/crawler-alert.env
|
||||
sudo editor /etc/crawler-alert.env # set CRAWLER_RECIPIENT
|
||||
sudo cp crawler-*.service crawler-*.timer /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now crawler-alert.timer crawler-weekly.timer
|
||||
|
||||
SMTP credentials are read from Forgejo's own `app.ini` `[mailer]`
|
||||
section via `docker exec`, so there is no second copy to maintain.
|
||||
Recipient and threshold are constants at the top of the script.
|
||||
Check it against your own log before enabling the timers:
|
||||
|
||||
## Requirements
|
||||
CRAWLER_LOG='/var/log/traefik/access.log*' crawler-alert.py --weekly --dry-run
|
||||
|
||||
Python 3.9+ (stdlib only), read access to `/opt/traefik/logs/access.log`,
|
||||
and `docker exec` on the `forgejo` container. Traefik must keep the
|
||||
`User-Agent` header in its access log:
|
||||
`--dry-run` prints the mail instead of sending it and needs no SMTP
|
||||
config, so it is safe to run repeatedly while you tune things.
|
||||
|
||||
accessLog:
|
||||
filePath: /var/log/traefik/access.log
|
||||
fields:
|
||||
headers:
|
||||
defaultMode: drop
|
||||
names:
|
||||
User-Agent: keep
|
||||
## Configuration
|
||||
|
||||
All settings are environment variables, read from
|
||||
`/etc/crawler-alert.env` by the systemd units. Only `CRAWLER_RECIPIENT`
|
||||
has no default.
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `CRAWLER_RECIPIENT` | *(required)* | Where reports are mailed |
|
||||
| `CRAWLER_SITE` | `this site` | Name shown in subject/header |
|
||||
| `CRAWLER_LOG` | `/var/log/traefik/access.log*` | Log glob |
|
||||
| `CRAWLER_ROUTER` | *(empty)* | Traefik router to count, e.g. `forgejo@docker`. Empty counts everything |
|
||||
| `CRAWLER_THRESHOLD` | `100000` | Daily alert fires above this |
|
||||
| `CRAWLER_CACHE` | `/var/cache/crawler-alert/ranges.json` | Cached ranges |
|
||||
| `CRAWLER_MAIL_CONTAINER` | `forgejo` | Container to read SMTP from |
|
||||
| `CRAWLER_MAIL_INI` | `/data/gitea/conf/app.ini` | Path inside it |
|
||||
| `CRAWLER_SMTP_*` | *(unset)* | Configure SMTP directly instead |
|
||||
|
||||
Set `CRAWLER_THRESHOLD` near your normal peak. Far above it, the alert
|
||||
never fires and is not a tripwire — run `--dry-run` for a few days and
|
||||
pick a number a busy day would actually reach.
|
||||
|
||||
## Modes
|
||||
|
||||
crawler-alert.py # daily: mail only if over threshold
|
||||
crawler-alert.py --weekly # 7-day summary, always mails
|
||||
crawler-alert.py --dry-run # print instead of mailing
|
||||
|
||||
Only `--weekly` fetches ranges; the daily check needs no network.
|
||||
|
||||
## Sample output
|
||||
|
||||
==============================================================
|
||||
LEGITIMATE CRAWLERS (source IP inside vendor's published ranges)
|
||||
==============================================================
|
||||
bot verified spoofed via CF IPs
|
||||
ChatGPT-User 0 561 0 0
|
||||
ClaudeBot 222 161 0 14
|
||||
OAI-SearchBot 23 209 0 14
|
||||
GPTBot 34 196 0 2
|
||||
PerplexityBot 0 196 0 0
|
||||
|
||||
1909 requests claimed an AI-bot identity; 323 verified (16.9%).
|
||||
1586 failed IP verification (spoofed). 0 arrived via Cloudflare
|
||||
and cannot be verified by IP - not counted either way.
|
||||
|
||||
Top paths fetched by verified crawlers:
|
||||
117 /robots.txt
|
||||
73 /sitemap.xml
|
||||
13 /explore/repos
|
||||
|
||||
## Limitations
|
||||
|
||||
**Cloudflare-proxied requests cannot be verified.** If traffic reaches
|
||||
Traefik through Cloudflare, the logged address is Cloudflare's edge and
|
||||
not the client. Those hits go in a separate `via CF` column rather than
|
||||
being guessed at. `traefik-accesslog.yml` shows how to recover real
|
||||
client IPs with `forwardedHeaders.trustedIPs`.
|
||||
|
||||
**Some vendor lists are stale.** Perplexity's has 8 prefixes and was
|
||||
last updated in February 2025; Apple's dates to 2023. Genuine traffic
|
||||
from those two may fail verification. A 0% verified rate for a small
|
||||
vendor is weaker evidence than it looks.
|
||||
|
||||
**No reverse-DNS verification.** Amazonbot and Meta's crawlers can only
|
||||
be checked by rDNS or ASN, neither of which is implemented. Their
|
||||
traffic appears in the user-agent tally but not the verified table.
|
||||
|
||||
**IPv6 is verified only where vendors publish v6 prefixes**, which most
|
||||
do; addresses outside those are treated as unverified.
|
||||
|
||||
**This is a reporting tool, not an enforcement one.** It tells you what
|
||||
happened. Blocking is a separate decision, and `robots.txt` only works
|
||||
on the bots that already respect it — which, per the numbers above, are
|
||||
exactly the ones not causing the load.
|
||||
|
||||
## License
|
||||
|
||||
0BSD — see LICENSE. Do what you like with it.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue