The path breakdown was populated only inside the IP-range branch, so
it silently covered one of the two verification methods. Vendors
confirmed by PTR or ASN - YouBot, Amazonbot, Meta - were counted in
their own table but contributed nothing to the only output that shows
what crawlers actually fetch. On the sample instance that hid roughly
40% of the verified requests from the path list.
Collect paths per address during the log pass and merge them once the
address is confirmed, whichever method confirms it. Private-source
hits are excluded, since those addresses are never judged either way.
Move the section to the end of the report and retitle it: it now spans
both tables above, so placing it under the IP-range table implied a
narrower scope than it has.
Assisted-by: Claude:opus-5
The rDNS check had one failure bucket, so "PTR points at a different
network" and "no PTR exists" landed in the same column. Those mean
opposite things. Meta's IPv6 crawler space publishes no PTR records at
all, so every genuine Meta request was reported under a heading that
read as forgery - an accusation the data did not support.
Split the outcome three ways: FORGED (a PTR exists and points
elsewhere - the only column that is evidence of a lie), no-PTR
(nothing to check, unknown), and PTR ok.
Add an ASN fallback for the no-PTR case, restricted to vendors whose
ASN is single-tenant. Meta is the motivating case: it publishes no
range file - their documentation says to email webmasters@meta.com -
and no PTR records, so without this there is nothing to check at all.
AS32934 is Meta's own network and cannot be rented, which is what
makes the match meaningful. This is deliberately not applied to cloud
ASNs: genuine YouBot and genuine Amazonbot both live in AS14618
alongside every EC2 instance a spoofer could rent, so a match there
would prove only that the traffic came from a cloud.
Verified against real traffic: Meta now shows 1 ASN-confirmed request
against 14 forged ones from googleusercontent.com hosts, which is the
distinction the old single column destroyed.
Assisted-by: Claude:opus-5
Amazonbot, meta-externalagent, YouBot, Bytespider and PetalBot publish
no IP range file, so they were counted in the user-agent tally and
nowhere else - the largest single source of traffic on the sample
instance was also the least examined.
Check them by FCrDNS: the PTR record must end in a vendor domain and
that hostname must resolve back to the same address. The forward step
is the part that matters. A PTR record alone is written by whoever
controls the address block, so without confirming it forward the check
would accept anything its owner chose to claim.
ASN verification was considered and deliberately left out. Genuine
YouBot and genuine Amazonbot both live in AS14618, which is also every
EC2 instance a spoofer could rent. An ASN match shows the traffic came
from a cloud, not from the vendor; presenting that as verification
would be worse than presenting nothing.
Report RFC1918 sources separately rather than as failures. On the
sample instance 21624 of meta-externalagent's 21651 requests came from
172.18.0.1, the Docker bridge gateway: the real client address was
replaced before it reached the log. Those are not spoofed, they are
unjudgeable, and calling them spoofed would be a false accusation
caused by the reader's own proxy configuration.
Resolution is capped at CRAWLER_RDNS_MAX unique addresses per bot
(busiest first) so a flood of distinct forgeries cannot stall the
weekly mail on DNS timeouts, and can be disabled with CRAWLER_RDNS=0.
Assisted-by: Claude:opus-5
Everything was hardcoded for one deployment: hostname in mail subjects,
recipient address, log path, and a router name of "forgejo@docker" that
also appeared inside the user-agent regex. That last one fails silently
rather than loudly - on any instance whose Traefik router is named
something else, the regex matches nothing, every user-agent is dropped,
and the report claims zero bot traffic instead of erroring.
Move configuration to environment variables read from
/etc/crawler-alert.env, with defaults that suit a plain Traefik host.
Only CRAWLER_RECIPIENT is required, and the script now refuses to run
without it rather than mailing into the void. Anchor the user-agent
regex on the request counter instead of the router name.
SMTP can now be configured directly via CRAWLER_SMTP_*, so the tool no
longer requires Forgejo in Docker; borrowing credentials from app.ini
stays the default since it avoids a second copy of the password.
Unauthenticated and non-TLS local relays are handled.
Document the Traefik access log configuration in
traefik-accesslog.yml. This is the one real prerequisite: Traefik drops
all headers by default, so without an explicit User-Agent: keep there
is nothing to analyse. Includes the field layout the parser expects,
logrotate config, and the forwardedHeaders setup needed to recover real
client IPs from behind Cloudflare.
Rewrite README for someone arriving without context, and add a 0BSD
LICENSE so the code can actually be reused.
Assisted-by: Claude:opus-5
The Traefik access log shows a large volume of requests carrying AI
crawler user-agents, but a user-agent header is self-declared and free
to forge. Over 2.27M requests (Nov 2025 - Aug 2026), most of that
traffic did not originate from the vendor it named: Claude-User was
4992/5000 forged, ChatGPT-User 2657/2670, Google-Extended 747/747. A
pool of roughly 100 Google Cloud hosts rotates through six vendor
identities and requests /.env, /terraform.tfstate and
/serviceAccountKey.json - credential harvesting, not crawling.
Counting these by user-agent alone makes the report worse than
useless: it attributes scraper load to vendors that never sent it, and
hides the fact that genuine crawler traffic is roughly 0.1% of
requests and fully robots.txt-compliant.
Verify instead against the vendors' published IP range lists (OpenAI,
Anthropic, Google, Perplexity, Apple) and split each bot into
verified / spoofed / via-Cloudflare buckets. Requests arriving through
Cloudflare are reported separately rather than guessed at, since the
logged address is the CF edge and not the client.
Ranges are fetched at run time and cached, with fallback to the cache
when a vendor endpoint is unreachable, so an outage degrades the
report rather than breaking the weekly mail. The fetch sends an
explicit User-Agent: claude.com and cloudflare.com sit behind
Cloudflare, which 403s the default Python-urllib agent.
Only --weekly performs the fetch; the daily threshold alert is
unchanged and still runs without network access.
Assisted-by: Claude:opus-5
