Separate forged from unverifiable, add ASN check for Meta

The rDNS check had one failure bucket, so "PTR points at a different
network" and "no PTR exists" landed in the same column. Those mean
opposite things. Meta's IPv6 crawler space publishes no PTR records at
all, so every genuine Meta request was reported under a heading that
read as forgery - an accusation the data did not support.

Split the outcome three ways: FORGED (a PTR exists and points
elsewhere - the only column that is evidence of a lie), no-PTR
(nothing to check, unknown), and PTR ok.

Add an ASN fallback for the no-PTR case, restricted to vendors whose
ASN is single-tenant. Meta is the motivating case: it publishes no
range file - their documentation says to email webmasters@meta.com -
and no PTR records, so without this there is nothing to check at all.
AS32934 is Meta's own network and cannot be rented, which is what
makes the match meaningful. This is deliberately not applied to cloud
ASNs: genuine YouBot and genuine Amazonbot both live in AS14618
alongside every EC2 instance a spoofer could rent, so a match there
would prove only that the traffic came from a cloud.

Verified against real traffic: Meta now shows 1 ASN-confirmed request
against 14 forged ones from googleusercontent.com hosts, which is the
distinction the old single column destroyed.

Assisted-by: Claude:opus-5
This commit is contained in:
Sergei Poljanski 2026-08-11 04:52:15 +04:00
commit 43a88182cd
Signed by: asxpi
GPG key ID: 4F8851660FA4121B
2 changed files with 101 additions and 25 deletions

View file

@ -47,3 +47,9 @@ CRAWLER_RDNS=1
# Max unique addresses resolved per bot per run. Each costs two DNS
# lookups; the busiest addresses are resolved first.
CRAWLER_RDNS_MAX=400
# ASN fallback for vendors with no PTR records and no published ranges
# (currently Meta / AS32934 only). Used ONLY where the ASN is
# single-tenant; never for cloud ASNs where anyone can rent space.
# Requires `dig`. Set to 0 to disable.
CRAWLER_ASN=1