Separate forged from unverifiable, add ASN check for Meta
The rDNS check had one failure bucket, so "PTR points at a different network" and "no PTR exists" landed in the same column. Those mean opposite things. Meta's IPv6 crawler space publishes no PTR records at all, so every genuine Meta request was reported under a heading that read as forgery - an accusation the data did not support. Split the outcome three ways: FORGED (a PTR exists and points elsewhere - the only column that is evidence of a lie), no-PTR (nothing to check, unknown), and PTR ok. Add an ASN fallback for the no-PTR case, restricted to vendors whose ASN is single-tenant. Meta is the motivating case: it publishes no range file - their documentation says to email webmasters@meta.com - and no PTR records, so without this there is nothing to check at all. AS32934 is Meta's own network and cannot be rented, which is what makes the match meaningful. This is deliberately not applied to cloud ASNs: genuine YouBot and genuine Amazonbot both live in AS14618 alongside every EC2 instance a spoofer could rent, so a match there would prove only that the traffic came from a cloud. Verified against real traffic: Meta now shows 1 ASN-confirmed request against 14 forged ones from googleusercontent.com hosts, which is the distinction the old single column destroyed. Assisted-by: Claude:opus-5
This commit is contained in:
parent
81716c2fc9
commit
43a88182cd
2 changed files with 101 additions and 25 deletions
|
|
@ -47,3 +47,9 @@ CRAWLER_RDNS=1
|
|||
# Max unique addresses resolved per bot per run. Each costs two DNS
|
||||
# lookups; the busiest addresses are resolved first.
|
||||
CRAWLER_RDNS_MAX=400
|
||||
|
||||
# ASN fallback for vendors with no PTR records and no published ranges
|
||||
# (currently Meta / AS32934 only). Used ONLY where the ASN is
|
||||
# single-tenant; never for cloud ASNs where anyone can rent space.
|
||||
# Requires `dig`. Set to 0 to disable.
|
||||
CRAWLER_ASN=1
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue