diff --git a/README.md b/README.md index 04250aa..bf1c8df 100644 --- a/README.md +++ b/README.md @@ -59,7 +59,9 @@ Meta, ByteDance, Amazon and You.com publish no range file. Those are checked by **forward-confirmed reverse DNS** instead: the address's PTR record must end in a vendor domain, *and* that hostname must resolve back to the same address. The forward step is what makes it evidence — -a PTR alone is written by whoever controls the address block. +a PTR alone is written by whoever controls the address block. Where +even a PTR is absent, a single-tenant ASN match is used as a fallback +(see below). | Bot | rDNS suffix | |---|---| @@ -73,15 +75,35 @@ a PTR alone is written by whoever controls the address block. Diffbot, Cohere and Common Crawl publish neither ranges nor a documented rDNS convention; they appear in the user-agent tally only. -### Why there is no ASN verification +### ASN, and why it is used for exactly one vendor -ASN was considered and rejected. It looks like verification but is not: -on the sample instance, genuine YouBot (`68.67.112.227`) and genuine -Amazonbot (`100.24.167.60`) both resolve to **AS14618, Amazon** — the -same ASN as every EC2 instance on the internet, including any spoofer -renting one. An ASN match proves the traffic came from a cloud -provider, not from the vendor. Reporting that as "verified" would be -worse than reporting nothing. +An ASN match is usually worthless as verification: genuine YouBot +(`68.67.112.227`) and genuine Amazonbot (`100.24.167.60`) both live in +**AS14618, Amazon** — the same ASN as every EC2 instance on the +internet, including any spoofer renting one. Matching there would prove +the traffic came from a cloud, not from the vendor. + +It is only evidence when the ASN is single-tenant. **Meta** is the one +case that qualifies here: `AS32934` is Meta's own network and sells no +compute, Meta publishes no IP range file (their documentation says to +email `webmasters@meta.com`), and their IPv6 crawler space carries no +PTR records — so without ASN there is nothing to check them against at +all. The allowlist is `BOT_ASN` in the script; adding a cloud ASN to it +would defeat the purpose. Disable with `CRAWLER_ASN=0`. + +### The four outcomes + +| Column | Meaning | +|---|---| +| `PTR ok` | Forward-confirmed reverse DNS | +| `ASN ok` | No PTR, but address is in the vendor's single-tenant ASN | +| `FORGED` | A PTR exists and points elsewhere — **the only column that is evidence of a lie** | +| `no-PTR` | No PTR and no usable ASN — unknown, not accused | + +The split matters. An earlier version had one "failed" bucket, which +put "lying about who you are" and "your vendor doesn't publish DNS +records" in the same number — and reported every genuine Meta request +as though it were a forgery. Ranges are fetched at run time and cached. If a vendor endpoint is unreachable the script falls back to the cache and says so in the mail, @@ -158,7 +180,23 @@ Only `--weekly` fetches ranges; the daily check needs no network. 1586 failed IP verification (spoofed). 0 arrived via Cloudflare and cannot be verified by IP - not counted either way. - Top paths fetched by verified crawlers: + -------------------------------------------------------------- + VERIFIED BY DNS / ASN (vendors publishing no IP ranges) + -------------------------------------------------------------- + bot PTR ok ASN ok FORGED no-PTR CF priv IPs + meta-externalagent 0 1 14 0 0 21636 1 + Amazonbot 47 0 1361 6 264 0 45 + YouBot 90 0 0 0 0 0 22 + Bytespider 0 0 11 0 2 0 0 + +Reading that: YouBot is entirely genuine. Amazonbot is mostly not — +1,361 requests carry a PTR pointing somewhere other than Amazon. +Meta's single `ASN ok` is a real request from AS32934; its 14 `FORGED` +resolve to `googleusercontent.com`, i.e. GCP hosts wearing Meta's +name. The 21,636 under `priv` are the IPv6 blind spot described in +Limitations, not a judgement. + + Top paths, verified crawlers: 117 /robots.txt 73 /sitemap.xml 13 /explore/repos @@ -177,13 +215,28 @@ from those two may fail verification. A 0% verified rate for a small vendor is weaker evidence than it looks. **Private source addresses cannot be judged at all.** If the logged -client IP is RFC1918 (`172.18.0.1`, a Docker bridge gateway, say), then -something in front of Traefik replaced the real address before it was -written. Those hits are counted in a `private` column, neither verified -nor spoofed. On the sample instance this was 21,624 of Meta's 21,651 -requests — the tool cannot tell you whether they were genuine, only -that the log does not contain the evidence. Fixing it is a Traefik -`forwardedHeaders` change, not a script change. +client IP is RFC1918 (`172.18.0.1`, a Docker bridge gateway, say), the +real address was replaced before it was written. Those hits go in the +`priv` column, neither verified nor spoofed. + +The most common cause is worth calling out, because it is silent and +it is easy to have without knowing. **If your host publishes a port on +`::` but the Docker network has `EnableIPv6=false`, every IPv6 visitor +is logged as the bridge gateway.** The kernel cannot DNAT v6 into a +v4-only bridge, so `docker-proxy` accepts the connection in userland +and opens a fresh IPv4 connection to your reverse proxy — the original +address is not forwarded, it is gone. + +On the sample instance this hid 21,624 requests, the entire IPv6 +population, behind one address. Check with: + + docker network inspect --format '{{.EnableIPv6}}' + dig +short AAAA your.site + +If that returns `false` and an AAAA record exists, you have this +problem. Fixing it means recreating the network with `--ipv6` and a +subnet from your provider's prefix, which requires restarting every +container attached to it. **rDNS is rate-limited by design.** Each unique address costs two DNS lookups, so only the busiest `CRAWLER_RDNS_MAX` (default 400) addresses