asxpi/traefik-crawlers-statistics
asxpi/traefik-crawlers-statistics
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Document the DNS/ASN columns and the IPv6 blind spot

Browse source 
The README described a two-column rDNS result that no longer exists,
and still said ASN verification was rejected outright - true for cloud
ASNs, wrong now that Meta's single-tenant AS32934 is used as a
fallback. Explain all four outcomes and why the FORGED column is the
only one that means someone lied.

Also document the IPv6 failure mode, which cost a long investigation
to find and is invisible until someone looks: a host publishing a port
on :: while the Docker network has EnableIPv6=false silently logs every
IPv6 visitor as the bridge gateway, because docker-proxy relays the
connection in userland and the original address is never forwarded.
On the sample instance that hid the entire IPv6 population - 21,624
requests - behind one address, and made genuine Meta traffic look like
it came from inside the network. Includes the two commands that detect
it.

Assisted-by: Claude:opus-5
This commit is contained in:
Sergei Poljanski 2026-08-11 04:53:24 +04:00
commit 2e9f48e4d2
Signed by: asxpi
GPG key ID: 4F8851660FA4121B
1 changed files with 70 additions and 17 deletions
Download patch file Download diff file Expand all files Collapse all files

87
README.md
View file

@ -59,7 +59,9 @@ Meta, ByteDance, Amazon and You.com publish no range file. Those are
checked by **forward-confirmed reverse DNS** instead: the address's PTR checked by **forward-confirmed reverse DNS** instead: the address's PTR
record must end in a vendor domain, *and* that hostname must resolve record must end in a vendor domain, *and* that hostname must resolve
back to the same address. The forward step is what makes it evidence — back to the same address. The forward step is what makes it evidence —
a PTR alone is written by whoever controls the address block. a PTR alone is written by whoever controls the address block. Where
even a PTR is absent, a single-tenant ASN match is used as a fallback
(see below).
| Bot | rDNS suffix | | Bot | rDNS suffix |
|---|---| |---|---|
@ -73,15 +75,35 @@ a PTR alone is written by whoever controls the address block.
Diffbot, Cohere and Common Crawl publish neither ranges nor a Diffbot, Cohere and Common Crawl publish neither ranges nor a
documented rDNS convention; they appear in the user-agent tally only. documented rDNS convention; they appear in the user-agent tally only.
### Why there is no ASN verification ### ASN, and why it is used for exactly one vendor
ASN was considered and rejected. It looks like verification but is not: An ASN match is usually worthless as verification: genuine YouBot
on the sample instance, genuine YouBot (`68.67.112.227`) and genuine (`68.67.112.227`) and genuine Amazonbot (`100.24.167.60`) both live in
Amazonbot (`100.24.167.60`) both resolve to **AS14618, Amazon** — the **AS14618, Amazon** — the same ASN as every EC2 instance on the
same ASN as every EC2 instance on the internet, including any spoofer internet, including any spoofer renting one. Matching there would prove
renting one. An ASN match proves the traffic came from a cloud the traffic came from a cloud, not from the vendor.
provider, not from the vendor. Reporting that as "verified" would be
worse than reporting nothing. It is only evidence when the ASN is single-tenant. **Meta** is the one
case that qualifies here: `AS32934` is Meta's own network and sells no
compute, Meta publishes no IP range file (their documentation says to
email `webmasters@meta.com`), and their IPv6 crawler space carries no
PTR records — so without ASN there is nothing to check them against at
all. The allowlist is `BOT_ASN` in the script; adding a cloud ASN to it
would defeat the purpose. Disable with `CRAWLER_ASN=0`.
### The four outcomes
| Column | Meaning |
|---|---|
| `PTR ok` | Forward-confirmed reverse DNS |
| `ASN ok` | No PTR, but address is in the vendor's single-tenant ASN |
| `FORGED` | A PTR exists and points elsewhere — **the only column that is evidence of a lie** |
| `no-PTR` | No PTR and no usable ASN — unknown, not accused |
The split matters. An earlier version had one "failed" bucket, which
put "lying about who you are" and "your vendor doesn't publish DNS
records" in the same number — and reported every genuine Meta request
as though it were a forgery.
Ranges are fetched at run time and cached. If a vendor endpoint is Ranges are fetched at run time and cached. If a vendor endpoint is
unreachable the script falls back to the cache and says so in the mail, unreachable the script falls back to the cache and says so in the mail,
@ -158,7 +180,23 @@ Only `--weekly` fetches ranges; the daily check needs no network.
1586 failed IP verification (spoofed). 0 arrived via Cloudflare 1586 failed IP verification (spoofed). 0 arrived via Cloudflare
and cannot be verified by IP - not counted either way. and cannot be verified by IP - not counted either way.
Top paths fetched by verified crawlers: --------------------------------------------------------------
VERIFIED BY DNS / ASN (vendors publishing no IP ranges)
--------------------------------------------------------------
bot PTR ok ASN ok FORGED no-PTR CF priv IPs
meta-externalagent 0 1 14 0 0 21636 1
Amazonbot 47 0 1361 6 264 0 45
YouBot 90 0 0 0 0 0 22
Bytespider 0 0 11 0 2 0 0
Reading that: YouBot is entirely genuine. Amazonbot is mostly not —
1,361 requests carry a PTR pointing somewhere other than Amazon.
Meta's single `ASN ok` is a real request from AS32934; its 14 `FORGED`
resolve to `googleusercontent.com`, i.e. GCP hosts wearing Meta's
name. The 21,636 under `priv` are the IPv6 blind spot described in
Limitations, not a judgement.
Top paths, verified crawlers:
117 /robots.txt 117 /robots.txt
73 /sitemap.xml 73 /sitemap.xml
13 /explore/repos 13 /explore/repos
@ -177,13 +215,28 @@ from those two may fail verification. A 0% verified rate for a small
vendor is weaker evidence than it looks. vendor is weaker evidence than it looks.
**Private source addresses cannot be judged at all.** If the logged **Private source addresses cannot be judged at all.** If the logged
client IP is RFC1918 (`172.18.0.1`, a Docker bridge gateway, say), then client IP is RFC1918 (`172.18.0.1`, a Docker bridge gateway, say), the
something in front of Traefik replaced the real address before it was real address was replaced before it was written. Those hits go in the
written. Those hits are counted in a `private` column, neither verified `priv` column, neither verified nor spoofed.
nor spoofed. On the sample instance this was 21,624 of Meta's 21,651
requests — the tool cannot tell you whether they were genuine, only The most common cause is worth calling out, because it is silent and
that the log does not contain the evidence. Fixing it is a Traefik it is easy to have without knowing. **If your host publishes a port on
`forwardedHeaders` change, not a script change. `::` but the Docker network has `EnableIPv6=false`, every IPv6 visitor
is logged as the bridge gateway.** The kernel cannot DNAT v6 into a
v4-only bridge, so `docker-proxy` accepts the connection in userland
and opens a fresh IPv4 connection to your reverse proxy — the original
address is not forwarded, it is gone.
On the sample instance this hid 21,624 requests, the entire IPv6
population, behind one address. Check with:
docker network inspect <net> --format '{{.EnableIPv6}}'
dig +short AAAA your.site
If that returns `false` and an AAAA record exists, you have this
problem. Fixing it means recreating the network with `--ipv6` and a
subnet from your provider's prefix, which requires restarting every
container attached to it.
**rDNS is rate-limited by design.** Each unique address costs two DNS **rDNS is rate-limited by design.** Each unique address costs two DNS
lookups, so only the busiest `CRAWLER_RDNS_MAX` (default 400) addresses lookups, so only the busiest `CRAWLER_RDNS_MAX` (default 400) addresses