asxpi/traefik-crawlers-statistics
asxpi/traefik-crawlers-statistics
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Document the DNS/ASN columns and the IPv6 blind spot

Browse source 
The README described a two-column rDNS result that no longer exists,
and still said ASN verification was rejected outright - true for cloud
ASNs, wrong now that Meta's single-tenant AS32934 is used as a
fallback. Explain all four outcomes and why the FORGED column is the
only one that means someone lied.

Also document the IPv6 failure mode, which cost a long investigation
to find and is invisible until someone looks: a host publishing a port
on :: while the Docker network has EnableIPv6=false silently logs every
IPv6 visitor as the bridge gateway, because docker-proxy relays the
connection in userland and the original address is never forwarded.
On the sample instance that hid the entire IPv6 population - 21,624
requests - behind one address, and made genuine Meta traffic look like
it came from inside the network. Includes the two commands that detect
it.

Assisted-by: Claude:opus-5
This commit is contained in:
Sergei Poljanski 2026-08-11 04:53:24 +04:00
commit 2e9f48e4d2
Signed by: asxpi
GPG key ID: 4F8851660FA4121B
1 changed files with 70 additions and 17 deletions
Download patch file Download diff file Expand all files Collapse all files

87
README.md
View file

@ -59,7 +59,9 @@ Meta, ByteDance, Amazon and You.com publish no range file. Those are
checked by **forward-confirmed reverse DNS** instead: the address's PTR
record must end in a vendor domain, *and* that hostname must resolve
back to the same address. The forward step is what makes it evidence —
a PTR alone is written by whoever controls the address block.
a PTR alone is written by whoever controls the address block. Where
even a PTR is absent, a single-tenant ASN match is used as a fallback
(see below).
| Bot | rDNS suffix |
|---|---|
@ -73,15 +75,35 @@ a PTR alone is written by whoever controls the address block.
Diffbot, Cohere and Common Crawl publish neither ranges nor a
documented rDNS convention; they appear in the user-agent tally only.
### Why there is no ASN verification
### ASN, and why it is used for exactly one vendor
ASN was considered and rejected. It looks like verification but is not:
on the sample instance, genuine YouBot (`68.67.112.227`) and genuine
Amazonbot (`100.24.167.60`) both resolve to **AS14618, Amazon** — the
same ASN as every EC2 instance on the internet, including any spoofer
renting one. An ASN match proves the traffic came from a cloud
provider, not from the vendor. Reporting that as "verified" would be
worse than reporting nothing.
An ASN match is usually worthless as verification: genuine YouBot
(`68.67.112.227`) and genuine Amazonbot (`100.24.167.60`) both live in
**AS14618, Amazon** — the same ASN as every EC2 instance on the
internet, including any spoofer renting one. Matching there would prove
the traffic came from a cloud, not from the vendor.
It is only evidence when the ASN is single-tenant. **Meta** is the one
case that qualifies here: `AS32934` is Meta's own network and sells no
compute, Meta publishes no IP range file (their documentation says to
email `webmasters@meta.com`), and their IPv6 crawler space carries no
PTR records — so without ASN there is nothing to check them against at
all. The allowlist is `BOT_ASN` in the script; adding a cloud ASN to it
would defeat the purpose. Disable with `CRAWLER_ASN=0`.
### The four outcomes
| Column | Meaning |
|---|---|
| `PTR ok` | Forward-confirmed reverse DNS |
| `ASN ok` | No PTR, but address is in the vendor's single-tenant ASN |
| `FORGED` | A PTR exists and points elsewhere — **the only column that is evidence of a lie** |
| `no-PTR` | No PTR and no usable ASN — unknown, not accused |
The split matters. An earlier version had one "failed" bucket, which
put "lying about who you are" and "your vendor doesn't publish DNS
records" in the same number — and reported every genuine Meta request
as though it were a forgery.
Ranges are fetched at run time and cached. If a vendor endpoint is
unreachable the script falls back to the cache and says so in the mail,
@ -158,7 +180,23 @@ Only `--weekly` fetches ranges; the daily check needs no network.
1586 failed IP verification (spoofed). 0 arrived via Cloudflare
and cannot be verified by IP - not counted either way.
Top paths fetched by verified crawlers:
--------------------------------------------------------------
VERIFIED BY DNS / ASN (vendors publishing no IP ranges)
--------------------------------------------------------------
bot PTR ok ASN ok FORGED no-PTR CF priv IPs
meta-externalagent 0 1 14 0 0 21636 1
Amazonbot 47 0 1361 6 264 0 45
YouBot 90 0 0 0 0 0 22
Bytespider 0 0 11 0 2 0 0
Reading that: YouBot is entirely genuine. Amazonbot is mostly not —
1,361 requests carry a PTR pointing somewhere other than Amazon.
Meta's single `ASN ok` is a real request from AS32934; its 14 `FORGED`
resolve to `googleusercontent.com`, i.e. GCP hosts wearing Meta's
name. The 21,636 under `priv` are the IPv6 blind spot described in
Limitations, not a judgement.
Top paths, verified crawlers:
117 /robots.txt
73 /sitemap.xml
13 /explore/repos
@ -177,13 +215,28 @@ from those two may fail verification. A 0% verified rate for a small
vendor is weaker evidence than it looks.
**Private source addresses cannot be judged at all.** If the logged
client IP is RFC1918 (`172.18.0.1`, a Docker bridge gateway, say), then
something in front of Traefik replaced the real address before it was
written. Those hits are counted in a `private` column, neither verified
nor spoofed. On the sample instance this was 21,624 of Meta's 21,651
requests — the tool cannot tell you whether they were genuine, only
that the log does not contain the evidence. Fixing it is a Traefik
`forwardedHeaders` change, not a script change.
client IP is RFC1918 (`172.18.0.1`, a Docker bridge gateway, say), the
real address was replaced before it was written. Those hits go in the
`priv` column, neither verified nor spoofed.
The most common cause is worth calling out, because it is silent and
it is easy to have without knowing. **If your host publishes a port on
`::` but the Docker network has `EnableIPv6=false`, every IPv6 visitor
is logged as the bridge gateway.** The kernel cannot DNAT v6 into a
v4-only bridge, so `docker-proxy` accepts the connection in userland
and opens a fresh IPv4 connection to your reverse proxy — the original
address is not forwarded, it is gone.
On the sample instance this hid 21,624 requests, the entire IPv6
population, behind one address. Check with:
docker network inspect <net> --format '{{.EnableIPv6}}'
dig +short AAAA your.site
If that returns `false` and an AAAA record exists, you have this
problem. Fixing it means recreating the network with `--ipv6` and a
subnet from your provider's prefix, which requires restarting every
container attached to it.
**rDNS is rate-limited by design.** Each unique address costs two DNS
lookups, so only the busiest `CRAWLER_RDNS_MAX` (default 400) addresses