asxpi/traefik-crawlers-statistics
asxpi/traefik-crawlers-statistics
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Verify range-less vendors by forward-confirmed reverse DNS

Browse source 
Amazonbot, meta-externalagent, YouBot, Bytespider and PetalBot publish
no IP range file, so they were counted in the user-agent tally and
nowhere else - the largest single source of traffic on the sample
instance was also the least examined.

Check them by FCrDNS: the PTR record must end in a vendor domain and
that hostname must resolve back to the same address. The forward step
is the part that matters. A PTR record alone is written by whoever
controls the address block, so without confirming it forward the check
would accept anything its owner chose to claim.

ASN verification was considered and deliberately left out. Genuine
YouBot and genuine Amazonbot both live in AS14618, which is also every
EC2 instance a spoofer could rent. An ASN match shows the traffic came
from a cloud, not from the vendor; presenting that as verification
would be worse than presenting nothing.

Report RFC1918 sources separately rather than as failures. On the
sample instance 21624 of meta-externalagent's 21651 requests came from
172.18.0.1, the Docker bridge gateway: the real client address was
replaced before it reached the log. Those are not spoofed, they are
unjudgeable, and calling them spoofed would be a false accusation
caused by the reader's own proxy configuration.

Resolution is capped at CRAWLER_RDNS_MAX unique addresses per bot
(busiest first) so a flood of distinct forgeries cannot stall the
weekly mail on DNS timeouts, and can be disabled with CRAWLER_RDNS=0.

Assisted-by: Claude:opus-5
This commit is contained in:
Sergei Poljanski 2026-08-11 02:55:57 +04:00
commit 81716c2fc9
Signed by: asxpi
GPG key ID: 4F8851660FA4121B
3 changed files with 187 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

47
README.md
View file

@ -55,10 +55,33 @@ range list belonging to the vendor its user-agent names.
| Perplexity | `perplexity.ai/perplexitybot.json` |
| Apple | `search.developer.apple.com/applebot.json` |
Meta, ByteDance, Amazon, Diffbot, Cohere and Common Crawl publish no
range file. Their traffic is reported but cannot be verified this way;
Amazon supports reverse-DNS verification instead, which this script
does not currently implement.
Meta, ByteDance, Amazon and You.com publish no range file. Those are
checked by **forward-confirmed reverse DNS** instead: the address's PTR
record must end in a vendor domain, *and* that hostname must resolve
back to the same address. The forward step is what makes it evidence —
a PTR alone is written by whoever controls the address block.
| Bot | rDNS suffix |
|---|---|
| Amazonbot | `.crawl.amazonbot.amazon` |
| meta-externalagent, FacebookBot | `.facebook.com`, `.fbsv.net` |
| YouBot | `.search.you.com` |
| Bytespider | `.bytedance.com`, `.byteoversea.com` |
| PetalBot | `.petalsearch.com`, `.aspiegel.com` |
| DuckAssistBot | `.duckduckgo.com` |
Diffbot, Cohere and Common Crawl publish neither ranges nor a
documented rDNS convention; they appear in the user-agent tally only.
### Why there is no ASN verification
ASN was considered and rejected. It looks like verification but is not:
on the sample instance, genuine YouBot (`68.67.112.227`) and genuine
Amazonbot (`100.24.167.60`) both resolve to **AS14618, Amazon** — the
same ASN as every EC2 instance on the internet, including any spoofer
renting one. An ASN match proves the traffic came from a cloud
provider, not from the vendor. Reporting that as "verified" would be
worse than reporting nothing.
Ranges are fetched at run time and cached. If a vendor endpoint is
unreachable the script falls back to the cache and says so in the mail,
@ -153,9 +176,19 @@ last updated in February 2025; Apple's dates to 2023. Genuine traffic
from those two may fail verification. A 0% verified rate for a small
vendor is weaker evidence than it looks.
**No reverse-DNS verification.** Amazonbot and Meta's crawlers can only
be checked by rDNS or ASN, neither of which is implemented. Their
traffic appears in the user-agent tally but not the verified table.
**Private source addresses cannot be judged at all.** If the logged
client IP is RFC1918 (`172.18.0.1`, a Docker bridge gateway, say), then
something in front of Traefik replaced the real address before it was
written. Those hits are counted in a `private` column, neither verified
nor spoofed. On the sample instance this was 21,624 of Meta's 21,651
requests — the tool cannot tell you whether they were genuine, only
that the log does not contain the evidence. Fixing it is a Traefik
`forwardedHeaders` change, not a script change.
**rDNS is rate-limited by design.** Each unique address costs two DNS
lookups, so only the busiest `CRAWLER_RDNS_MAX` (default 400) addresses
per bot are resolved; the rest are reported as unresolved. Raise it if
your DNS resolver is fast and local.
**IPv6 is verified only where vendors publish v6 prefixes**, which most
do; addresses outside those are treated as unverified.