The README described a two-column rDNS result that no longer exists,
and still said ASN verification was rejected outright - true for cloud
ASNs, wrong now that Meta's single-tenant AS32934 is used as a
fallback. Explain all four outcomes and why the FORGED column is the
only one that means someone lied.
Also document the IPv6 failure mode, which cost a long investigation
to find and is invisible until someone looks: a host publishing a port
on :: while the Docker network has EnableIPv6=false silently logs every
IPv6 visitor as the bridge gateway, because docker-proxy relays the
connection in userland and the original address is never forwarded.
On the sample instance that hid the entire IPv6 population - 21,624
requests - behind one address, and made genuine Meta traffic look like
it came from inside the network. Includes the two commands that detect
it.
Assisted-by: Claude:opus-5
Amazonbot, meta-externalagent, YouBot, Bytespider and PetalBot publish
no IP range file, so they were counted in the user-agent tally and
nowhere else - the largest single source of traffic on the sample
instance was also the least examined.
Check them by FCrDNS: the PTR record must end in a vendor domain and
that hostname must resolve back to the same address. The forward step
is the part that matters. A PTR record alone is written by whoever
controls the address block, so without confirming it forward the check
would accept anything its owner chose to claim.
ASN verification was considered and deliberately left out. Genuine
YouBot and genuine Amazonbot both live in AS14618, which is also every
EC2 instance a spoofer could rent. An ASN match shows the traffic came
from a cloud, not from the vendor; presenting that as verification
would be worse than presenting nothing.
Report RFC1918 sources separately rather than as failures. On the
sample instance 21624 of meta-externalagent's 21651 requests came from
172.18.0.1, the Docker bridge gateway: the real client address was
replaced before it reached the log. Those are not spoofed, they are
unjudgeable, and calling them spoofed would be a false accusation
caused by the reader's own proxy configuration.
Resolution is capped at CRAWLER_RDNS_MAX unique addresses per bot
(busiest first) so a flood of distinct forgeries cannot stall the
weekly mail on DNS timeouts, and can be disabled with CRAWLER_RDNS=0.
Assisted-by: Claude:opus-5
Everything was hardcoded for one deployment: hostname in mail subjects,
recipient address, log path, and a router name of "forgejo@docker" that
also appeared inside the user-agent regex. That last one fails silently
rather than loudly - on any instance whose Traefik router is named
something else, the regex matches nothing, every user-agent is dropped,
and the report claims zero bot traffic instead of erroring.
Move configuration to environment variables read from
/etc/crawler-alert.env, with defaults that suit a plain Traefik host.
Only CRAWLER_RECIPIENT is required, and the script now refuses to run
without it rather than mailing into the void. Anchor the user-agent
regex on the request counter instead of the router name.
SMTP can now be configured directly via CRAWLER_SMTP_*, so the tool no
longer requires Forgejo in Docker; borrowing credentials from app.ini
stays the default since it avoids a second copy of the password.
Unauthenticated and non-TLS local relays are handled.
Document the Traefik access log configuration in
traefik-accesslog.yml. This is the one real prerequisite: Traefik drops
all headers by default, so without an explicit User-Agent: keep there
is nothing to analyse. Includes the field layout the parser expects,
logrotate config, and the forwardedHeaders setup needed to recover real
client IPs from behind Cloudflare.
Rewrite README for someone arriving without context, and add a 0BSD
LICENSE so the code can actually be reused.
Assisted-by: Claude:opus-5
