Sergei Poljanski
81716c2fc9
Amazonbot, meta-externalagent, YouBot, Bytespider and PetalBot publish no IP range file, so they were counted in the user-agent tally and nowhere else - the largest single source of traffic on the sample instance was also the least examined. Check them by FCrDNS: the PTR record must end in a vendor domain and that hostname must resolve back to the same address. The forward step is the part that matters. A PTR record alone is written by whoever controls the address block, so without confirming it forward the check would accept anything its owner chose to claim. ASN verification was considered and deliberately left out. Genuine YouBot and genuine Amazonbot both live in AS14618, which is also every EC2 instance a spoofer could rent. An ASN match shows the traffic came from a cloud, not from the vendor; presenting that as verification would be worse than presenting nothing. Report RFC1918 sources separately rather than as failures. On the sample instance 21624 of meta-externalagent's 21651 requests came from 172.18.0.1, the Docker bridge gateway: the real client address was replaced before it reached the log. Those are not spoofed, they are unjudgeable, and calling them spoofed would be a false accusation caused by the reader's own proxy configuration. Resolution is capped at CRAWLER_RDNS_MAX unique addresses per bot (busiest first) so a flood of distinct forgeries cannot stall the weekly mail on DNS timeouts, and can be disabled with CRAWLER_RDNS=0. Assisted-by: Claude:opus-5
49 lines
1.9 KiB
Text
49 lines
1.9 KiB
Text
# Copy to /etc/crawler-alert.env and edit.
|
|
# Only CRAWLER_RECIPIENT is required.
|
|
|
|
# Where the reports are mailed.
|
|
CRAWLER_RECIPIENT=you@example.com
|
|
|
|
# Name used in mail subjects and the report header.
|
|
CRAWLER_SITE=git.example.com
|
|
|
|
# Traefik access log. Glob so rotated/gzipped files are included.
|
|
CRAWLER_LOG=/var/log/traefik/access.log*
|
|
|
|
# Traefik router to count, e.g. forgejo@docker. Leave empty to count
|
|
# everything that reaches Traefik (right for a single-site host).
|
|
CRAWLER_ROUTER=
|
|
|
|
# Daily alert fires when yesterday exceeds this many requests.
|
|
# Set it near your normal peak, not far above it: a threshold that never
|
|
# fires is not a tripwire. Check a few days of real traffic first with
|
|
# crawler-alert.py --dry-run
|
|
CRAWLER_THRESHOLD=100000
|
|
|
|
# Cached vendor IP ranges; used when a vendor endpoint is unreachable.
|
|
CRAWLER_CACHE=/var/cache/crawler-alert/ranges.json
|
|
|
|
# --- SMTP -------------------------------------------------------------
|
|
# By default the script reads SMTP settings from a Forgejo/Gitea
|
|
# app.ini [mailer] section, so credentials live in one place only:
|
|
CRAWLER_MAIL_CONTAINER=forgejo
|
|
CRAWLER_MAIL_INI=/data/gitea/conf/app.ini
|
|
|
|
# Or configure SMTP directly and ignore the container entirely.
|
|
# Setting CRAWLER_SMTP_ADDR switches to this path.
|
|
#CRAWLER_SMTP_ADDR=smtp.example.com
|
|
#CRAWLER_SMTP_PORT=587
|
|
#CRAWLER_SMTP_USER=alerts@example.com
|
|
#CRAWLER_SMTP_PASSWORD=
|
|
#CRAWLER_SMTP_FROM=alerts@example.com
|
|
# smtp+starttls (default) | smtps | smtp (no TLS, local relay)
|
|
#CRAWLER_SMTP_PROTOCOL=smtp+starttls
|
|
|
|
# --- reverse DNS ------------------------------------------------------
|
|
# FCrDNS verification for vendors that publish no IP ranges
|
|
# (Amazonbot, meta-externalagent, YouBot, Bytespider, PetalBot).
|
|
# Set to 0 to disable if your resolver is slow or unavailable.
|
|
CRAWLER_RDNS=1
|
|
# Max unique addresses resolved per bot per run. Each costs two DNS
|
|
# lookups; the busiest addresses are resolved first.
|
|
CRAWLER_RDNS_MAX=400
|