Sergei Poljanski
5860cbee9a
Everything was hardcoded for one deployment: hostname in mail subjects, recipient address, log path, and a router name of "forgejo@docker" that also appeared inside the user-agent regex. That last one fails silently rather than loudly - on any instance whose Traefik router is named something else, the regex matches nothing, every user-agent is dropped, and the report claims zero bot traffic instead of erroring. Move configuration to environment variables read from /etc/crawler-alert.env, with defaults that suit a plain Traefik host. Only CRAWLER_RECIPIENT is required, and the script now refuses to run without it rather than mailing into the void. Anchor the user-agent regex on the request counter instead of the router name. SMTP can now be configured directly via CRAWLER_SMTP_*, so the tool no longer requires Forgejo in Docker; borrowing credentials from app.ini stays the default since it avoids a second copy of the password. Unauthenticated and non-TLS local relays are handled. Document the Traefik access log configuration in traefik-accesslog.yml. This is the one real prerequisite: Traefik drops all headers by default, so without an explicit User-Agent: keep there is nothing to analyse. Includes the field layout the parser expects, logrotate config, and the forwardedHeaders setup needed to recover real client IPs from behind Cloudflare. Rewrite README for someone arriving without context, and add a 0BSD LICENSE so the code can actually be reused. Assisted-by: Claude:opus-5
83 lines
2.8 KiB
YAML
83 lines
2.8 KiB
YAML
# Traefik static configuration: the access log this script needs.
|
|
# Merge into your existing traefik.yml (or docker-compose command flags).
|
|
#
|
|
# The critical part is keeping the User-Agent header. Traefik drops all
|
|
# headers by default, and without User-Agent every request is anonymous
|
|
# and no bot analysis is possible at all.
|
|
|
|
accessLog:
|
|
filePath: /var/log/traefik/access.log
|
|
# Traefik's default. The script parses this, not JSON.
|
|
format: common
|
|
bufferingSize: 100
|
|
fields:
|
|
# Keep the standard CLF fields.
|
|
defaultMode: keep
|
|
headers:
|
|
# Drop every header except the two named below: an access log
|
|
# holding Cookie or Authorization is a liability, not telemetry.
|
|
defaultMode: drop
|
|
names:
|
|
User-Agent: keep
|
|
Referer: keep
|
|
|
|
# Resulting line (one line, wrapped here for readability):
|
|
#
|
|
# 1.2.3.4 - - [10/Aug/2026:22:30:24 +0000] "GET /explore HTTP/2.0"
|
|
# 200 12062 "https://git.example.com/" "Mozilla/5.0 ... ClaudeBot/1.0"
|
|
# 93971 "forgejo@docker" "http://172.18.0.3:3000" 2ms
|
|
#
|
|
# Fields, in order:
|
|
# 1 client IP <- verified against vendor ranges
|
|
# 2 ident, 3 user (always "-")
|
|
# 4 timestamp <- day bucketing
|
|
# 5 request line <- path, for the per-path breakdown
|
|
# 6 status, 7 bytes
|
|
# 8 Referer (kept above)
|
|
# 9 User-Agent <- bot token
|
|
# 10 request counter
|
|
# 11 router name <- CRAWLER_ROUTER filter, e.g. forgejo@docker
|
|
# 12 backend URL, 13 duration
|
|
#
|
|
# Mount the log directory so both Traefik and the host can see it:
|
|
#
|
|
# volumes:
|
|
# - /opt/traefik/logs:/var/log/traefik
|
|
#
|
|
# Rotate it. Traefik reopens on USR1; without rotation this file grows
|
|
# without bound (it reached 533 MB in nine months on the instance this
|
|
# was written for). /etc/logrotate.d/traefik:
|
|
#
|
|
# /opt/traefik/logs/*.log {
|
|
# daily
|
|
# rotate 14
|
|
# compress
|
|
# delaycompress
|
|
# missingok
|
|
# notifempty
|
|
# postrotate
|
|
# docker kill --signal=USR1 traefik 2>/dev/null || true
|
|
# endscript
|
|
# }
|
|
#
|
|
# The script reads rotated and gzipped files too, as long as
|
|
# CRAWLER_LOG keeps its trailing glob (access.log*).
|
|
|
|
# --- If you sit behind Cloudflare -------------------------------------
|
|
# The logged client IP will be Cloudflare's edge, not the real visitor,
|
|
# and no such request can be IP-verified (the report counts these in a
|
|
# separate "via CF" column). To recover real IPs, trust CF's ranges on
|
|
# the entrypoint:
|
|
#
|
|
# entryPoints:
|
|
# websecure:
|
|
# address: ":443"
|
|
# forwardedHeaders:
|
|
# trustedIPs:
|
|
# # https://www.cloudflare.com/ips-v4 - keep this list current
|
|
# - 173.245.48.0/20
|
|
# - 103.21.244.0/22
|
|
# # ... remaining CF ranges ...
|
|
#
|
|
# Only list address ranges you actually trust: anyone in trustedIPs can
|
|
# spoof X-Forwarded-For and forge their apparent source address.
|