Sergei Poljanski
5860cbee9a
Everything was hardcoded for one deployment: hostname in mail subjects, recipient address, log path, and a router name of "forgejo@docker" that also appeared inside the user-agent regex. That last one fails silently rather than loudly - on any instance whose Traefik router is named something else, the regex matches nothing, every user-agent is dropped, and the report claims zero bot traffic instead of erroring. Move configuration to environment variables read from /etc/crawler-alert.env, with defaults that suit a plain Traefik host. Only CRAWLER_RECIPIENT is required, and the script now refuses to run without it rather than mailing into the void. Anchor the user-agent regex on the request counter instead of the router name. SMTP can now be configured directly via CRAWLER_SMTP_*, so the tool no longer requires Forgejo in Docker; borrowing credentials from app.ini stays the default since it avoids a second copy of the password. Unauthenticated and non-TLS local relays are handled. Document the Traefik access log configuration in traefik-accesslog.yml. This is the one real prerequisite: Traefik drops all headers by default, so without an explicit User-Agent: keep there is nothing to analyse. Includes the field layout the parser expects, logrotate config, and the forwardedHeaders setup needed to recover real client IPs from behind Cloudflare. Rewrite README for someone arriving without context, and add a 0BSD LICENSE so the code can actually be reused. Assisted-by: Claude:opus-5
40 lines
1.5 KiB
Text
40 lines
1.5 KiB
Text
# Copy to /etc/crawler-alert.env and edit.
|
|
# Only CRAWLER_RECIPIENT is required.
|
|
|
|
# Where the reports are mailed.
|
|
CRAWLER_RECIPIENT=you@example.com
|
|
|
|
# Name used in mail subjects and the report header.
|
|
CRAWLER_SITE=git.example.com
|
|
|
|
# Traefik access log. Glob so rotated/gzipped files are included.
|
|
CRAWLER_LOG=/var/log/traefik/access.log*
|
|
|
|
# Traefik router to count, e.g. forgejo@docker. Leave empty to count
|
|
# everything that reaches Traefik (right for a single-site host).
|
|
CRAWLER_ROUTER=
|
|
|
|
# Daily alert fires when yesterday exceeds this many requests.
|
|
# Set it near your normal peak, not far above it: a threshold that never
|
|
# fires is not a tripwire. Check a few days of real traffic first with
|
|
# crawler-alert.py --dry-run
|
|
CRAWLER_THRESHOLD=100000
|
|
|
|
# Cached vendor IP ranges; used when a vendor endpoint is unreachable.
|
|
CRAWLER_CACHE=/var/cache/crawler-alert/ranges.json
|
|
|
|
# --- SMTP -------------------------------------------------------------
|
|
# By default the script reads SMTP settings from a Forgejo/Gitea
|
|
# app.ini [mailer] section, so credentials live in one place only:
|
|
CRAWLER_MAIL_CONTAINER=forgejo
|
|
CRAWLER_MAIL_INI=/data/gitea/conf/app.ini
|
|
|
|
# Or configure SMTP directly and ignore the container entirely.
|
|
# Setting CRAWLER_SMTP_ADDR switches to this path.
|
|
#CRAWLER_SMTP_ADDR=smtp.example.com
|
|
#CRAWLER_SMTP_PORT=587
|
|
#CRAWLER_SMTP_USER=alerts@example.com
|
|
#CRAWLER_SMTP_PASSWORD=
|
|
#CRAWLER_SMTP_FROM=alerts@example.com
|
|
# smtp+starttls (default) | smtps | smtp (no TLS, local relay)
|
|
#CRAWLER_SMTP_PROTOCOL=smtp+starttls
|