Make the tool reusable on other instances

Everything was hardcoded for one deployment: hostname in mail subjects,
recipient address, log path, and a router name of "forgejo@docker" that
also appeared inside the user-agent regex. That last one fails silently
rather than loudly - on any instance whose Traefik router is named
something else, the regex matches nothing, every user-agent is dropped,
and the report claims zero bot traffic instead of erroring.

Move configuration to environment variables read from
/etc/crawler-alert.env, with defaults that suit a plain Traefik host.
Only CRAWLER_RECIPIENT is required, and the script now refuses to run
without it rather than mailing into the void. Anchor the user-agent
regex on the request counter instead of the router name.

SMTP can now be configured directly via CRAWLER_SMTP_*, so the tool no
longer requires Forgejo in Docker; borrowing credentials from app.ini
stays the default since it avoids a second copy of the password.
Unauthenticated and non-TLS local relays are handled.

Document the Traefik access log configuration in
traefik-accesslog.yml. This is the one real prerequisite: Traefik drops
all headers by default, so without an explicit User-Agent: keep there
is nothing to analyse. Includes the field layout the parser expects,
logrotate config, and the forwardedHeaders setup needed to recover real
client IPs from behind Cloudflare.

Rewrite README for someone arriving without context, and add a 0BSD
LICENSE so the code can actually be reused.

Assisted-by: Claude:opus-5
This commit is contained in:
Sergei Poljanski 2026-08-11 02:49:51 +04:00
commit 5860cbee9a
Signed by: asxpi
GPG key ID: 4F8851660FA4121B
7 changed files with 343 additions and 85 deletions

View file

@ -1,7 +1,8 @@
[Unit]
Description=Weekly Forgejo crawler report mail
Description=Weekly crawler report mail
After=docker.service
[Service]
Type=oneshot
EnvironmentFile=/etc/crawler-alert.env
ExecStart=/usr/local/bin/crawler-alert.py --weekly