crawler_alert, monitoring: AI crawler metrics and dashboards
Some checks failed
deploy / ansible (push) Failing after 1m17s

crawler-alert.py --metrics (5-minute timer) reads what the Traefik
access log gained since the last run, judges every request that names
an AI bot exactly like the weekly report (vendor IP ranges, forward-
confirmed rDNS, Meta's ASN) and writes cumulative counters to the
node_exporter textfile directory. Verdicts are cached per bot and IP
for a week, so DNS only sees new addresses; range files are refetched
once a day. It covers every router, while the mails keep watching
Forgejo only.

New dashboard "AI crawlers", and a row of it on the overview.

The overview's load gauge divided node_load1 by a count with fewer
labels, which matches nothing; aggregate both sides by host.

Assisted-by: Claude:opus-5-5
This commit is contained in:
Sergei Poljanski 2026-11-13 07:21:57 +04:00
commit bc0dac9188
Signed by: asxpi
GPG key ID: 4F8851660FA4121B
7 changed files with 1924 additions and 6 deletions

View file

@ -22,6 +22,8 @@
- crawler-alert.timer
- crawler-weekly.service
- crawler-weekly.timer
- crawler-metrics.service
- crawler-metrics.timer
notify: Reload systemd
- name: Flush handlers
@ -35,3 +37,4 @@
loop:
- crawler-alert.timer
- crawler-weekly.timer
- crawler-metrics.timer