crawler_alert, monitoring: AI crawler metrics and dashboards

Browse source

crawler-alert.py --metrics (5-minute timer) reads what the Traefik
access log gained since the last run, judges every request that names
an AI bot exactly like the weekly report (vendor IP ranges, forward-
confirmed rDNS, Meta's ASN) and writes cumulative counters to the
node_exporter textfile directory. Verdicts are cached per bot and IP
for a week, so DNS only sees new addresses; range files are refetched
once a day. It covers every router, while the mails keep watching
Forgejo only.

New dashboard "AI crawlers", and a row of it on the overview.

The overview's load gauge divided node_load1 by a count with fewer
labels, which matches nothing; aggregate both sides by host.

Assisted-by: Claude:opus-5-5

...

This commit is contained in:

Sergei Poljanski 2026-11-13 07:21:57 +04:00

parent 3b25f4ae25

commit bc0dac9188

Signed by: asxpi

GPG key ID: 4F8851660FA4121B

7 changed files with 1924 additions and 6 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download patch file Download diff file Expand all files Collapse all files

1148

ansible/roles/monitoring/files/dashboards/Services/crawlers.json Normal file

View file

File diff suppressed because it is too large Load diff