crawler_alert, monitoring: AI crawler metrics and dashboards
Some checks failed
deploy / ansible (push) Failing after 1m17s
Some checks failed
deploy / ansible (push) Failing after 1m17s
crawler-alert.py --metrics (5-minute timer) reads what the Traefik access log gained since the last run, judges every request that names an AI bot exactly like the weekly report (vendor IP ranges, forward- confirmed rDNS, Meta's ASN) and writes cumulative counters to the node_exporter textfile directory. Verdicts are cached per bot and IP for a week, so DNS only sees new addresses; range files are refetched once a day. It covers every router, while the mails keep watching Forgejo only. New dashboard "AI crawlers", and a row of it on the overview. The overview's load gauge divided node_load1 by a count with fewer labels, which matches nothing; aggregate both sides by host. Assisted-by: Claude:opus-5-5
This commit is contained in:
parent
3b25f4ae25
commit
bc0dac9188
7 changed files with 1924 additions and 6 deletions
12
ansible/roles/crawler_alert/files/crawler-metrics.service
Normal file
12
ansible/roles/crawler_alert/files/crawler-metrics.service
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
[Unit]
|
||||
Description=Crawler counters for Prometheus (node_exporter textfile)
|
||||
After=docker.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
EnvironmentFile=/etc/crawler-alert.env
|
||||
# Every router, not only Forgejo's; fewer DNS lookups per 5-minute run
|
||||
Environment=CRAWLER_METRICS_ROUTER= CRAWLER_RDNS_MAX=100
|
||||
ExecStart=/usr/local/bin/crawler-alert.py --metrics
|
||||
Nice=10
|
||||
TimeoutStartSec=4min
|
||||
Loading…
Add table
Add a link
Reference in a new issue