crawler_alert, monitoring: AI crawler metrics and dashboards
Some checks failed
deploy / ansible (push) Failing after 1m17s
Some checks failed
deploy / ansible (push) Failing after 1m17s
crawler-alert.py --metrics (5-minute timer) reads what the Traefik access log gained since the last run, judges every request that names an AI bot exactly like the weekly report (vendor IP ranges, forward- confirmed rDNS, Meta's ASN) and writes cumulative counters to the node_exporter textfile directory. Verdicts are cached per bot and IP for a week, so DNS only sees new addresses; range files are refetched once a day. It covers every router, while the mails keep watching Forgejo only. New dashboard "AI crawlers", and a row of it on the overview. The overview's load gauge divided node_load1 by a count with fewer labels, which matches nothing; aggregate both sides by host. Assisted-by: Claude:opus-5-5
This commit is contained in:
parent
3b25f4ae25
commit
bc0dac9188
7 changed files with 1924 additions and 6 deletions
|
|
@ -6,3 +6,5 @@ CRAWLER_THRESHOLD={{ crawler_threshold }}
|
|||
CRAWLER_MAIL_CONTAINER=forgejo
|
||||
CRAWLER_MAIL_INI=/data/gitea/conf/app.ini
|
||||
CRAWLER_ROBOTS={{ crawler_robots_url }}
|
||||
CRAWLER_LIVE_LOG=/opt/traefik/logs/access.log
|
||||
CRAWLER_TEXTFILE={{ monitoring_textfile_dir }}/crawler.prom
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue