diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..a274cb9 --- /dev/null +++ b/LICENSE @@ -0,0 +1,12 @@ +Copyright (C) 2026 by Sergei Poljanski + +Permission to use, copy, modify, and/or distribute this software for any +purpose with or without fee is hereby granted. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES +WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR +ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF +OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. diff --git a/README.md b/README.md index 64a749a..a5df110 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,21 @@ -# forgejo-crawlers-statistics +# crawler-alert -Crawler monitoring for the Forgejo instance at `git.asxp.io` / `git.czsk.it`. -Reads the Traefik access log, mails a threshold alert daily and a summary -weekly, and separates real AI crawlers from the ones only claiming to be. +Tells you which AI crawlers are really visiting your site, and which are +lying about it. -## Why IP verification +Reads a Traefik access log, mails a threshold alert daily and a summary +weekly. Every bot is checked against its vendor's published IP ranges, +so a forged `User-Agent` does not get counted as the real thing. -A user-agent header is self-declared and free to forge. Measured over -2.27M requests (Nov 2025 - Aug 2026), most traffic labelled as an AI -crawler did not come from the vendor it named: +Python 3.9+, standard library only. No database, no daemon, no agent. -| Bot | Claimed | Verified | Spoofed | +## Why user-agent alone is not enough + +A `User-Agent` header is a string the client chooses. Anyone can send +`ClaudeBot/1.0`. On the Forgejo instance this was written for, measured +over 2.27M requests between November 2025 and August 2026: + +| Bot | Claimed | Verified | Forged | |---|---:|---:|---:| | Claude-User | 5,000 | 0 | 4,992 | | ChatGPT-User | 2,670 | 13 | 2,657 | @@ -19,17 +24,28 @@ crawler did not come from the vendor it named: | Google-Extended | 879 | 0 | 747 | | GPTBot | 1,864 | 766 | 1,098 | -The forgeries share infrastructure: a pool of ~100 Google Cloud hosts -rotates through six vendor identities, sometimes within the same hour. -Their requests target `/.env`, `/terraform.tfstate`, -`/serviceAccountKey.json` — credential harvesting wearing a crawler -badge. Verified bots, by contrast, made zero robots.txt violations -across 2,682 requests and never fetched repository source. +About 83% of AI-labelled traffic was not from the vendor it named. -So the report counts a request as legitimate only when the source IP -falls inside the vendor's published range list. +The forgeries shared infrastructure: roughly 100 Google Cloud hosts +rotating through six vendor identities, sometimes within the same hour. +Their requests went to `/.env`, `/terraform.tfstate`, +`/serviceAccountKey.json`, `/.env.prod.bak` — credential harvesting +wearing a crawler's name. -## Verification sources +The genuine crawlers behaved completely differently: 2,682 requests, +**zero** `robots.txt` violations, no repository source fetched, ~125 MB +total. Mostly they re-read `robots.txt` and `sitemap.xml`. + +That gap is the point. Counting by user-agent tells you AI crawlers are +hammering your server. Counting by verified IP tells you they are not, +and that something else is — which is a different problem with a +different fix. + +## How verification works + +Most major vendors publish their crawler IP ranges as JSON. A request +counts as legitimate only when its source address falls inside the +range list belonging to the vendor its user-agent names. | Vendor | Endpoint | |---|---| @@ -39,54 +55,116 @@ falls inside the vendor's published range list. | Perplexity | `perplexity.ai/perplexitybot.json` | | Apple | `search.developer.apple.com/applebot.json` | -Meta, ByteDance, Amazon, Diffbot and Cohere publish no range file; -their traffic is reported but cannot be verified by IP. +Meta, ByteDance, Amazon, Diffbot, Cohere and Common Crawl publish no +range file. Their traffic is reported but cannot be verified this way; +Amazon supports reverse-DNS verification instead, which this script +does not currently implement. -Ranges are fetched at run time and cached at -`/var/cache/crawler-alert/ranges.json`. A failed fetch falls back to -the cache and the mail says so — a vendor outage degrades the report -rather than breaking it. +Ranges are fetched at run time and cached. If a vendor endpoint is +unreachable the script falls back to the cache and says so in the mail, +so an outage degrades the report instead of breaking it. -## Modes +## Requirements - crawler-alert.py # daily: mail only if yesterday > 100k requests - crawler-alert.py --weekly # Monday: 7-day summary, always mails - crawler-alert.py --dry-run # print instead of mailing - -Only `--weekly` fetches ranges; the daily threshold check stays fast. - -## Caveats - -**Cloudflare-proxied requests cannot be verified.** For traffic arriving -through Cloudflare the logged IP is the CF edge, not the client. Those -hits are counted in a separate `via CF` column rather than being called -real or fake. Enabling a real-IP header in Traefik would resolve them. - -**Perplexity's list is stale** (8 prefixes, last updated 2025-02) and -Apple's dates to 2023, so some genuine traffic from those two may fail -verification. +- Traefik writing an access log **with the User-Agent header kept** — + it is dropped by default. See `traefik-accesslog.yml`; without it + nothing here works. +- Python 3.9+ +- An SMTP account, or a Forgejo/Gitea container whose `app.ini` + `[mailer]` section the script can borrow credentials from. ## Install sudo install -m755 crawler-alert.py /usr/local/bin/ - sudo cp crawler-*.{service,timer} /etc/systemd/system/ + sudo cp crawler-alert.env.example /etc/crawler-alert.env + sudo editor /etc/crawler-alert.env # set CRAWLER_RECIPIENT + sudo cp crawler-*.service crawler-*.timer /etc/systemd/system/ sudo systemctl daemon-reload sudo systemctl enable --now crawler-alert.timer crawler-weekly.timer -SMTP credentials are read from Forgejo's own `app.ini` `[mailer]` -section via `docker exec`, so there is no second copy to maintain. -Recipient and threshold are constants at the top of the script. +Check it against your own log before enabling the timers: -## Requirements + CRAWLER_LOG='/var/log/traefik/access.log*' crawler-alert.py --weekly --dry-run -Python 3.9+ (stdlib only), read access to `/opt/traefik/logs/access.log`, -and `docker exec` on the `forgejo` container. Traefik must keep the -`User-Agent` header in its access log: +`--dry-run` prints the mail instead of sending it and needs no SMTP +config, so it is safe to run repeatedly while you tune things. - accessLog: - filePath: /var/log/traefik/access.log - fields: - headers: - defaultMode: drop - names: - User-Agent: keep +## Configuration + +All settings are environment variables, read from +`/etc/crawler-alert.env` by the systemd units. Only `CRAWLER_RECIPIENT` +has no default. + +| Variable | Default | Meaning | +|---|---|---| +| `CRAWLER_RECIPIENT` | *(required)* | Where reports are mailed | +| `CRAWLER_SITE` | `this site` | Name shown in subject/header | +| `CRAWLER_LOG` | `/var/log/traefik/access.log*` | Log glob | +| `CRAWLER_ROUTER` | *(empty)* | Traefik router to count, e.g. `forgejo@docker`. Empty counts everything | +| `CRAWLER_THRESHOLD` | `100000` | Daily alert fires above this | +| `CRAWLER_CACHE` | `/var/cache/crawler-alert/ranges.json` | Cached ranges | +| `CRAWLER_MAIL_CONTAINER` | `forgejo` | Container to read SMTP from | +| `CRAWLER_MAIL_INI` | `/data/gitea/conf/app.ini` | Path inside it | +| `CRAWLER_SMTP_*` | *(unset)* | Configure SMTP directly instead | + +Set `CRAWLER_THRESHOLD` near your normal peak. Far above it, the alert +never fires and is not a tripwire — run `--dry-run` for a few days and +pick a number a busy day would actually reach. + +## Modes + + crawler-alert.py # daily: mail only if over threshold + crawler-alert.py --weekly # 7-day summary, always mails + crawler-alert.py --dry-run # print instead of mailing + +Only `--weekly` fetches ranges; the daily check needs no network. + +## Sample output + + ============================================================== + LEGITIMATE CRAWLERS (source IP inside vendor's published ranges) + ============================================================== + bot verified spoofed via CF IPs + ChatGPT-User 0 561 0 0 + ClaudeBot 222 161 0 14 + OAI-SearchBot 23 209 0 14 + GPTBot 34 196 0 2 + PerplexityBot 0 196 0 0 + + 1909 requests claimed an AI-bot identity; 323 verified (16.9%). + 1586 failed IP verification (spoofed). 0 arrived via Cloudflare + and cannot be verified by IP - not counted either way. + + Top paths fetched by verified crawlers: + 117 /robots.txt + 73 /sitemap.xml + 13 /explore/repos + +## Limitations + +**Cloudflare-proxied requests cannot be verified.** If traffic reaches +Traefik through Cloudflare, the logged address is Cloudflare's edge and +not the client. Those hits go in a separate `via CF` column rather than +being guessed at. `traefik-accesslog.yml` shows how to recover real +client IPs with `forwardedHeaders.trustedIPs`. + +**Some vendor lists are stale.** Perplexity's has 8 prefixes and was +last updated in February 2025; Apple's dates to 2023. Genuine traffic +from those two may fail verification. A 0% verified rate for a small +vendor is weaker evidence than it looks. + +**No reverse-DNS verification.** Amazonbot and Meta's crawlers can only +be checked by rDNS or ASN, neither of which is implemented. Their +traffic appears in the user-agent tally but not the verified table. + +**IPv6 is verified only where vendors publish v6 prefixes**, which most +do; addresses outside those are treated as unverified. + +**This is a reporting tool, not an enforcement one.** It tells you what +happened. Blocking is a separate decision, and `robots.txt` only works +on the bots that already respect it — which, per the numbers above, are +exactly the ones not causing the load. + +## License + +0BSD — see LICENSE. Do what you like with it. diff --git a/crawler-alert.env.example b/crawler-alert.env.example new file mode 100644 index 0000000..244d8f8 --- /dev/null +++ b/crawler-alert.env.example @@ -0,0 +1,40 @@ +# Copy to /etc/crawler-alert.env and edit. +# Only CRAWLER_RECIPIENT is required. + +# Where the reports are mailed. +CRAWLER_RECIPIENT=you@example.com + +# Name used in mail subjects and the report header. +CRAWLER_SITE=git.example.com + +# Traefik access log. Glob so rotated/gzipped files are included. +CRAWLER_LOG=/var/log/traefik/access.log* + +# Traefik router to count, e.g. forgejo@docker. Leave empty to count +# everything that reaches Traefik (right for a single-site host). +CRAWLER_ROUTER= + +# Daily alert fires when yesterday exceeds this many requests. +# Set it near your normal peak, not far above it: a threshold that never +# fires is not a tripwire. Check a few days of real traffic first with +# crawler-alert.py --dry-run +CRAWLER_THRESHOLD=100000 + +# Cached vendor IP ranges; used when a vendor endpoint is unreachable. +CRAWLER_CACHE=/var/cache/crawler-alert/ranges.json + +# --- SMTP ------------------------------------------------------------- +# By default the script reads SMTP settings from a Forgejo/Gitea +# app.ini [mailer] section, so credentials live in one place only: +CRAWLER_MAIL_CONTAINER=forgejo +CRAWLER_MAIL_INI=/data/gitea/conf/app.ini + +# Or configure SMTP directly and ignore the container entirely. +# Setting CRAWLER_SMTP_ADDR switches to this path. +#CRAWLER_SMTP_ADDR=smtp.example.com +#CRAWLER_SMTP_PORT=587 +#CRAWLER_SMTP_USER=alerts@example.com +#CRAWLER_SMTP_PASSWORD= +#CRAWLER_SMTP_FROM=alerts@example.com +# smtp+starttls (default) | smtps | smtp (no TLS, local relay) +#CRAWLER_SMTP_PROTOCOL=smtp+starttls diff --git a/crawler-alert.py b/crawler-alert.py index 4abc9e8..2a0a359 100755 --- a/crawler-alert.py +++ b/crawler-alert.py @@ -1,23 +1,47 @@ #!/usr/bin/env python3 -"""Forgejo crawler watch on git.asxp.io. Two modes: - (default) daily threshold alert: mail only if yesterday > 100k requests - --weekly Monday summary: always mail stats for the past 7 days +"""Crawler watch for a Traefik-fronted Forgejo (or any Traefik site). + +Separates real AI crawlers from traffic that only claims to be, by +checking the source IP against each vendor's published range list. + +Modes: + (default) daily threshold alert: mail only if yesterday exceeded + CRAWLER_THRESHOLD requests + --weekly summary: always mail stats for the past 7 days --dry-run print instead of mailing (combinable with --weekly) -Set up 2026-07-15; timers: crawler-alert.timer, crawler-weekly.timer.""" + +Configuration is by environment variable; see README.md. The only +value with no sensible default is CRAWLER_RECIPIENT. +""" import configparser, glob, gzip, ipaddress, json, os, re, smtplib, subprocess, sys, urllib.request from collections import Counter from datetime import date, timedelta from email.mime.text import MIMEText -THRESHOLD = 100_000 -RECIPIENT = "me@asxp.io" +# --- configuration ------------------------------------------------------ +THRESHOLD = int(os.environ.get("CRAWLER_THRESHOLD", "100000")) +RECIPIENT = os.environ.get("CRAWLER_RECIPIENT", "") +SITE = os.environ.get("CRAWLER_SITE", "this site") +LOG_GLOB = os.environ.get("CRAWLER_LOG", "/var/log/traefik/access.log*") +# Traefik router name to filter on, e.g. "forgejo@docker". Empty = all +# traffic reaching Traefik, which is what you want for a single-site host. +ROUTER = os.environ.get("CRAWLER_ROUTER", "") +# Container to read SMTP settings from (Forgejo/Gitea app.ini [mailer]). +# Set CRAWLER_SMTP_* instead to configure SMTP directly. +MAIL_CONTAINER = os.environ.get("CRAWLER_MAIL_CONTAINER", "forgejo") +MAIL_INI = os.environ.get("CRAWLER_MAIL_INI", "/data/gitea/conf/app.ini") + WEEKLY = "--weekly" in sys.argv DRY = "--dry-run" in sys.argv +if not RECIPIENT and not DRY: + sys.exit("CRAWLER_RECIPIENT is not set (no --dry-run either); refusing " + "to run. See README.md.") + # --- verified-crawler support ------------------------------------------- # Vendors publishing machine-readable IP ranges. A UA token alone proves -# nothing: as of Aug 2026 ~77% of AI-bot-labelled traffic here came from -# hosts outside these ranges (mostly GCP). Only IP-verified hits count. +# nothing: on the instance this was written for, ~83% of AI-bot-labelled +# traffic came from hosts outside these ranges. Only IP-verified hits count. RANGE_SOURCES = { "openai": ["https://openai.com/chatgpt-user.json", "https://openai.com/gptbot.json", @@ -38,11 +62,11 @@ BOT_VENDOR = { "Applebot-Extended": "apple", "Applebot": "apple", "Googlebot": "google", "Google-Extended": "google", "GoogleOther": "google", } -CACHE = "/var/cache/crawler-alert/ranges.json" +CACHE = os.environ.get("CRAWLER_CACHE", "/var/cache/crawler-alert/ranges.json") CF_URL = "https://www.cloudflare.com/ips-v4" # claude.com and cloudflare.com sit behind Cloudflare, which 403s the # default Python-urllib UA. Send a real one. -UA = "Mozilla/5.0 (compatible; crawler-alert/1.0; +https://git.asxp.io)" +UA = "Mozilla/5.0 (compatible; crawler-alert/1.0; +https://codeberg.org/)" def fetch(url, timeout=20): @@ -118,11 +142,11 @@ spoofed = Counter() # bot -> UA claimed, IP outside vendor ranges proxied = Counter() # bot -> arrived via Cloudflare, unverifiable ver_paths = Counter() # paths fetched by verified bots ver_ips = {} # bot -> set of verified source IPs -for path in sorted(glob.glob("/opt/traefik/logs/access.log*")): +for path in sorted(glob.glob(LOG_GLOB)): opener = gzip.open if path.endswith(".gz") else open with opener(path, "rt", errors="replace") as f: for line in f: - if "forgejo@docker" not in line: + if ROUTER and ROUTER not in line: continue for s in stamps: if s in line: @@ -131,7 +155,10 @@ for path in sorted(glob.glob("/opt/traefik/logs/access.log*")): parts = line.split() if len(parts) > 6 and parts[6].count("/") >= 2: repos["/".join(parts[6].split("/")[1:3])] += 1 - ua = re.search(r"\" \"([^\"]*)\" \d+ \"forgejo", line) + # Traefik CLF: ... "" "" "" ... + # Anchor on the request counter so this works whatever + # the router is called. + ua = re.search(r"\"[^\"]*\" \"([^\"]*)\" \d+ \"", line) if ua: u = ua.group(1) m = re.search(r"(bot|crawler|spider|scrapy|externalagent|gpt|claude|perplexity)[\w./-]*", u, re.I) @@ -163,15 +190,15 @@ all_ips = set().union(*(v[1] for v in per_day.values())) if not WEEKLY: s0 = stamps[0] count, ips = per_day[s0][0], len(per_day[s0][1]) - print(f"{s0}: {count} forgejo requests, {ips} unique IPs (threshold {THRESHOLD})") + print(f"{s0}: {count} requests, {ips} unique IPs (threshold {THRESHOLD})") if DRY or count <= THRESHOLD: sys.exit(0) - subject = f"git.asxp.io: {count} requests on {s0} (>100k/day threshold)" + subject = f"{SITE}: {count} requests on {s0} (over {THRESHOLD}/day threshold)" body = (f"Forgejo served {count} requests from {ips} unique IPs on {s0} - " f"above the {THRESHOLD}/day watch threshold.\n\n" "Time to consider Anubis (PoW challenge) in front of Forgejo.\n") else: - lines = [f"git.asxp.io weekly crawler report ({stamps[0]} - {stamps[-1]})", ""] + lines = [f"{SITE} weekly crawler report ({stamps[0]} - {stamps[-1]})", ""] lines.append(f"Total: {total} requests, {len(all_ips)} unique IPs") lines.append(f"Daily threshold alert fires above {THRESHOLD} req/day (none = quiet week)") lines.append("") @@ -215,20 +242,34 @@ else: for p, c in ver_paths.most_common(8): lines.append(f" {c:>6} {p}") body = "\n".join(lines) + "\n" - subject = f"git.asxp.io weekly crawler report: {total} requests, {len(all_ips)} IPs" + subject = f"{SITE} weekly crawler report: {total} requests, {len(all_ips)} IPs" if DRY: print(subject); print(); print(body) sys.exit(0) -ini = subprocess.run(["docker","exec","forgejo","cat","/data/gitea/conf/app.ini"], - capture_output=True, text=True).stdout -cp = configparser.ConfigParser(interpolation=None, strict=False) -cp.read_string("[DEFAULT]\n" + ini) -m = cp["mailer"] -proto = m.get("PROTOCOL", "smtps").strip() -addr, port = m.get("SMTP_ADDR").strip(), int(m.get("SMTP_PORT", "465").strip()) -user, passwd = m.get("USER").strip(), m.get("PASSWD").strip().strip("`\"") -sender = m.get("FROM", user).strip() +# SMTP settings: use CRAWLER_SMTP_* if given, otherwise borrow them from +# a Forgejo/Gitea app.ini [mailer] section so there is no second copy of +# the credentials to keep in sync. +if os.environ.get("CRAWLER_SMTP_ADDR"): + addr = os.environ["CRAWLER_SMTP_ADDR"] + port = int(os.environ.get("CRAWLER_SMTP_PORT", "587")) + user = os.environ.get("CRAWLER_SMTP_USER", "") + passwd = os.environ.get("CRAWLER_SMTP_PASSWORD", "") + sender = os.environ.get("CRAWLER_SMTP_FROM", user or RECIPIENT) + proto = os.environ.get("CRAWLER_SMTP_PROTOCOL", "smtp+starttls") +else: + ini = subprocess.run(["docker", "exec", MAIL_CONTAINER, "cat", MAIL_INI], + capture_output=True, text=True).stdout + if not ini.strip(): + sys.exit(f"could not read {MAIL_INI} from container " + f"'{MAIL_CONTAINER}' and CRAWLER_SMTP_ADDR is unset") + cp = configparser.ConfigParser(interpolation=None, strict=False) + cp.read_string("[DEFAULT]\n" + ini) + m = cp["mailer"] + proto = m.get("PROTOCOL", "smtps").strip() + addr, port = m.get("SMTP_ADDR").strip(), int(m.get("SMTP_PORT", "465").strip()) + user, passwd = m.get("USER", "").strip(), m.get("PASSWD", "").strip().strip("`\"") + sender = m.get("FROM", user).strip() msg = MIMEText(body) msg["Subject"], msg["From"], msg["To"] = subject, sender, RECIPIENT @@ -236,8 +277,10 @@ if proto == "smtps": s = smtplib.SMTP_SSL(addr, port, timeout=30) else: s = smtplib.SMTP(addr, port, timeout=30) - s.starttls() -s.login(user, passwd) + if proto != "smtp": # plain smtp = no TLS (local relay) + s.starttls() +if user: # unauthenticated local relays exist + s.login(user, passwd) s.sendmail(sender, [RECIPIENT], msg.as_string()) s.quit() print(f"mailed to {RECIPIENT}") diff --git a/crawler-alert.service b/crawler-alert.service index 9407edb..6df88b7 100644 --- a/crawler-alert.service +++ b/crawler-alert.service @@ -1,7 +1,8 @@ [Unit] -Description=Mail alert when Forgejo exceeds 100k requests/day +Description=Daily crawler threshold alert After=docker.service [Service] Type=oneshot +EnvironmentFile=/etc/crawler-alert.env ExecStart=/usr/local/bin/crawler-alert.py diff --git a/crawler-weekly.service b/crawler-weekly.service index 93349bd..ed9916f 100644 --- a/crawler-weekly.service +++ b/crawler-weekly.service @@ -1,7 +1,8 @@ [Unit] -Description=Weekly Forgejo crawler report mail +Description=Weekly crawler report mail After=docker.service [Service] Type=oneshot +EnvironmentFile=/etc/crawler-alert.env ExecStart=/usr/local/bin/crawler-alert.py --weekly diff --git a/traefik-accesslog.yml b/traefik-accesslog.yml new file mode 100644 index 0000000..0a96a2b --- /dev/null +++ b/traefik-accesslog.yml @@ -0,0 +1,83 @@ +# Traefik static configuration: the access log this script needs. +# Merge into your existing traefik.yml (or docker-compose command flags). +# +# The critical part is keeping the User-Agent header. Traefik drops all +# headers by default, and without User-Agent every request is anonymous +# and no bot analysis is possible at all. + +accessLog: + filePath: /var/log/traefik/access.log + # Traefik's default. The script parses this, not JSON. + format: common + bufferingSize: 100 + fields: + # Keep the standard CLF fields. + defaultMode: keep + headers: + # Drop every header except the two named below: an access log + # holding Cookie or Authorization is a liability, not telemetry. + defaultMode: drop + names: + User-Agent: keep + Referer: keep + +# Resulting line (one line, wrapped here for readability): +# +# 1.2.3.4 - - [10/Aug/2026:22:30:24 +0000] "GET /explore HTTP/2.0" +# 200 12062 "https://git.example.com/" "Mozilla/5.0 ... ClaudeBot/1.0" +# 93971 "forgejo@docker" "http://172.18.0.3:3000" 2ms +# +# Fields, in order: +# 1 client IP <- verified against vendor ranges +# 2 ident, 3 user (always "-") +# 4 timestamp <- day bucketing +# 5 request line <- path, for the per-path breakdown +# 6 status, 7 bytes +# 8 Referer (kept above) +# 9 User-Agent <- bot token +# 10 request counter +# 11 router name <- CRAWLER_ROUTER filter, e.g. forgejo@docker +# 12 backend URL, 13 duration +# +# Mount the log directory so both Traefik and the host can see it: +# +# volumes: +# - /opt/traefik/logs:/var/log/traefik +# +# Rotate it. Traefik reopens on USR1; without rotation this file grows +# without bound (it reached 533 MB in nine months on the instance this +# was written for). /etc/logrotate.d/traefik: +# +# /opt/traefik/logs/*.log { +# daily +# rotate 14 +# compress +# delaycompress +# missingok +# notifempty +# postrotate +# docker kill --signal=USR1 traefik 2>/dev/null || true +# endscript +# } +# +# The script reads rotated and gzipped files too, as long as +# CRAWLER_LOG keeps its trailing glob (access.log*). + +# --- If you sit behind Cloudflare ------------------------------------- +# The logged client IP will be Cloudflare's edge, not the real visitor, +# and no such request can be IP-verified (the report counts these in a +# separate "via CF" column). To recover real IPs, trust CF's ranges on +# the entrypoint: +# +# entryPoints: +# websecure: +# address: ":443" +# forwardedHeaders: +# trustedIPs: +# # https://www.cloudflare.com/ips-v4 - keep this list current +# - 173.245.48.0/20 +# - 103.21.244.0/22 +# # ... remaining CF ranges ... +# +# Only list address ranges you actually trust: anyone in trustedIPs can +# spoof X-Forwarded-For and forge their apparent source address.