Postmortem — CrowdSec AppSec: Heuristic False Positive on Sonarr/Radarr

2026-05-25 547 words 3 minutes

/images/postmortem-crowdsec-appsec-false-positive-sonarr-featured.jpg

Contents

Summary

On May 25, 2026 at around 10:02 PM (local time), Sonarr and Radarr became completely inaccessible from the home IP (82.XX.XX.XX), returning 403 on every URL including /login. The service was fully operational. Initial suspicion fell on the day’s crowdsec-cf-sync refactor deployment — the real cause was a CrowdSec AppSec heuristic false positive.

Timeline

Time (local)	Event
~10:00 PM	Sonarr browser session cookie expired
10:02:31 PM	Browser loads Sonarr library → attempts to fetch 20+ `/MediaCover/*.jpg` simultaneously
10:02:31 PM	Sonarr returns 302 → `/login` for each image (invalid session)
10:02:34 PM	SignalR WebSocket connection succeeds (101) via `access_token` in URL
~10:05 PM	CrowdSec AppSec triggers heuristic rule `http-probing`: burst of failed requests from same IP
10:11:37 PM	All requests from `82.XX.XX.XX` return 403 — `cs_reason=heuristic` in nginx logs
10:13:34 PM	Even `/login` is blocked — IP cannot authenticate

Root Cause

CrowdSec AppSec maintains an in-memory heuristic state, separate from LAPI decisions. When the browser simultaneously tries to load many resources and receives 302/403 from the upstream application (Sonarr), AppSec interprets the burst of failures as aggressive probing (http-probing) and blocks the source IP.

The issue: the home IP is in the LAPI allowlist (my_allowlist) and in the nginx geo $lan block, but AppSec heuristics do not consult the LAPI allowlist — they operate independently on behavioral signals.

cscli decisions list --ip 82.XX.XX.XX
→ No active decisions   ← LAPI allowlist is respected

nginx access log:
82.XX.XX.XX ... "GET /" 403 ... cs_reason=heuristic   ← AppSec in-memory block

What Was NOT the Cause

Multiple leads were ruled out in order:

Lead	Result
`crowdsec-cf-sync` refactor (deployed today)	❌ Does not touch nginx, AppSec, or Sonarr
CrowdSec LAPI ban (`cscli decisions list`)	❌ No active decisions for the IP
`bans.json` Lua IPC	❌ Only 3 third-party IPs, not the home IP
nginx `geo $lan` block	❌ IP is present in the block (`$lan = 1`)
Invalid Sonarr API key	❌ `/api/v3/system/status` returns 200 with the key
Network / Sonarr VM issue	❌ VM responds (302 expected for expired session)

Impact

Duration: ~15–20 minutes (manual detection)
Scope: Sonarr and Radarr on vm-media only
Users: 1 (home IP)
Data loss: none

Immediate Fix

Restart CrowdSec to flush the in-memory heuristic state:

sudo systemctl restart crowdsec

Permanent Fix

Added a Lua bypass in /etc/openresty/snippets/crowdsec_access.conf to exempt LAN IPs from any AppSec check before LAPI verification:

access_by_lua_block {
    if ngx.var.lan == "1" then return end
    require("crowdsec.access").check()
}

The $lan variable is defined by the geo block in conf.d/lan.conf and covers RFC 1918 ranges (192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12) plus the fixed home IP.

External IPs continue to go through the full check (LAPI + AppSec).

Lessons Learned

The LAPI allowlist does not protect against AppSec heuristics. Both systems are independent — an IP can be in my_allowlist and still be blocked in AppSec’s memory.
Legitimate request bursts trigger heuristic rules. Sonarr and Radarr load dozens of cover images on first view — a pattern indistinguishable from an asset scanner.
cs_nginx_ban: false does not mean “not blocked”. That field reflects only the LAPI bouncer. An AppSec heuristic block returns 403 with cs_reason=heuristic in the nginx log — a field absent from the Vector/BetterStack pipeline.
BetterStack diagnostic has limits. The nginx_status source does not include cs_reason — the real cause was only visible in the raw nginx logs (/var/log/nginx/vm-media.access.log).