Summary

On May 25, 2026 at around 10:02 PM (local time), Sonarr and Radarr became completely inaccessible from the home IP (82.XX.XX.XX), returning 403 on every URL including /login. The service was fully operational. Initial suspicion fell on the day’s crowdsec-cf-sync refactor deployment — the real cause was a CrowdSec AppSec heuristic false positive.

Timeline

Root Cause

CrowdSec AppSec maintains an in-memory heuristic state, separate from LAPI decisions. When the browser simultaneously tries to load many resources and receives 302/403 from the upstream application (Sonarr), AppSec interprets the burst of failures as aggressive probing (http-probing) and blocks the source IP.

After years running ModSecurity + OWASP CRS on nginx, I migrated arleo.eu to a more modern stack: CrowdSec AppSec on OpenResty. The result is a tighter inline WAF architecture — better integrated, easier to maintain, and fully coherent with the rest of the security stack.

Why Drop ModSecurity?

ModSecurity v2 is in maintenance mode. Managing OWASP CRS rules on classic nginx generates friction: frequent false positives, logs that are hard to correlate with CrowdSec, and a configuration spread across multiple tools with no unified view.

⚡ TL;DR

A security stack audit on the homelab NUC reveals redundant double WAF inspection: ModSecurity + OWASP CRS load 11,872 rules into memory despite SecRuleEngine Off, running in parallel with CrowdSec AppSec which already covers the same surface. After removing the ModSecurity nginx module and five other targeted fixes, nginx drops from ~520 MB to ~27 MB PSS. Same security, memory footprint divided by 20.

🏗️ Architecture Before the Audit

The security stack had six stacked layers:

⚡ In short

The initial Vector pipeline was flooding BetterStack with ~500 events/24h, of which 434 were CAPI pulls with no local monitoring value. This work reconfigures the Vector filter to keep only high-value bans (cscli) and fixes a major blind spot: actual nginx-lua bouncer bans were not appearing anywhere in BetterStack.

🧠 Why

This homelab’s security stack relies on three components working together:

• nginx with the CrowdSec lua bouncer (lua-resty-crowdsec) for real-time request blocking
• CrowdSec for threat detection and ban decision management
• Vector centralizing logs to BetterStack for monitoring

After setting up the initial pipeline, two problems quickly became apparent. First, the signal was drowned in noise: out of 500 events/24h, 434 came from the hourly community CAPI pull and 66 from third-party lists — neither represents a threat detected on this infrastructure. Second, actual lua bouncer bans (real-time blocks in nginx) were not appearing anywhere in BetterStack, creating a blind spot on real security activity.

2. CrowdSec → Cloudflare Sync (crowdsec-cf-sync.py)

Location: /usr/local/bin/crowdsec-cf-sync.py Service: crowdsec-cf-sync.service Logs: /var/log/crowdsec/cf-sync.log

Location: /usr/local/bin/crowdsec-cf-sync.py Systemd service: crowdsec-cf-sync.service Logs: /var/log/crowdsec/cf-sync.log

Installation

cp crowdsec-cf-sync.py /usr/local/bin/
chmod +x /usr/local/bin/crowdsec-cf-sync.py
systemctl enable crowdsec-cf-sync
systemctl start crowdsec-cf-sync

Features

• Syncs active CrowdSec bans → Cloudflare IP Access Rules (tag crowdsec-local-ban)
• Reports banned IPs → AbuseIPDB (48h window, deduplicated)
• Repeat-offender escalation: 1st CrowdSec ban handles | 2nd → 24h | 3rd+ → 7d (7d window)
• ModSecurity score ≥ 5 → immediate 2h CF ban (tag modsec-ban) + AbuseIPDB report
• Automatic /24 ban: 2+ IPs from same block in 7d → CF + CrowdSec ban 24h (tag crowdsec-cidr-ban)

Bug fixes (April 2026)

• cs_origin vs origin in get_recent_local_bans() — the JSON field is cs_origin
• REST API pagination /v1/decisions?limit=1000 misses cscli bans → replaced with cscli decisions list --origin
• Items → items (lowercase) in CrowdSec allowlist JSON parsing

⚠️ Important: real tokens (CF_API_TOKEN, CF_ZONE_ID, CS_API_KEY, ABUSEIPDB_KEY) must be stored in /etc/secrets/ with chmod 600 and loaded via environment variables, not hardcoded in the script. The VOTRE_* values below are placeholders.

]]>https://www.arleo.eu/en/posts/vector-logs-harmonisation-betterstack/Fri, 10 Apr 2026 21:04:00 +0200Jmrhttps://www.arleo.eu/en/posts/vector-logs-harmonisation-betterstack/

⚡ In short

Two problems coexisted in BetterStack: mcp-oauth.access.log logs arrived as unreadable raw JSON, and CrowdSec logs produced visual duplicates. This work normalizes all logs so they display as structured clickable tags, with correct timestamps and without parasitic fields.

🧠 Why

BetterStack displays logs as highlighted clickable tags in the Live Tail when JSON fields are properly structured. Before this work, observation was degraded on two fronts:

• mcp-oauth.access.log logs arrived as unreadable raw JSON (custom format incompatible with Vector’s nginx parser) — fields nginx.client, nginx.path, nginx.status were not extracted
• CrowdSec and CF WAF logs arrived as plain text with duplicates (Ban ban | ... | Ban ban)

The goal was to normalize all logs in BetterStack to display like standard nginx logs:

]]>https://www.arleo.eu/en/posts/crowdsec-cloudflare-waf-autoban/Fri, 10 Apr 2026 00:23:00 +0200Jmrhttps://www.arleo.eu/en/posts/crowdsec-cloudflare-waf-autoban/

⚡ In short

Passive monitoring is not enough. This pipeline automates closing the loop in under 5 minutes between an attack detected by Cloudflare WAF and the effective ban of the IP in CrowdSec, its synchronization to Cloudflare, and its report to AbuseIPDB. A Python script polls the Cloudflare GraphQL API every 5 minutes, applies a 3-hit threshold, and triggers the ban with recidivist escalation.

🧠 Why

Seeing an attack in BetterStack logs after the fact does not stop the malicious IP from continuing to hammer the server. Without automation, the detection → ban loop takes hours or never closes. Cloudflare WAF actions (block, challenge, managed_challenge, jschallenge) are clear attack signals, but they remain confined to the Cloudflare dashboard — without a bridge to CrowdSec, no IP is banned locally, none is reported to the AbuseIPDB community.

]]>