Infrastructure Scripts

This page lists the scripts and tools running arleo.eu’s infrastructure — what’s actually in use today first, the Python history after. Full architecture/systemd detail for the CrowdSec ↔ Cloudflare stack is in Infrastructure Documentation; this page focuses on concrete, downloadable examples. No real token appears anywhere on this page — placeholders only, secrets always kept out of the repo.


1. Currently in use

security-automation-go — CrowdSec ↔ Cloudflare orchestration

Repository: github.com/jmrGrav/security-automation-go (Go, Apache-2.0, active)

Replaces every Python script in section 2 below. Full detail (architecture, systemd services, verification commands) in Infrastructure Documentation §2 — in short:

CommandWhat it does
cf-syncMain daemon: syncs CrowdSec bans → Cloudflare, reports to AbuseIPDB (ban gated on local corroboration, not reputation score alone), owns the full ban lifecycle (auto-expiry), and serves the web operator console
cf-allowlist-syncSyncs the CrowdSec allowlist with BetterStack IPs + Cloudflare ranges (every 15 min)
cf-cleanupCleans up stale Cloudflare IP access rules

Installation, .deb releases, and the setup wizard (encrypted credentials, no plaintext token on disk): see the project README.

csp-monitor.sh — CSP monitoring and auto-repair

Cron: */15 * * * * /home/jm/scripts/csp-monitor.sh · Exit code: 0 OK / 1 WARN / 2 ALERT

Detects a missing or incomplete CSP header, a spike in violations over 15 minutes, a never-seen violation pattern, a regression (inline script produced by Hugo — should always be 0), and checks the Mozilla Observatory grade once a day. Self-heals in two cases: automatic rollback if openresty -t fails (to the last .bak-*), and a Cloudflare cache purge if an expected CSP directive is missing from the observed response (usually a stale CF cache, not a real config problem). Alerts and heartbeat go through BetterStack; Cloudflare tokens are decrypted on the fly via age — never stored in plaintext.

#!/bin/bash
# csp-monitor.sh — Detection AND automatic repair of CSP on www.arleo.eu
# Cron: */15 * * * * /home/jm/scripts/csp-monitor.sh >> /var/log/csp-monitor.log 2>&1
#
# Detects:
#   1. Missing CSP header or missing critical directives (default-src 'none' / frame-ancestors)
#   2. Burst of violations in the last 15 minutes
#   3. A new violation pattern (never seen before)
#   4. Regression: inline scripts from the origin (Hugo should produce 0)
#   5. Mozilla Observatory grade < A+ (once a day)
#
# Auto-repairs:
#   A. If openresty -t fails → rollback to the last .bak-*
#   B. If a CSP directive is missing → purge CF cache
#   C. (Info only) If CF injects a script — requires manual dashboard action
#
# Exit code: 0 = OK, 1 = WARN, 2 = ALERT
# Alerts via BETTERSTACK_WEBHOOK env var if set.
# Heartbeat via BETTERSTACK_HEARTBEAT env var if set.

set -euo pipefail

LOG_FILE="/var/log/nginx/csp-violations.log"
STATE_DIR="/var/lib/csp-monitor"
KNOWN_PATTERNS_FILE="${STATE_DIR}/known-patterns.txt"
AGE_KEY_FILE="${STATE_DIR}/age-key.txt"
CF_TOKEN_AGE="${STATE_DIR}/cf-token.age"
CF_ZONE_AGE="${STATE_DIR}/cf-zone.age"
WEBHOOK_URL="${BETTERSTACK_WEBHOOK:-}"
HEARTBEAT_URL="${BETTERSTACK_HEARTBEAT:-}"
SITE="https://www.arleo.eu"
LOCK_FILE="/run/lock/csp-monitor.lock"

REQUIRED_DIRECTIVES=(
    "default-src 'none'"
    "frame-ancestors 'self'"
    "object-src 'none'"
    "base-uri 'self'"
    "form-action 'self'"
    "upgrade-insecure-requests"
    "worker-src 'self' https://cdn.jsdelivr.net"
)

BURST_THRESHOLD=20
WARN_THRESHOLD=5
ALERT_LEVEL=0
ALERTS=()

log()   { printf '[%s] %s\n' "$(date -Iseconds)" "$*"; }
alert() { ALERT_LEVEL=2; ALERTS+=("ALERT: $*"); log "ALERT: $*"; }
warn()  { (( ALERT_LEVEL < 1 )) && ALERT_LEVEL=1; ALERTS+=("WARN: $*"); log "WARN: $*"; }
info()  { log "INFO: $*"; }

exec 9>"$LOCK_FILE"
flock -n 9 || { log "Another instance is running — skipping"; exit 0; }

mkdir -p "$STATE_DIR"
chmod 700 "$STATE_DIR"
touch "$KNOWN_PATTERNS_FILE"
chmod 600 "$KNOWN_PATTERNS_FILE"

OPENRESTY_BIN="/usr/bin/openresty"

if sudo "$OPENRESTY_BIN" -t >/dev/null 2>&1; then
    sudo /usr/local/bin/csp-backup.sh 2>/dev/null || true
fi

if ! sudo "$OPENRESTY_BIN" -t >/dev/null 2>&1; then
    alert "openresty -t failed — invalid CSP config, automatic rollback in progress"
    restored=$(sudo /usr/local/bin/csp-restore.sh 2>/dev/null || true)
    if [[ -n "$restored" ]]; then
        if sudo "$OPENRESTY_BIN" -t >/dev/null 2>&1; then
            sudo /bin/systemctl reload openresty
            info "Rollback succeeded to $restored"
        else
            alert "Rollback FAILED — vhost still invalid, manual intervention required"
        fi
    else
        alert "No backup available — manual intervention required"
    fi
fi

csp_header=$(curl -sk -I "$SITE/" 2>/dev/null | grep -i '^content-security-policy:' | tr -d '\r' || true)
if [[ -z "$csp_header" ]]; then
    alert "No Content-Security-Policy header served by $SITE"
else
    csp_ok=1
    for directive in "${REQUIRED_DIRECTIVES[@]}"; do
        if ! grep -qF "$directive" <<<"$csp_header"; then
            warn "Missing CSP directive: $directive"
            csp_ok=0
        fi
    done

    if (( csp_ok == 0 )); then
        warn "CSP diverges from expected config — attempting CF cache purge"
        if [[ -r "$CF_TOKEN_AGE" && -r "$CF_ZONE_AGE" && -r "$AGE_KEY_FILE" ]]; then
            cf_token=$(age -d -i "$AGE_KEY_FILE" "$CF_TOKEN_AGE" 2>/dev/null || true)
            cf_zone=$(age -d -i "$AGE_KEY_FILE" "$CF_ZONE_AGE" 2>/dev/null || true)
            if [[ -n "$cf_token" && -n "$cf_zone" ]]; then
                curl -s -X POST "https://api.cloudflare.com/client/v4/zones/${cf_zone}/purge_cache" \
                    -H "Authorization: Bearer ${cf_token}" -H "Content-Type: application/json" \
                    --data '{"purge_everything":true}' >/dev/null
                info "CF purge triggered — will re-check on next run"
            fi
        else
            warn "Missing CF credentials — purge skipped"
        fi
    fi
fi

since=$(date -d '15 minutes ago' -Iseconds)
recent_count=$(awk -v since="$since" '
    match($0, /"timestamp":"[^"]+"/) {
        ts = substr($0, RSTART+13, RLENGTH-14)
        if (ts >= since) print
    }
' "$LOG_FILE" 2>/dev/null | wc -l || echo 0)

if (( recent_count > BURST_THRESHOLD )); then
    alert "CSP violation burst: $recent_count in 15 min (threshold $BURST_THRESHOLD)"
elif (( recent_count > WARN_THRESHOLD )); then
    warn "High CSP violation count: $recent_count in 15 min (threshold $WARN_THRESHOLD)"
fi

recent_patterns=$(awk -v since="$since" '
    match($0, /"timestamp":"[^"]+"/) {
        ts = substr($0, RSTART+13, RLENGTH-14)
        if (ts < since) next
        d = ""; b = ""
        if (match($0, /"directive":"[^"]*"/)) d = substr($0, RSTART+13, RLENGTH-14)
        if (match($0, /"blocked_uri":"[^"]*"/)) b = substr($0, RSTART+15, RLENGTH-16)
        if (d != "" || b != "") print d "|" b
    }
' "$LOG_FILE" 2>/dev/null | sort -u || true)

new_patterns=""
while IFS= read -r pattern; do
    [[ -z "$pattern" ]] && continue
    if ! grep -qxF "$pattern" "$KNOWN_PATTERNS_FILE"; then
        new_patterns+="$pattern"$'\n'
        echo "$pattern" >> "$KNOWN_PATTERNS_FILE"
    fi
done <<<"$recent_patterns"

if [[ -n "$new_patterns" ]]; then
    warn "New CSP pattern(s):"$'\n'"$new_patterns"
fi

pattern_count=$(wc -l < "$KNOWN_PATTERNS_FILE" || echo 0)
if (( pattern_count > 1000 )); then
    tail -1000 "$KNOWN_PATTERNS_FILE" > "${KNOWN_PATTERNS_FILE}.tmp"
    mv "${KNOWN_PATTERNS_FILE}.tmp" "$KNOWN_PATTERNS_FILE"
fi

inline_count=$(curl -sk --resolve www.arleo.eu:443:127.0.0.1 "$SITE/" 2>/dev/null \
    | python3 -c "
import sys, re
html = sys.stdin.read()
pattern = re.compile(
    r'<script'
    r'(?![^>]*\bsrc\s*=)'
    r'(?![^>]*\btype\s*=\s*[\"\'']?(?:application/ld\+json|application/json|module)[\"\'']?)'
    r'[^>]*>',
    re.IGNORECASE
)
print(len(pattern.findall(html)))
" 2>/dev/null || echo "0")

if (( inline_count > 0 )); then
    warn "Origin Hugo produces $inline_count inline script(s) — should be 0"
fi

cf_inline=$(curl -sk "$SITE/" 2>/dev/null | grep -c '__CF\$cv\|challenge-platform' || true)
if (( cf_inline > 0 )); then
    info "CF injects $cf_inline script(s) at the edge — disable: CF Dashboard → Security → Bots → JS Detections → Off"
fi

MDN_CACHE="${STATE_DIR}/mdn-score.txt"
if [[ ! -f "$MDN_CACHE" ]] || find "$MDN_CACHE" -mtime +1 -print 2>/dev/null | grep -q .; then
    grade=$(curl -s -X POST "https://observatory-api.mdn.mozilla.net/api/v2/scan?host=www.arleo.eu" 2>/dev/null \
        | python3 -c "import sys,json
try: print(json.load(sys.stdin).get('grade','?'))
except: print('?')" 2>/dev/null || echo "?")
    echo "$grade" > "$MDN_CACHE"
    if [[ "$grade" != "A+" && "$grade" != "?" ]]; then
        warn "MDN Observatory degraded: $grade (expected A+)"
    else
        info "MDN Observatory grade = $grade"
    fi
fi

if (( ALERT_LEVEL > 0 )) && [[ -n "$WEBHOOK_URL" ]]; then
    payload=$(printf '%s\n' "${ALERTS[@]}" | python3 -c "import sys,json; print(json.dumps({'text': sys.stdin.read()}))")
    curl -s -X POST "$WEBHOOK_URL" -H 'Content-Type: application/json' --data "$payload" >/dev/null || true
fi

if [[ -n "$HEARTBEAT_URL" ]]; then
    curl -s "$HEARTBEAT_URL" >/dev/null || true
fi

exit "$ALERT_LEVEL"

2. Legacy — former Python scripts (retired)

All replaced in production by security-automation-go (section 1). Python service stopped since June 9, 2026 — kept below purely as historical reference, not to be reinstalled.

crowdsec-cf-sync.py

Synced active CrowdSec bans to Cloudflare, reported to AbuseIPDB, handled repeat-offender escalation, and an immediate 2h ban on ModSecurity score ≥ 5 (the local WAF has itself since migrated to CrowdSec AppSec — ModSecurity is no longer the WAF in place).

⚠️ The YOUR_* values below are placeholders, never real tokens. Condensed for readability — the full script was ~700 lines.

#!/usr/bin/env python3
"""
CrowdSec → Cloudflare IP Sync + AbuseIPDB Reporter + Recidivist Escalation
+ ModSecurity → immediate CF Ban (2h) + automatic /24 Ban
"""

import ipaddress
import json
import logging
import subprocess
import time
import urllib.request
import urllib.error
from pathlib import Path

CF_API_TOKEN    = "YOUR_CF_API_TOKEN"
CF_ZONE_ID      = "YOUR_CF_ZONE_ID"
CS_API_KEY      = "YOUR_CS_API_KEY"
ABUSEIPDB_KEY   = "YOUR_ABUSEIPDB_KEY"
ABUSEIPDB_URL   = "https://api.abuseipdb.com/api/v2/report"
INTERVAL        = 60
NOTE_TAG        = "crowdsec-local-ban"
NOTE_TAG_MODSEC = "modsec-ban"
NOTE_TAG_CIDR   = "crowdsec-cidr-ban"
LOCAL_ORIGINS   = {"crowdsec", "cscli"}
CF_LOG_FILE     = Path("/var/log/crowdsec/cf-sync.log")
LOOKBACK_HOURS  = 48
RECIDIV_WINDOW  = 7
CIDR_WINDOW     = 7
MODSEC_SCORE_MIN    = 5
MODSEC_BAN_SECS     = 7200
CIDR_BAN_DURATION   = "24h"
CIDR_THRESHOLD      = 2

RECIDIV_ESCALATION = {0: None, 1: "24h"}
RECIDIV_DEFAULT = "168h"

logging.basicConfig(level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s",
    handlers=[logging.FileHandler(CF_LOG_FILE), logging.StreamHandler()])
log = logging.getLogger(__name__)


def get_crowdsec_allowlist() -> set:
    try:
        result = subprocess.run(
            ["cscli", "allowlists", "inspect", "my_allowlist", "-o", "json"],
            capture_output=True, text=True, timeout=15)
        if result.returncode != 0:
            return set()
        data = json.loads(result.stdout)
        items = data.get("items", []) or []
        return {item.get("value", "") for item in items if item.get("value")}
    except Exception as e:
        log.warning("Error reading CrowdSec allowlist: %s", e)
        return set()


def cf_request(method: str, path: str, data=None) -> dict:
    url = f"https://api.cloudflare.com/client/v4{path}"
    body = json.dumps(data).encode() if data is not None else None
    req = urllib.request.Request(url, data=body, method=method,
        headers={"Authorization": f"Bearer {CF_API_TOKEN}", "Content-Type": "application/json"})
    try:
        with urllib.request.urlopen(req, timeout=15) as resp:
            result = json.loads(resp.read().decode())
            if not result.get("success"):
                raise RuntimeError(f"CF API error: {result.get('errors')}")
            return result
    except urllib.error.HTTPError as e:
        raise RuntimeError(f"HTTP {e.code} on {method} {path}: {e.read().decode()}") from e


def get_active_bans() -> set:
    """Active local bans (crowdsec + cscli) via cscli decisions list."""
    bans = set()
    for origin in LOCAL_ORIGINS:
        try:
            result = subprocess.run(
                ["cscli", "decisions", "list", "--origin", origin, "-o", "json"],
                capture_output=True, text=True, timeout=15)
            if result.returncode != 0 or not result.stdout.strip():
                continue
            alerts = json.loads(result.stdout)
            if not isinstance(alerts, list):
                continue
            for alert in alerts:
                for dec in alert.get("decisions") or []:
                    if (dec.get("type") == "ban" and dec.get("scope", "").lower() == "ip"
                            and dec.get("value")):
                        bans.add(dec["value"])
        except Exception as e:
            log.warning("Error running cscli decisions list --origin %s: %s", origin, e)
    return bans


def main():
    log.info("=== CrowdSec CF Sync started (interval=%ds) ===", INTERVAL)
    cs_allowlist = get_crowdsec_allowlist()
    log.info("CrowdSec allowlist: %d entries", len(cs_allowlist))
    while True:
        try:
            active_bans = get_active_bans()
            log.info("CrowdSec: %d active bans", len(active_bans))
            # ... (full logic: CF sync, AbuseIPDB, recidivists, ModSec, CIDR)
        except Exception as e:
            log.error("Sync error: %s", e, exc_info=True)
        time.sleep(INTERVAL)


if __name__ == "__main__":
    main()

cloudflare-allowlist-update.py

Hourly cron — synced Cloudflare’s allowed_ip list with BetterStack + Cloudflare IPv4/IPv6 ranges, then the CrowdSec allowlist. Replaced by cf-allowlist-sync (section 1), now running every 15 min instead of once an hour.

SOURCES = {
    "betterstack": "https://uptime.betterstack.com/ips.txt",
    "cloudflare_v4": "https://www.cloudflare.com/ips-v4/",
    "cloudflare_v6": "https://www.cloudflare.com/ips-v6/",
}

def main():
    existing_cf = get_existing_cf_ips(account_id, list_id)
    for source_name, url in SOURCES.items():
        fetched = fetch_ips(url, source_name)
        new_ips = fetched - existing_cf
        if new_ips:
            add_cf_ips(account_id, list_id, new_ips, source_name)

    existing_cs = get_crowdsec_allowlist()
    new_cs_ips = all_cf_ips - existing_cs
    if new_cs_ips:
        add_crowdsec_ips(new_cs_ips, "sync-cloudflare-allowlist")

cloudflare-cleanup-ip-rules.py

One-off use — removed ~49,974 stale Cloudflare IP access rules (Fail2Ban/UFW/ModSecurity) accumulated before March 2026, keeping anything tagged easycron in its note. Current equivalent: cf-cleanup (section 1), available on an ongoing basis rather than one-off.


This script came from the pre-migration Grav theme (quark). The path /var/www/grav/user/themes/quark/js/cookie-banner.js no longer exists since the Hugo cutover — the current theme (LoveIt) doesn’t carry it over, and the site today serves no cookie-consent banner (no non-essential cookie is set). Kept below only as a reusable code example — not as a description of what runs on arleo.eu today.

(function () {
    'use strict';

    var COOKIE_NAME = 'arleo_cookie_consent';
    var COOKIE_DURATION = 365; // days

    function getCookie(name) {
        var match = document.cookie.match(new RegExp('(^| )' + name + '=([^;]+)'));
        return match ? match[2] : null;
    }

    function setCookie(name, value, days) {
        var expires = new Date();
        expires.setTime(expires.getTime() + days * 24 * 60 * 60 * 1000);
        document.cookie = name + '=' + value
            + '; expires=' + expires.toUTCString()
            + '; path=/'
            + '; SameSite=Strict'
            + '; Secure';
    }

    function createBanner() {
        var banner = document.createElement('div');
        banner.id = 'arleo-cookie-banner';
        banner.setAttribute('role', 'dialog');
        banner.setAttribute('aria-live', 'polite');
        banner.setAttribute('aria-label', 'Cookie consent');
        banner.innerHTML =
            '<div class="arleo-cookie-inner">' +
                '<p class="arleo-cookie-text">' +
                    'This site uses session cookies necessary for it to function. ' +
                    '<a href="https://www.arleo.eu/privacy-policies" rel="noopener noreferrer">Learn more</a>' +
                '</p>' +
                '<button id="arleo-cookie-accept" aria-label="Accept cookies">OK</button>' +
            '</div>';

        document.body.appendChild(banner);

        document.getElementById('arleo-cookie-accept').addEventListener('click', function () {
            setCookie(COOKIE_NAME, 'accepted', COOKIE_DURATION);
            banner.style.transition = 'opacity 0.3s';
            banner.style.opacity = '0';
            setTimeout(function () {
                if (banner.parentNode) {
                    banner.parentNode.removeChild(banner);
                }
            }, 300);
        });
    }

    function init() {
        if (getCookie(COOKIE_NAME) === 'accepted') {
            return;
        }
        if (document.readyState === 'loading') {
            document.addEventListener('DOMContentLoaded', createBanner);
        } else {
            createBanner();
        }
    }

    init();
})();

4. Maintenance commands (current)

# Status and logs
systemctl status cf-sync
systemctl status cf-allowlist-sync.timer
journalctl -fu cf-sync

# Operator console (loopback only)
curl -s http://127.0.0.1:9091/  # from the NUC itself

# Check active CrowdSec bans
cscli decisions list --origin cscli
cscli decisions list --origin crowdsec

# Built-in diagnostics
cf-sync -mode doctor

5. Security — secret storage

security-automation-go encrypts credentials at rest (SQLite, AES-GCM) via its first-run setup wizard — no more plaintext .env file to manage for that component. csp-monitor.sh (section 1) decrypts its Cloudflare tokens on the fly via age, never storing them in plaintext.

⚠️ No real token appears on this page. Any YOUR_* value in the legacy scripts above is a placeholder.