Post-Mortem — OpenResty 500: CrowdSec Lua Failure

In short

Two Internet scan bursts exposed a latent defect in the OpenResty/CrowdSec request path. The scanners did not overwhelm OpenResty by volume; the request rate was modest for nginx. They triggered a code path that attempted to load a custom Lua module that fresh OpenResty workers could not resolve.

The root cause was this call inside access_by_lua:

require("crowdsec.access")

while the active lua_package_path did not include the directory containing the custom crowdsec/access.lua module.

The consequence was simple: any worker that had to load the module cold returned HTTP 500 before authentication or Hugo content handling was reached.

What triggered the incident

Two scanner sources were observed during the investigation:

• 34.75.198.72 — first wave around 08:27 UTC, followed by another probe series around 08:35;
• 34.169.50.247 — another burst around 09:14 UTC, with roughly 95 requests in 12 seconds and 94 HTTP 500 responses.

The probes targeted configuration files, secrets, and AI/MCP tooling paths such as:

/.env
/.claude.json
/.claude/settings.json
/.mcp.json
/.codex/config.toml
/.cursor/mcp.json
/config/secrets.yml
/config/master.key
/application.yml
/appsettings.json
/google-credentials.json
/service-account.json
/debug/pprof/
/metrics
/@fs/proc/self/environ?import&raw??

Part of the traffic used this User-Agent:

Mozilla/5.0 (compatible; GrokBot/1.0; +https://x.ai/grokbot)

A User-Agent can be trivially spoofed, so this is not attribution to xAI.

Condensed timeline

Exact failure signature

In /var/log/nginx/www.arleo.eu.error.log:

lua entry thread aborted: runtime error:
access_by_lua(snippets/crowdsec_access.conf:26):3:
module 'crowdsec.access' not found:
    no field package.preload['crowdsec.access']
    no file '/usr/local/openresty/nginx//../lualib/plugins/crowdsec/crowdsec/access.lua'
    ...

The relevant block was straightforward:

access_by_lua_block {
    if ngx.var.lan == "1" then return end
    require("crowdsec.access")
}

The failed require() was not caught, so the whole request failed.

The official bouncer loaded this package path:

lua_package_path '$prefix/../lualib/plugins/crowdsec/?.lua;;';

while the custom modules lived under:

/usr/local/openresty/nginx/conf/lua/crowdsec/

access.lua, mitigation.lua, sync.lua, heuristics.lua, tarpit.lua, and waf_refs.lua were present on disk. The missing piece was the search path.

Why it looked intermittent

Lua require() caches a module only after a successful load. With lua_code_cache on, a worker that had already loaded crowdsec.access could keep serving normally.

A fresh or recycled worker had to resolve the module again, used the wrong package path, and returned 500.

This explains why:

• cached content pages could appear healthy behind Cloudflare;
• /ping, configured as no-store, kept reaching the origin and exposing the problem;
• a scan burst could suddenly expose several bad workers;
• the failure looked like load exhaustion even though the traffic volume itself was not exceptional for OpenResty.

Initial false leads

Several explanations initially looked plausible:

• recently edited Hugo content;
• a broken apex arleo.eu vhost while www.arleo.eu stayed healthy;
• OpenResty/CrowdSec overload under scanning;
• broken authentication on /ping.

The Hugo MCP quickly ruled content out: the build was healthy, publication was consistent and public pages were present.

The decisive evidence came from reading the full local OpenResty error log, not only aggregated HTTP metrics: the same Lua traceback accompanied the 500 responses.

Applied fix

Immediate fix: add the missing path

The first repair was:

lua_package_path '$prefix/../lualib/plugins/crowdsec/?.lua;$prefix/lua/?.lua;;';

followed by:

sudo openresty -t
sudo openresty -s reload

After reload, fresh workers started cleanly, module 'crowdsec.access' not found stopped appearing, and the 500s disappeared.

Durable fix: do not edit the package-rewritten file

crowdsec_openresty.conf is rewritten by CrowdSec bouncer upgrades and is not protected as a dpkg conffile. Editing its lua_package_path directly would therefore make the problem return after a future update.

The file was restored to its stock state and a symlink was created instead:

/usr/local/openresty/lualib/plugins/crowdsec/crowdsec
    -> /usr/local/openresty/nginx/conf/lua/crowdsec

The official path:

.../plugins/crowdsec/?.lua

can now naturally resolve:

require("crowdsec.access")
→ .../plugins/crowdsec/crowdsec/access.lua

without modifying the package-managed configuration.

Validation after the fix

Checks performed after reload:

• openresty -t: OK;
• fresh OpenResty workers started cleanly;
• 15 sequential /ping requests: no 500s;
• 30 concurrent /ping requests: no 500s;
• no new module 'crowdsec.access' not found errors;
• no 500s in the access log after the fix;
• / and Hugo pages: 200;
• apex arleo.eu: unchanged 301 redirect to www;
• BetterStack: /ping monitor returned UP after the repair.

A 401 from /ping without credentials is expected because the endpoint uses Basic Auth. BetterStack already had the correct authentication configured; previously access_by_lua crashed before nginx reached auth_basic.

If this happens again

Do not start by rolling back Hugo.

1. Check the Lua signature immediately

sudo grep -n "crowdsec.access\|lua entry thread aborted" \
  /var/log/nginx/www.arleo.eu.error.log | tail -50

If this appears:

module 'crowdsec.access' not found

the root cause is probably the same.

2. Confirm the module is resolvable

ls -la /usr/local/openresty/lualib/plugins/crowdsec/crowdsec
ls -la /usr/local/openresty/nginx/conf/lua/crowdsec/access.lua

The first path should be a symlink to the custom module directory.

3. Inspect the effective configuration

sudo openresty -T 2>&1 | grep -n "lua_package_path\|crowdsec_access"

4. Test configuration before reload

sudo openresty -t

Important: openresty -t checks syntax but does not execute the request-phase require(). A real HTTP request is mandatory.

5. Test the real request path

curl -sk -o /dev/null -w '%{http_code}\n' https://www.arleo.eu/ping

Without credentials, a clean 401 is better than a 500. With the monitoring credentials, the expected result is 200.

6. Confirm there are no new 500s

sudo tail -f /var/log/nginx/www.arleo.eu.error.log

Then issue several concurrent requests.

Conclusion

The scanners 34.75.198.72 and 34.169.50.247 were the observed triggers, not the root cause. Around one hundred requests in a few seconds should not bring OpenResty down.

The actual failure was a custom CrowdSec Lua module that existed on disk but was missing from the OpenResty workers’ module search path. The security layer had become a global failure point: one failed require() inside access_by_lua converted any request into HTTP 500 before the rest of the nginx chain could run.

The main lesson is simple: when an outage correlates with hostile traffic, verify whether the traffic is merely exposing a latent defect in your own defensive layer.

Useful internal links

• CrowdSec AppSec + OpenResty: modern WAF without ModSecurity
• CrowdSec with Vector: filter noise, keep bans
• CrowdSec postmortem: false positive on Sonarr/Radarr
• Post-Mortem — Incident 522 / WAN Failover