---
title: "Post-Mortem — OpenResty 500: CrowdSec Lua Failure"
description: "Why Internet scans triggered HTTP 500s on arleo.eu: missing CrowdSec Lua module, root cause, durable fix, and recovery runbook."
url: "https://www.arleo.eu/en/posts/post-mortem-openresty-crowdsec-lua-500/"
language: "en"
datePublished: "2026-08-17T11:34:51Z"
dateModified: "2026-08-17T11:34:51Z"
tags: ["openresty","crowdsec","incident","security","homelab"]
categories: ["incidents"]
contentSignal: "ai-train=no, search=yes, ai-input=yes"
---

# Post-Mortem — OpenResty 500: CrowdSec Lua Failure
Why Internet scans triggered HTTP 500s on arleo.eu: missing CrowdSec Lua module, root cause, durable fix, and recovery runbook.



{{< admonition type=info title="What happened" open=false >}}
- **Date:** August 16–17, 2026
- **Severity:** P1
- **Status:** Resolved
- **Impact:** HTTP 500 responses at the origin, including legitimate traffic and the `/ping` monitoring endpoint
{{< /admonition >}}

## In short

Two Internet scan bursts exposed a latent defect in the OpenResty/CrowdSec request path. The scanners did **not** overwhelm OpenResty by volume; the request rate was modest for nginx. They triggered a code path that attempted to load a custom Lua module that fresh OpenResty workers could not resolve.

The root cause was this call inside `access_by_lua`:

```lua
require("crowdsec.access")
```

while the active `lua_package_path` did not include the directory containing the custom `crowdsec/access.lua` module.

The consequence was simple: **any worker that had to load the module cold returned HTTP 500 before authentication or Hugo content handling was reached**.

## What triggered the incident

Two scanner sources were observed during the investigation:

- `34.75.198.72` — first wave around 08:27 UTC, followed by another probe series around 08:35;
- `34.169.50.247` — another burst around 09:14 UTC, with roughly 95 requests in 12 seconds and 94 HTTP 500 responses.

The probes targeted configuration files, secrets, and AI/MCP tooling paths such as:

```text
/.env
/.claude.json
/.claude/settings.json
/.mcp.json
/.codex/config.toml
/.cursor/mcp.json
/config/secrets.yml
/config/master.key
/application.yml
/appsettings.json
/google-credentials.json
/service-account.json
/debug/pprof/
/metrics
/@fs/proc/self/environ?import&raw??
```

Part of the traffic used this User-Agent:

```text
Mozilla/5.0 (compatible; GrokBot/1.0; +https://x.ai/grokbot)
```

A User-Agent can be trivially spoofed, so **this is not attribution to xAI**.

## Condensed timeline

| Time (UTC) | Event |
|---|---|
| ~08:27 | First burst from `34.75.198.72`: AI/MCP configs, secrets and sensitive files |
| ~08:28 | Malicious and legitimate requests begin returning HTTP 500 |
| ~08:35 | Second PHP/webshell probe wave from the same source |
| 09:08 | CrowdSec finally bans `34.75.198.72` via `crowdsecurity/http-crawl-non_statics` |
| 09:14:30–09:14:42 | `34.169.50.247` sends ~95 requests; 94 return 500 |
| 09:16 | BetterStack `/ping` probes also receive 500 from multiple regions |
| Aug 17 investigation | Local OpenResty logs reveal `module 'crowdsec.access' not found` |
| 13:20 local | Initial `lua_package_path` fix, reload, 500s disappear |
| 13:23 local | Durable fix switched to a symlink outside the package-rewritten file |
| 11:31 UTC | BetterStack confirms `/ping` is UP again |

## Exact failure signature

In `/var/log/nginx/www.arleo.eu.error.log`:

```text
lua entry thread aborted: runtime error:
access_by_lua(snippets/crowdsec_access.conf:26):3:
module 'crowdsec.access' not found:
    no field package.preload['crowdsec.access']
    no file '/usr/local/openresty/nginx//../lualib/plugins/crowdsec/crowdsec/access.lua'
    ...
```

The relevant block was straightforward:

```lua
access_by_lua_block {
    if ngx.var.lan == "1" then return end
    require("crowdsec.access")
}
```

The failed `require()` was not caught, so the whole request failed.

The official bouncer loaded this package path:

```nginx
lua_package_path '$prefix/../lualib/plugins/crowdsec/?.lua;;';
```

while the custom modules lived under:

```text
/usr/local/openresty/nginx/conf/lua/crowdsec/
```

`access.lua`, `mitigation.lua`, `sync.lua`, `heuristics.lua`, `tarpit.lua`, and `waf_refs.lua` were present on disk. **The missing piece was the search path.**

## Why it looked intermittent

Lua `require()` caches a module only after a successful load. With `lua_code_cache on`, a worker that had already loaded `crowdsec.access` could keep serving normally.

A fresh or recycled worker had to resolve the module again, used the wrong package path, and returned 500.

This explains why:

- cached content pages could appear healthy behind Cloudflare;
- `/ping`, configured as no-store, kept reaching the origin and exposing the problem;
- a scan burst could suddenly expose several bad workers;
- the failure looked like load exhaustion even though the traffic volume itself was not exceptional for OpenResty.

## Initial false leads

Several explanations initially looked plausible:

- recently edited Hugo content;
- a broken apex `arleo.eu` vhost while `www.arleo.eu` stayed healthy;
- OpenResty/CrowdSec overload under scanning;
- broken authentication on `/ping`.

The Hugo MCP quickly ruled content out: the build was healthy, publication was consistent and public pages were present.

The decisive evidence came from reading the **full local OpenResty error log**, not only aggregated HTTP metrics: the same Lua traceback accompanied the 500 responses.

## Applied fix

### Immediate fix: add the missing path

The first repair was:

```nginx
lua_package_path '$prefix/../lualib/plugins/crowdsec/?.lua;$prefix/lua/?.lua;;';
```

followed by:

```bash
sudo openresty -t
sudo openresty -s reload
```

After reload, fresh workers started cleanly, `module 'crowdsec.access' not found` stopped appearing, and the 500s disappeared.

### Durable fix: do not edit the package-rewritten file

`crowdsec_openresty.conf` is rewritten by CrowdSec bouncer upgrades and is not protected as a dpkg conffile. Editing its `lua_package_path` directly would therefore make the problem return after a future update.

The file was restored to its stock state and a symlink was created instead:

```text
/usr/local/openresty/lualib/plugins/crowdsec/crowdsec
    -> /usr/local/openresty/nginx/conf/lua/crowdsec
```

The official path:

```text
.../plugins/crowdsec/?.lua
```

can now naturally resolve:

```text
require("crowdsec.access")
→ .../plugins/crowdsec/crowdsec/access.lua
```

without modifying the package-managed configuration.

## Validation after the fix

Checks performed after reload:

- `openresty -t`: OK;
- fresh OpenResty workers started cleanly;
- 15 sequential `/ping` requests: no 500s;
- 30 concurrent `/ping` requests: no 500s;
- no new `module 'crowdsec.access' not found` errors;
- no 500s in the access log after the fix;
- `/` and Hugo pages: 200;
- apex `arleo.eu`: unchanged 301 redirect to `www`;
- BetterStack: `/ping` monitor returned **UP** after the repair.

A `401` from `/ping` without credentials is expected because the endpoint uses Basic Auth. BetterStack already had the correct authentication configured; previously `access_by_lua` crashed **before** nginx reached `auth_basic`.

## If this happens again

Do not start by rolling back Hugo.

### 1. Check the Lua signature immediately

```bash
sudo grep -n "crowdsec.access\|lua entry thread aborted" \
  /var/log/nginx/www.arleo.eu.error.log | tail -50
```

If this appears:

```text
module 'crowdsec.access' not found
```

the root cause is probably the same.

### 2. Confirm the module is resolvable

```bash
ls -la /usr/local/openresty/lualib/plugins/crowdsec/crowdsec
ls -la /usr/local/openresty/nginx/conf/lua/crowdsec/access.lua
```

The first path should be a symlink to the custom module directory.

### 3. Inspect the effective configuration

```bash
sudo openresty -T 2>&1 | grep -n "lua_package_path\|crowdsec_access"
```

### 4. Test configuration before reload

```bash
sudo openresty -t
```

Important: `openresty -t` checks syntax but **does not execute the request-phase `require()`**. A real HTTP request is mandatory.

### 5. Test the real request path

```bash
curl -sk -o /dev/null -w '%{http_code}\n' https://www.arleo.eu/ping
```

Without credentials, a clean `401` is better than a `500`. With the monitoring credentials, the expected result is `200`.

### 6. Confirm there are no new 500s

```bash
sudo tail -f /var/log/nginx/www.arleo.eu.error.log
```

Then issue several concurrent requests.

## Conclusion

The scanners `34.75.198.72` and `34.169.50.247` were the **observed triggers**, not the root cause. Around one hundred requests in a few seconds should not bring OpenResty down.

The actual failure was a **custom CrowdSec Lua module that existed on disk but was missing from the OpenResty workers' module search path**. The security layer had become a global failure point: one failed `require()` inside `access_by_lua` converted any request into HTTP 500 before the rest of the nginx chain could run.

The main lesson is simple: **when an outage correlates with hostile traffic, verify whether the traffic is merely exposing a latent defect in your own defensive layer.**

---

![CrowdSec, OpenResty and Lua handling malicious traffic and HTTP 500 failures](/images/logos/crowdsec-openresty-appsec.webp)

## Useful internal links

- [CrowdSec AppSec + OpenResty: modern WAF without ModSecurity](/en/posts/crowdsec-appsec-openresty/)
- [CrowdSec with Vector: filter noise, keep bans](/en/posts/crowdsec-vector-pipeline/)
- [CrowdSec postmortem: false positive on Sonarr/Radarr](/en/posts/postmortem-crowdsec-appsec-false-positive-sonarr/)
- [Post-Mortem — Incident 522 / WAN Failover](/en/posts/post-mortem-522-wan-failover/)

## Tags

- openresty
- crowdsec
- incident
- security
- homelab

## Categories

- incidents