Complete MCP Arleo v1.9.2 Audit by Claude.ai
Jmr
In short
Claude.ai (Pro account), connected to the same self-hosted MCP server (mcp.arleo.eu), ran a complete audit of the 72 tools exposed at admin scope on version v1.9.2 — tool by tool, with a systematic justification for every verdict.
What Claude.ai says about the server overall
Before the tool-by-tool detail, the overall assessment as formulated by Claude.ai itself:
- Real write cycles hold up end to end: create → update → plan → apply → rollback → delete, on both simple pages AND bilingual bundles, with hashes returning exactly to their starting value on every rollback
- Security/concurrency guardrails (revision guards,
asset_referenced,test_contentblockingdraft:false, the destructive quota) behave exactly as documented, including in deliberately provoked edge cases (chunk replay, an attempt to delete a referenced asset) - The three investigation points left open in the previous audit (determinism, stale cache, metric clarification) are not only fixed but verified under real conditions, not just read off a changelog
- Final cleanup: the site returns bit-for-bit to its initial state (86 pages, 0 residue, health score 100)
But Claude.ai was also explicit about what its own audit method cannot cover — structural blind spots, not identified defects:
- No real concurrency was tested. Every call is sequential, one agent, one principal. The server has multi-agent mechanisms (
change_set_id,foreign_change_set_present, per-caller isolation on previews/uploads) that were never put under real contention — two simultaneous writers on the same page, a genuine race condition on a chunked upload, and so on. - Post-build hooks were never actually triggered (no hook configured on this deployment) — only the
dry_run/no_hooks_configuredbehavior is validated, never a real HTTP delivery to a webhook that fails. - A single pass. A thorough but single audit cannot rule out a rare bug (a race, a volume edge case, behavior under load) that a single sequential pass cannot reveal.
activate_hugo/stage_hugo_upgrade/rollback_hugowere never executed for real (alwaysdry_run, out of caution — this action restarts the service) — this part of the contract remains verified on paper rather than in execution.
Audit context
Claude.ai picked up the thread of the three points left open in the previous audit (v1.9.1) and verified, on v1.9.2’s real runtime, that they were indeed resolved:
source_revisiondeterminism:.mcp-audit.logand in-progress chunked-upload files (.upload-*.part) are now excluded from the hash — a caller no longer sees its revision change without a real content change.- Stale-cache fingerprint for the tool catalog:
get_capabilitiesnow exposestool_catalog.tool_names_revision/visible_count, letting a client detect on its own a mismatch between its local cache and the catalog actually registered server-side. missing_counterpartsclarification: description updated inget_capabilities.
Three new chunked-upload tools (begin_asset_upload, upload_asset_chunk, commit_asset_upload, #1196) were also tested through a full cycle.
Verdict scale: very good / good / so-so / needs improvement / broken. On this v1.9.2 pass, every tool is rated very good — no “doesn’t work well” entry was forced in to artificially balance the summary.
Tool-by-tool detail
1. Reading — site, taxonomy, content
| Tool | Verdict | Why |
|---|---|---|
get_capabilities |
Very good | Exact structured response (version, scopes, limits, tool_catalog fingerprint); detected the v1.9.1→v1.9.2 transition instantly |
get_runtime_status |
Very good | safe_to_publish, external_unknown_changes, source_revision consistent every call; distinguishes drift from a real problem |
get_site_health |
Very good | Score, taxonomy_inconsistency_details, responsive_summary correct; distinguishes info (translation) from warning (real issue) |
get_site_information |
Very good | Minimal but exact response, no reason to make it more complex |
explain_structure |
Very good | Exact section/tag/category counts, detected the new abuseipdb-verification page without being asked to look for it explicitly |
list_content_types |
Very good | Correct archetypes and expected fields for the 5 observed types |
list_categories / list_tags |
Very good | Sorted lists, consistent with get_site_health |
get_sitemap |
Very good | summary_only returns exact counts without paying the cost of the 247 entries |
list_pages / get_recent_posts / get_feed |
Very good | The three scopes (all / posts / site-wide) are clearly distinct and consistent with each other, as documented |
search_pages |
Very good | Broad matching but correctly scored |
search_content |
Very good | FTS5 with working snippets, relevance sort respected |
get_broken_links |
Very good | 0 broken links across 247 documents, consistent before/after test writes |
validate_site / validate_frontmatter |
Very good | 86/86 pages valid, correctly included the test page in the scan then confirmed its disappearance after deletion |
check_sri_versions |
Very good | 9/9 SRI hashes verified against the real CDN |
get_theme_status |
Very good | Hugo version, theme commit, table_overflow_protection all exact |
get_storage_health |
Very good | 0 residue before and after the full cycle |
get_rate_limits |
Very good | Reflects other calls’ consumption in real time, verified by contrasting a free refusal against a real deletion that decrements it |
get_changelog |
Very good | Faithful raw Markdown body, since_version filters exactly as advertised |
get_hugo_update |
Very good | Real network comparison against the latest official release |
get_mutation_status |
Very good | Retrieved a delete_bundle by idempotency_key with the full envelope, succeeded status |
2. Planning & editing assistance
| Tool | Verdict | Why |
|---|---|---|
plan_page |
Very good | Combines content_types + relevant tags/categories + link suggestions in one call, no info loss vs the individual tools |
get_page_frontmatter |
Very good | Metadata only, without paying the body’s cost — useful for a quick confirmation |
get_page_markdown |
Very good | Faithful raw source, including on a draft/source-only page |
get_page_for_edit |
Very good | Exact compact bundle (frontmatter+markdown+state+revision), include works to add backlinks/impact on request |
build_agent_context |
Very good | Relevant related_pages (scored by shared tags), consistent with get_related_content |
export_agent_context |
Very good | Correct pagination, include_body:false correctly widened the limit to 50 |
suggest_links |
Very good | Relevant, scored suggestions, consistent between a direct call and via plan_page/get_related_content |
get_related_content |
Very good | All 4 facets (related/backlinks/suggested/translations) are consistent, include:["impact"] works |
get_backlinks |
Very good | Correctly returned 0 results for an isolated test page |
check_ai_readiness |
Very good | Correctly flagged the warn for missing metadata (no description) on a minimal test page |
3. Writing — a page’s lifecycle
| Tool | Verdict | Why |
|---|---|---|
create_page |
Very good | test_content correctly forces draft:true, idempotency_key available, dry_run consistent with the real write |
update_page |
Very good | data.changed distinguishes no-op from real edit; v1.9.2: bundle_revision returned on every success, FR→EN chaining direct |
delete_page |
Very good | bundle_fully_removed correct, expected_revision properly required and verified |
plan_content_change |
Very good | Exact previewed diff, add_tag/set_field correctly applied, no write in dry mode |
apply_content_plan |
Very good | Writes exactly what was previewed, plan consumed after use (single-use confirmed) |
rollback_change |
Very good | Exact return to the original hash on every test |
list_page_snapshots |
Very good | Lists the 2 expected snapshots after create+update, 24h expiry consistent |
list_page_revisions |
Very good | Real git history returned for a published page, distinct from snapshots as documented |
diff_page |
Very good | Correct behavior on a fresh, untracked file (git_untracked, full source returned as fallback) |
4. Writing — bilingual bundles
| Tool | Verdict | Why |
|---|---|---|
create_bundle |
Very good | v1.9.2: normalize_taxonomy_casing:true correctly transforms Mcp→mcp and Debug→debug, in dry_run and real writes |
plan_bundle_change / apply_bundle_plan |
Very good | Atomicity confirmed by a single bundle_revision consistent before/after |
rollback_bundle |
Very good | Atomic restoration of both translations together, never partial |
delete_bundle |
Very good | expected_revisions correctly required per language (a dict, not an array), atomic deletion confirmed |
5. Assets & images
| Tool | Verdict | Why |
|---|---|---|
upload_page_asset |
Very good | Byte sniffing confirmed (no blind trust in the declared extension) |
list_page_assets |
Very good | Correctly distinguishes bundle assets from generated_assets (hero image); tested on both normal and chunked cases |
delete_page_asset |
Very good | asset_referenced refusal confirmed without consuming the destructive quota, contrast verified against a real deletion right after |
generate_hero_image |
Very good | dry_run previews the exact contract without writing; the real 1200×675 image is compliant |
begin_asset_upload |
Very good (new in v1.9.2) | Immediate size validation before any transfer, upload_id scoped to the caller |
upload_asset_chunk |
Very good (new in v1.9.2) | Strict ordering respected; replay tested: resending the same chunk at offset 0 returned replayed:true with no double write |
commit_asset_upload |
Very good (new in v1.9.2) | sha256 verified at finalization, assembled file identical to a direct upload (same hash as the test file) |
6. Preview & rendering
| Tool | Verdict | Why |
|---|---|---|
preview_build |
Very good | Successful in-memory build, no artifact written to public disk |
create_preview |
Very good | Opaque URL, TTL respected, HttpOnly session cookie (behavior consistent with the docs) |
inspect_preview |
Very good | Correct SEO/security checks on an isolated build, correctly flagged the missing alt text (warn, not fail) |
list_previews |
Very good | Exactly reflects the active preview created just before |
revoke_preview / revoke_all_previews |
Very good | Immediate revocation confirmed, never any impact on another caller’s previews |
inspect_rendered |
Very good | 12 checks correct on a real published page, include_preview combines diff+broken_links+frontmatter with no duplication |
7. Build & publishing
| Tool | Verdict | Why |
|---|---|---|
build_site |
Very good | Every stage ok (hugo_build, output_swap, cloudflare_purge…); new search_index_submit callback in v1.9.2 |
publish_changes |
Very good | status:published only when build + verification are genuinely clean; intentionally_unpublished confirmed word for word |
verify_publication |
Very good | Real HTTP 200 verified, not just an internal status |
run_post_build_hooks |
Very good | no_hooks_configured correctly distinguished from a real failure; no hook configured here, so never tested under real HTTP delivery |
create_change_set |
Very good | Opaque ID generated, usable as change_set_id on subsequent mutations |
8. Hugo management (admin)
| Tool | Verdict | Why |
|---|---|---|
bootstrap_hugo |
Very good | Correct, explicit refusal (“already active”) rather than a generic error |
stage_hugo_upgrade |
Very good | dry_run previews the exact asset to download, checksum not verified in dry mode as expected |
activate_hugo |
Very good | dry_run shows previous_version for rollback, restart_required:true explicit |
rollback_hugo |
Very good | Correct restoration target, never executed for real out of caution (this action restarts the service) |
What changed since v1.9.1
- v1.9.1 → v1.9.2, +4 tools (68 → 72 at admin scope)
- All three previously open points are now fixed and verified under real conditions, not just on paper
- Real new content was detected on the site during the audit (the CrowdSec/OpenResty post-mortem, the
abuseipdb-verificationpage) — unrelated to the tests, correctly flagged as legitimate external drift (safe_to_publish:true,external_unknown_changes:0)
Why this contrast matters
Same server, same version, same week: ChatGPT (Plus account) can no longer get past initialize — zero tools loaded, zero calls possible — while Claude.ai (Pro account) runs a complete read and write cycle across 72 tools, with an honest account of its own coverage limits rather than an unqualified pass.
This is not a question of the MCP protocol’s general robustness: a third-party client explicitly emulating ChatGPT (MCPJam) also completes the flow without a hitch. The most likely cause therefore sits on the plan and client side of the providers rather than in the server implementation — though this cannot yet be stated with certainty at this stage (see the update below).
Update — the readOnlyHint/destructiveHint annotations hypothesis verified against the source code
After this article’s initial publication, a complementary lead was explored with Claude.ai: the ChatGPT block might stem, in addition to or instead of a simple subscription-tier restriction, from how ChatGPT classifies the perceived danger of tools exposed by an MCP server. A similar case documented on the OpenAI developer forum (a Business account, not Plus) had been unblocked by adding explicit readOnlyHint/destructiveHint annotations to tool descriptions.
Verified directly against origin/main of the mcp-hugo-server-go repository: this hypothesis was already anticipated, before the problem ever showed up on this server.
TestAllToolsHaveAnnotations(internal/server/tool_annotations_test.go) passes and covers the full catalog — all 72 tools, including the 4 admin Hugo tools, sinceNewStdionow builds the complete privileged catalog.- The test loads the catalog through a real in-memory MCP client/server exchange, not a plain inspection of Go structs — it exercises the same path a real client would see.
- Every tool must carry a non-nil
Annotationsblock. - Read-only tools named
get_*,list_*,search_*,validate_*,explain_*,diff_*,check_*,inspect_*, orsuggest_*must haveReadOnlyHint: true. - Exactly three tools have
DestructiveHint: true:delete_page,delete_page_asset,delete_bundle. Every other tool hasDestructiveHint: false.
Important nuance: not every tool necessarily has one of the two hints set to true. A recoverable mutation can legitimately carry both ReadOnlyHint: false and DestructiveHint: false — meaning “this tool writes, but is not considered destructive,” which is correct for create_page, update_page, plan_content_change, or rollback_change.
Conclusion of this check: all 72 tools do carry an annotations block and respect this classification policy — the missing-annotations hypothesis is therefore verified and largely ruled out as an additional explanation for the block. This does not settle the exact cause on ChatGPT’s side, which is still better stated as external to the server and likely tied to plan eligibility or a decision made by the ChatGPT client itself, rather than as a formally proven Plus restriction.
Why the model’s own opinion matters especially here
One point deserves to be stated explicitly: this tool is not generic software evaluated after the fact by human users — it is an interface designed from the start to be consumed by an LLM, and largely designed with LLMs (this MCP server is itself mostly developed by AI agents, through this same protocol, on this same repository).
That changes what “works well” actually means:
- The end consumer of this tool is the model, not a human clicking around. Tool descriptions, input/output schemas, structured error messages (
code,resolution.action,retry_after_seconds…) are not documentation about the product — they are literally the interface the model reads and interprets to decide what to do next. A human testing a web UI judges an experience they live through themselves. Here, Claude.ai auditing the server is the real end user of the contract it is evaluating — not a proxy for one. - A description poorly calibrated for a human can be even worse for a model, and vice versa. A human tolerates ambiguity, guesses intent, rereads the docs. A model that receives an incomplete schema or a non-actionable error message either hallucinates a fix, loops, or gives up — without this necessarily being visible before a real agent attempts the operation under real conditions. The fact that Claude.ai never had to guess, hallucinate a parameter, or reread a separate file to understand how to chain
create_bundle→update_page→rollback_changeis a direct signal about the interface’s design quality, not just the absence of bugs. - It loops back on itself, and that’s the most interesting part. Tool descriptions, structured error codes, fields like
bundle_revisionortool_names_revisionthat exist specifically so an agent doesn’t have to guess or reread — a good share of these design choices were proposed, discussed, and implemented by AI agents themselves, iteration after iteration. An LLM auditing this today is therefore finding, in part, the result of design work done by other LLM instances for LLMs. That the audit comes back “no ambiguity, no workaround needed” validates this circular design process, not just the day’s implementation. - The contrast with ChatGPT then takes on a different meaning. It is not just “one client works, the other doesn’t” in a networking sense — it is that, for the very use case this tool is built for (an AI agent completing a task end to end without constant human supervision), one of the two clients simply can never evaluate whether the tool is good, since it never reaches the stage where it could read it.
Concretely, the value of this kind of feedback is not “an LLM says it’s good, therefore it’s good” — it’s that the user best positioned to judge whether an interface designed to be read and acted on by a model fulfills its role is a model that has just read it and acted on it under real conditions.
See the full article on the ChatGPT Plus block for the log details, the A/B test across 4 server versions, and the official OpenAI sources consulted.