Hugo SEO: noindex on taxonomies (Bing thin-content fix)

In short

Bing Webmaster Tools was raising a “There are too many pages with insufficient content” recommendation (severity: moderate) — 15 pages flagged, all of the same type:

• https://www.arleo.eu/en/tags/svg/
• https://www.arleo.eu/en/tags/sonarr/
• https://www.arleo.eu/en/categories/incidents/
• etc.

These are Hugo taxonomy pages — tag and category listing pages. They contain no article content of their own, just a list of links. From Bing’s perspective, that’s thin content.

This follows the same investigation habit as an earlier Googlebot 404s and sitemap normalization fix on this site: check the search console first, understand why Hugo generated the offending pages, then fix at the template level instead of patching each URL by hand.

Chosen strategy

Three options were available:

1. Block via robots.txt → discouraged: prevents crawling, hides internal links
2. Enrich taxonomy page content → too costly for a technical blog
3. noindex,follow → the right answer: tell robots not to index the page, but still follow links to articles

Option 3 is the correct call. The follow preserves internal linking — robots continue discovering articles through these pages.

Implementation

Hugo concept: .Kind

Hugo classifies every page by its kind:

The pages to handle are .Kind == "taxonomy" and .Kind == "term".

Change: layouts/baseof.html

The site uses a custom baseof.html that already overrides the LoveIt theme. It’s the natural place to centralise robots logic.

Before:

<meta name="robots" content="index, follow" />

After:

{{- if .Params.robots -}}
<meta name="robots" content="{{ .Params.robots }}" />
{{- else if or (eq .Kind "taxonomy") (eq .Kind "term") -}}
<meta name="robots" content="noindex,follow" />
{{- else -}}
<meta name="robots" content="index, follow" />
{{- end -}}

A three-level cascade:

1. Front matter robots: — absolute priority for future one-off exceptions
2. Taxonomies — automatic noindex,follow
3. Everything else — index, follow

One file changed, no duplication, no taxonomy layout overrides needed.

Removing taxonomies from the sitemap

Having noindex pages in a sitemap is inconsistent — you’re telling robots not to index a page while explicitly pointing them to it. Best practice is to remove them.

The LoveIt sitemap includes all pages. A minimal override in layouts/sitemap.xml:

{{- $excludedKinds := slice "taxonomy" "term" -}}
{{- range (where .Data.Pages "Section" "!=" "gallery") -}}
    {{- if not (in $excludedKinds .Kind) -}}
    <url>
        <loc>{{- .Permalink -}}</loc>
        ...
    </url>
    {{- end -}}
{{- end -}}

Result: the EN sitemap drops from 117 to 40 URLs (77 taxonomy entries removed).

Verification

After rebuild:

# Taxonomy pages → noindex,follow
curl -s https://www.arleo.eu/tags/ | grep -o 'name=robots[^>]*>'
# → name=robots content="noindex,follow">

curl -s https://www.arleo.eu/en/tags/sonarr/ | grep -o 'name=robots[^>]*>'
# → name=robots content="noindex,follow">

# Articles → index, follow
curl -s https://www.arleo.eu/en/posts/debug-seo-404-broken-links/ | grep -o 'name=robots[^>]*>'
# → name=robots content="index, follow">

# Clean sitemap
curl -s https://www.arleo.eu/en/sitemap.xml | grep -o '<loc>[^<]*</loc>' | grep 'tags\|categories' | wc -l
# → 0

Modified files

robots.txt was not touched. Taxonomy URLs remain accessible and crawlable — only their indexing is disabled.

Same principle as the site’s SRI hash automation for CDN assets: one small, centralised template change instead of a per-page workaround.

On the Bing side

The fix is immediate on the technical side. Bing will take a few days to re-crawl the pages and update the recommendation. To speed things up: in Bing Webmaster Tools → Recommendations → manually validate or submit a sitemap recrawl request.