Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

SEO › Technical SEO

Index Rules for Tags, Archives and Utility Pages

Which pages a CMS creates automatically deserve the index: tags, categories, author and date archives, internal search, utility pages, and how to set rules.
Guideintermediate9 min readUpdated

Pages a CMS creates without being asked

Publishing an article on a typical content platform creates more than one page. The article appears on its own URL, on the page for each tag and category it carries, on its author's archive, on archives for its day, month and year, on the home page and the feed, and sometimes on an attachment page for each image. Internal search adds a results page for every query anyone types. Print views, preview links, login and account pages, carts and confirmation pages complete the set.

None of these was written. Each is a list or a function, generated from the same articles, and most of them look alike: a heading, a few excerpts and pagination. On a site with a few hundred articles and liberal tagging, the generated pages can outnumber the written ones many times over. Search Console's Page Indexing report is where the consequence shows: generated lists in the crawled but not indexed row, archives competing with the articles they list, and tag pages appearing for queries where an article should.

The crawled, currently not indexed diagnosis ends with the principle of publishing only pages that deserve the index. This guide applies that principle to every page type a content platform generates. Store filters, which multiply in the same way, follow the rules in the faceted navigation guide.

The test a generated page must pass

Google's guidance on helpful, people-first content asks whether a page provides substantial value compared with other pages in search results, and whether someone arriving at it would leave feeling they had a satisfying experience. A generated page has to pass the same questions as a written one. In practice, a generated page deserves the index when all four of these are true:

  1. It has a distinct purpose. It answers a need no other page on the site answers better: a collection on a topic people search for, a person whose work readers seek out.
  2. It has enough substance. It lists enough items to be a useful collection, and preferably carries text of its own that says what the collection is.
  3. It does not duplicate another page. A tag that matches a category, two tags that are spellings of one idea, or an archive that lists the same articles as the home page fail this test.
  4. Someone owns it. A page that will be kept current, introduced and linked deliberately.

A page that fails is one of three things: a candidate for merging into the page it duplicates, a page that exists for visitors but stays out of the index, or a page that should not exist at all. The keyword cannibalisation guide covers the merge decision when two pages compete for the same searches.

Tags and categories

Categories are usually a small, planned set that mirrors the site's main topics, and they are the natural hubs of a content site. Treat each as a page: a written introduction, the best articles first rather than only the newest, links to sub-topics, and a self-referencing canonical. They belong in the index and in the sitemap, and their structure is the topic cluster the site is built on.

Tags are the opposite: an open set that grows with every article and every author's habits. A few tags describe topics people search for and gather enough articles to be useful; most describe one article. The policy:

  • Promote the tags that pass the test into hubs with introductions, or into categories if they are really sections of the site.
  • Merge duplicates. Plurals, spellings, synonyms and tags that match a category are combined, with a permanent redirect from the retired tag to the surviving page.
  • Noindex the rest. Tags kept for navigation but failing the test carry a noindex, leave the sitemap and are not linked from site-wide tag clouds.
  • Limit new tags. A controlled list, or a rule that new tags are reviewed, stops the problem coming back.

Paginated pages of a hub follow the hub: if the hub is indexable, each page in its Pagination series is self-canonical and linked with plain links; if the hub is noindexed, its later pages are too.

Author and date archives

Author archives pass the test when the author is someone readers look for: a columnist, an expert contributor, a named reviewer. Then the page is a profile with a biography, credentials and the author's best work, and it supports the site's claim to experience and expertise. Sites that publish outside writers can follow the guide to accepting guest contributions for the bylines and author pages that make a contributor's archive worth indexing. On a single-author site the author archive duplicates the home page or the blog index, and on a site where authors are staff writers with no public profile it is a thin list; noindex it, or redirect it to the about page.

Date archives almost never pass. Nobody searches for a site's articles from a particular month, and every date archive duplicates the main list in a different slice. Noindex them, remove them from sitemaps and navigation, or turn them off if the platform allows. The exception is a news or events site where dates are the way people browse, and even there a dated listing is navigation, not a search result.

Internal search and sorting views

Internal search results are an unbounded set of list pages, one per query, and they should not be indexed. The faceted navigation guide covers them for stores; the rule is the same on a content site. Sorting and display views, such as oldest first or a grid instead of a list, are the same list in a different order and canonicalise to the default view.

The sequence matters for search pages that are already indexed. Google's noindex documentation is explicit that the rule only works if the page can be crawled; a page blocked by robots.txt never shows its noindex and can remain in results if other pages link to it. So set noindex on the search template first, remove crawlable links to search results, and let the pages be recrawled and drop out. Only then, if crawling of search URLs is still a burden, add a robots rule for the pattern. On a new site, keep search results unlinked and blocked from the start. The comparison of noindex and robots.txt sets out why the two controls do different jobs.

Utility pages: login, cart, thank-you and print

Utility pages serve visitors who are already on the site and have nothing to offer a searcher:

  • Login, registration and account pages. Noindex, or behind authentication where they hold personal content.
  • Cart and checkout. Noindex; they are empty or personal to a crawler.
  • Thank-you and confirmation pages. Noindex, since a confirmation reached from search is meaningless and, where it delivers a download, a leak.
  • Print views. A duplicate of the article. Google's page on consolidating duplicate URLs describes the canonical link as the way to name the preferred version, so a print view that must stay reachable canonicalises to the article; better still, use a print stylesheet and no separate URL.
  • Image attachment pages. A page per image with nothing but the image; redirect to the article or turn them off.
  • Threads and question pages with no replies, on sites with forums or community answers. Noindex until they have substance, then let them in.

The Page Indexing report shows the outcome: noindexed templates appear as excluded by the noindex tag, canonicalised views as alternate pages with a proper canonical tag. Both are expected states, not errors.

Setting the rules per template

Rules belong to templates, not pages, so that every new tag, author or month inherits the right treatment. Write the policy as a table, implement it in the theme or platform settings, and check it with a crawl.

TemplateDefault ruleCanonicalIn sitemapLinked from
CategoryIndex, as a hub with its own textSelfYesNavigation, articles, breadcrumbs
Tag that passes the testIndex, as a hub with its own textSelfYesArticles, related hubs
Tag that fails the testNoindex, or merge and redirectSelf, if keptNoArticles only, sparingly
Author with a public profileIndex, as a profileSelfYesBylines
Author without a profileNoindex, or redirect to the about pageSelf, if keptNoBylines, if at all
Date archivesNoindex, or disableSelf, if keptNoNowhere prominent
Internal search resultsNoindex; robots rule once droppedNone neededNoSearch form only
Sorting and display viewsIndex the default onlyDefault viewNoControls without crawlable links
Print and attachment viewsNo separate URL, or canonical to the articleArticleNoNowhere
Login, account, cart, checkout, thank-youNoindexSelfNoFunctional links only

Two cautions apply to the whole table. First, the crawling and indexing guide notes Google's statement that a page left out of the index for a long time is eventually crawled less and its links may stop being followed, so every article must also be reachable through pages that stay indexable; a noindexed archive must never be the only path. Second, do not combine noindex and a canonical to another page on the same template; one says index nothing, the other says index that one instead. Check every Internal Link in navigation, sidebars and footers against the table, because a site-wide tag cloud can undo the policy in one template.

Illustrative example — not a real result

Pebblequill Journal prunes its tag pages

Pebblequill Journal, an invented home and garden magazine, has 1,900 articles carrying an average of six tags each and 4,700 tag pages, 3,600 of which list one or two articles. The Page Indexing report shows 3,100 URLs crawled but not indexed, most of them tag pages, and several tags appear in results for queries that a full guide on the site answers better. Applying the test, the editors keep 90 tags as hubs and write an introduction for each, merge 620 duplicates and synonyms into those hubs or into categories with permanent redirects, and set the remaining 3,990 tags to noindex, removing them from the sitemap and the site-wide tag cloud. Date archives are noindexed and author archives kept only for 14 regular contributors with profiles. Four months later, the crawled but not indexed count is 700, the 90 tag hubs are indexed, and they receive more search clicks than all tag pages together did before.

Common mistakes

  • Indexing everything the platform generates. Thousands of thin lists compete with the articles they list.
  • Noindexing every archive, categories included. The site loses its hubs and articles lose their strongest paths.
  • Blocking already-indexed archives in robots.txt. The noindex can never be seen, and the URLs linger in results.
  • Leaving tag clouds on every page. Site-wide links keep feeding crawlers the pages the policy retired.
  • Canonicalising paginated archive pages to page one. Items on later pages lose their path.
  • Deciding tag by tag. Without a written test, the decisions drift and the next author recreates the problem.
  • Forgetting utility pages. A thank-you page in the index can hand out a gated download to anyone.

Frequently asked questions

Should tag pages be indexed?

Only the ones that work as hubs: a tag that collects enough articles on a topic people search for, with an introduction that says what the collection covers and why it is useful. Most tag pages list one or two posts and duplicate a category or another tag, and those are better merged into a stronger page or kept out of the index with noindex. Decide by a written test, not tag by tag.

Can I block tag and archive pages in robots.txt instead of using noindex?

Not for pages that are already indexed. Google's noindex documentation explains that the rule is seen only if the page can be crawled; a page blocked by robots.txt never shows its noindex and can stay in results if other pages link to it. Use noindex, let the pages be recrawled and drop out, and keep crawlable links to them to a minimum. A robots rule is a crawl control, which the comparison of noindex and robots.txt on this site sets out.

Do noindexed archive pages still pass links to my articles?

Do not rely on it. The crawling and indexing guide on this site notes Google's statement that a page kept out of the index for a long time is eventually crawled less and its links may stop being followed. Make sure every article is also reachable through pages that stay indexable, such as categories, topic hubs and related-article links, so that noindexed archives are never the only path.

Terms used on this page

Tools to try

Frameworks

Also related

Sources

  1. Block Search indexing with noindex — Google Search Central
  2. Consolidate duplicate URLs — Google Search Central
  3. Creating helpful, reliable, people-first content — Google Search Central
  4. Page Indexing report — Google Search Console Help

Related