Pages a CMS creates without being asked
Publishing an article on a typical content platform creates more than one page. The article appears on its own URL, on the page for each tag and category it carries, on its author's archive, on archives for its day, month and year, on the home page and the feed, and sometimes on an attachment page for each image. Internal search adds a results page for every query anyone types. Print views, preview links, login and account pages, carts and confirmation pages complete the set.
None of these was written. Each is a list or a function, generated from the same articles, and most of them look alike: a heading, a few excerpts and pagination. On a site with a few hundred articles and liberal tagging, the generated pages can outnumber the written ones many times over. Search Console's Page Indexing report is where the consequence shows: generated lists in the crawled but not indexed row, archives competing with the articles they list, and tag pages appearing for queries where an article should.
The crawled, currently not indexed diagnosis ends with the principle of publishing only pages that deserve the index. This guide applies that principle to every page type a content platform generates. Store filters, which multiply in the same way, follow the rules in the faceted navigation guide.
The test a generated page must pass
Google's guidance on helpful, people-first content asks whether a page provides substantial value compared with other pages in search results, and whether someone arriving at it would leave feeling they had a satisfying experience. A generated page has to pass the same questions as a written one. In practice, a generated page deserves the index when all four of these are true:
- It has a distinct purpose. It answers a need no other page on the site answers better: a collection on a topic people search for, a person whose work readers seek out.
- It has enough substance. It lists enough items to be a useful collection, and preferably carries text of its own that says what the collection is.
- It does not duplicate another page. A tag that matches a category, two tags that are spellings of one idea, or an archive that lists the same articles as the home page fail this test.
- Someone owns it. A page that will be kept current, introduced and linked deliberately.
A page that fails is one of three things: a candidate for merging into the page it duplicates, a page that exists for visitors but stays out of the index, or a page that should not exist at all. The keyword cannibalisation guide covers the merge decision when two pages compete for the same searches.
Tags and categories
Categories are usually a small, planned set that mirrors the site's main topics, and they are the natural hubs of a content site. Treat each as a page: a written introduction, the best articles first rather than only the newest, links to sub-topics, and a self-referencing canonical. They belong in the index and in the sitemap, and their structure is the topic cluster the site is built on.
Tags are the opposite: an open set that grows with every article and every author's habits. A few tags describe topics people search for and gather enough articles to be useful; most describe one article. The policy:
- Promote the tags that pass the test into hubs with introductions, or into categories if they are really sections of the site.
- Merge duplicates. Plurals, spellings, synonyms and tags that match a category are combined, with a permanent redirect from the retired tag to the surviving page.
- Noindex the rest. Tags kept for navigation but failing the test carry a noindexA robots directive telling search engines not to show a page in their results., leave the sitemap and are not linked from site-wide tag clouds.
- Limit new tags. A controlled list, or a rule that new tags are reviewed, stops the problem coming back.
Paginated pages of a hub follow the hub: if the hub is indexable, each page in its PaginationSplitting a long list of items, such as products or articles, across a series of numbered pages. series is self-canonical and linked with plain links; if the hub is noindexed, its later pages are too.
Author and date archives
Author archives pass the test when the author is someone readers look for: a columnist, an expert contributor, a named reviewer. Then the page is a profile with a biography, credentials and the author's best work, and it supports the site's claim to experience and expertise. Sites that publish outside writers can follow the guide to accepting guest contributions for the bylines and author pages that make a contributor's archive worth indexing. On a single-author site the author archive duplicates the home page or the blog index, and on a site where authors are staff writers with no public profile it is a thin list; noindex it, or redirect it to the about page.
Date archives almost never pass. Nobody searches for a site's articles from a particular month, and every date archive duplicates the main list in a different slice. Noindex them, remove them from sitemaps and navigation, or turn them off if the platform allows. The exception is a news or events site where dates are the way people browse, and even there a dated listing is navigation, not a search result.
Internal search and sorting views
Internal search results are an unbounded set of list pages, one per query, and they should not be indexed. The faceted navigation guide covers them for stores; the rule is the same on a content site. Sorting and display views, such as oldest first or a grid instead of a list, are the same list in a different order and canonicalise to the default view.
The sequence matters for search pages that are already indexed. Google's noindex documentation is explicit that the rule only works if the page can be crawled; a page blocked by robots.txtA file at a site's root telling crawlers which URLs they may and may not fetch. never shows its noindex and can remain in results if other pages link to it. So set noindex on the search template first, remove crawlable links to search results, and let the pages be recrawled and drop out. Only then, if crawling of search URLs is still a burden, add a robots rule for the pattern. On a new site, keep search results unlinked and blocked from the start. The comparison of noindex and robots.txt sets out why the two controls do different jobs.
Utility pages: login, cart, thank-you and print
Utility pages serve visitors who are already on the site and have nothing to offer a searcher:
- Login, registration and account pages. Noindex, or behind authentication where they hold personal content.
- Cart and checkout. Noindex; they are empty or personal to a crawler.
- Thank-you and confirmation pages. Noindex, since a confirmation reached from search is meaningless and, where it delivers a download, a leak.
- Print views. A duplicate of the article. Google's page on consolidating duplicate URLs describes the canonical link as the way to name the preferred version, so a print view that must stay reachable canonicalises to the article; better still, use a print stylesheet and no separate URL.
- Image attachment pages. A page per image with nothing but the image; redirect to the article or turn them off.
- Threads and question pages with no replies, on sites with forums or community answers. Noindex until they have substance, then let them in.
The Page Indexing report shows the outcome: noindexed templates appear as excluded by the noindex tag, canonicalised views as alternate pages with a proper canonical tag. Both are expected states, not errors.
Setting the rules per template
Rules belong to templates, not pages, so that every new tag, author or month inherits the right treatment. Write the policy as a table, implement it in the theme or platform settings, and check it with a crawl.
| Template | Default rule | Canonical | In sitemap | Linked from |
|---|---|---|---|---|
| Category | Index, as a hub with its own text | Self | Yes | Navigation, articles, breadcrumbs |
| Tag that passes the test | Index, as a hub with its own text | Self | Yes | Articles, related hubs |
| Tag that fails the test | Noindex, or merge and redirect | Self, if kept | No | Articles only, sparingly |
| Author with a public profile | Index, as a profile | Self | Yes | Bylines |
| Author without a profile | Noindex, or redirect to the about page | Self, if kept | No | Bylines, if at all |
| Date archives | Noindex, or disable | Self, if kept | No | Nowhere prominent |
| Internal search results | Noindex; robots rule once dropped | None needed | No | Search form only |
| Sorting and display views | Index the default only | Default view | No | Controls without crawlable links |
| Print and attachment views | No separate URL, or canonical to the article | Article | No | Nowhere |
| Login, account, cart, checkout, thank-you | Noindex | Self | No | Functional links only |
Two cautions apply to the whole table. First, the crawling and indexing guide notes Google's statement that a page left out of the index for a long time is eventually crawled less and its links may stop being followed, so every article must also be reachable through pages that stay indexable; a noindexed archive must never be the only path. Second, do not combine noindex and a canonical to another page on the same template; one says index nothing, the other says index that one instead. Check every Internal LinkA link from one page to another page on the same website. in navigation, sidebars and footers against the table, because a site-wide tag cloud can undo the policy in one template.
Pebblequill Journal prunes its tag pages
Pebblequill Journal, an invented home and garden magazine, has 1,900 articles carrying an average of six tags each and 4,700 tag pages, 3,600 of which list one or two articles. The Page Indexing report shows 3,100 URLs crawled but not indexed, most of them tag pages, and several tags appear in results for queries that a full guide on the site answers better. Applying the test, the editors keep 90 tags as hubs and write an introduction for each, merge 620 duplicates and synonyms into those hubs or into categories with permanent redirects, and set the remaining 3,990 tags to noindex, removing them from the sitemap and the site-wide tag cloud. Date archives are noindexed and author archives kept only for 14 regular contributors with profiles. Four months later, the crawled but not indexed count is 700, the 90 tag hubs are indexed, and they receive more search clicks than all tag pages together did before.
Common mistakes
- Indexing everything the platform generates. Thousands of thin lists compete with the articles they list.
- Noindexing every archive, categories included. The site loses its hubs and articles lose their strongest paths.
- Blocking already-indexed archives in robots.txt. The noindex can never be seen, and the URLs linger in results.
- Leaving tag clouds on every page. Site-wide links keep feeding crawlers the pages the policy retired.
- Canonicalising paginated archive pages to page one. Items on later pages lose their path.
- Deciding tag by tag. Without a written test, the decisions drift and the next author recreates the problem.
- Forgetting utility pages. A thank-you page in the index can hand out a gated download to anyone.
