Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

SEO › Technical SEO

PDFs and Other Files in Search

How PDFs and other files end up in search results, when a file should rank instead of a page, and how response headers control indexing and canonical URLs.
Guideintermediate8 min readUpdated

Why files end up in search results

A file on a public URL behaves like a page to a crawler. Google's robots.txt documentation refers to PDFs and other text-based formats Google supports alongside web pages, which is to say that they can be crawled and appear in results like pages. Files are found the way pages are found: through links on the site, links from other sites, sitemaps and shared addresses in emails and documents that end up public.

Files are indexed for three common reasons. The site links to them prominently, often with descriptive anchor text, so they look important. They often contain the most specific text the site has, such as part numbers, specifications and figures, which matches specific searches better than a summary page does. And nobody told search engines otherwise, because the meta tags that control pages cannot be placed inside a PDF, and few teams think about response headers.

The result is familiar on technical and business-to-business sites: a datasheet ranks for a product's part number instead of the product page, an old brochure outranks the current service page, a price list from two years ago appears for a branded query, and a report meant to sit behind a sign-up form is one click from the results page.

When the file should rank and when its page should

A page is usually the better result. It carries the site's navigation and internal links, so a visitor can go further; it carries analytics, so the visit is measured; it carries the call to action, so the visit can convert; and it is easier to keep current. A PDF opened from a results page is a dead end on all four counts.

The file should rank when the file itself is what people want:

SituationWhat should rankWhy
Product datasheet; the product page has the same specificationsThe product pageIt can sell, link and be measured
Datasheet with detail the page does not showThe page, after it gains the detailMove the substance into HTML; keep the file as a download
Form people must print, sign or submit on paperThe file, with a page describing itThe file is the task
Formal report or publication people cite by titleEither, if the page presents it fullyThe file is the record; the page is the entry point
Brochure that repeats the service pageThe service pageThe brochure adds nothing for a searcher
Old version of a current documentNeither; retire the old versionOutdated files mislead
Gated white paper or researchThe landing pageThe file should not be reachable from search

When the page should rank and the file competes with it, the fix is rarely the file alone. Make the page deserve the position, by carrying the specifications, figures or detail that made the file match, then tell search engines what to do with the file.

Controlling indexing with response headers

Google's noindex documentation describes two ways to set the rule: a robots meta tag in a page's HTML, or an X-Robots-Tag HTTP response header, which can be used for non-HTML resources such as PDFs, video and image files. For files, the header is the only option:

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex

The same documentation states the condition that matters most: the rule works only if crawlers can fetch the file. If robots.txt blocks the file, the crawler never sees the header and the URL can still appear in results if other pages link to it. Google's robots documentation makes the same point from the other side and says not to use robots.txt to hide pages, including PDFs and other text-based formats, from search results. Its main use for robots rules on documents is managing crawl load. The comparison of noindex and robots.txt explains why the two controls do different jobs, and the crawling and indexing guide places both in the wider process.

Headers are set on the server or at the content delivery network, usually by rule rather than file by file: every file under a downloads folder, every file of a given type, or every file in a gated folder. Check that the hosting allows it; some document hosts and site builders do not let you set headers at all, which is a reason to serve important files from your own domain. Verify by requesting the file and reading the response headers, which any command-line HTTP client or browser developer panel shows.

Canonicals and alternates for files

Files are duplicated even more than pages: the same brochure uploaded to three folders, a report on the main site and on a campaign subdomain, a datasheet at a versioned and an unversioned address. Google's page on consolidating duplicate URLs describes a rel="canonical" Link header for non-HTML documents such as PDFs, which names the preferred address:

Link: <https://www.example.com/datasheets/pump-series-a.pdf>; rel="canonical"

A canonical is a statement that two URLs carry the same content. Use it between copies of the same document, or from a file to an HTML page that reproduces it in full. Where the HTML page only summarises the file, a canonical would claim a duplication that does not exist; noindex on the file, with the page made strong enough to rank, is the honest control. Better still, keep one address per document and redirect the others.

Translated documents need the same treatment as translated pages. Google's documentation on localised versions lists HTTP headers as one of the three ways to declare alternates and notes they are useful for non-HTML files such as PDFs. Each language version lists itself and every other version, with the hreflang value for each:

Link: <https://www.example.com/en/guide.pdf>; rel="alternate"; hreflang="en",
      <https://www.example.com/de/guide.pdf>; rel="alternate"; hreflang="de"

Gated files and download pages

Gated content sits behind a form, but a form does not protect a file. If the file has a public URL, it can be found: the thank-you page links to it, the confirmation email contains it, the page source reveals it, someone shares it in a forum, and a crawler that finds the link fetches the file without filling in anything. Search results then offer the white paper with no form at all.

Protect gated files in layers. Serve them from a folder that sends X-Robots-Tag: noindex on every file. Better, deliver them through links that expire or require the visitor's session, so a copied address stops working. Keep the thank-you page itself out of the index, since it exists to deliver the file. And make the landing page deserve its ranking: a substantial summary of what the asset contains, which is what can rank. Whether to gate an asset at all is a lead-generation decision, weighed in the comparison of gated and ungated content and the business-to-business content guide.

Making a file worth finding

Files that should rank deserve the same care as pages. Practice that makes a document readable and useful as a search result:

  • Real text, not images of text. A scanned page or a brochure exported as pictures gives a search engine and a screen reader little to read. Export from the source document with the text intact.
  • A meaningful document title in the file's properties, and a descriptive file name.
  • Headings and structure carried over from the source, so the document has an outline.
  • Links back to the site, so a reader who arrives from search has somewhere to go.
  • A stable address. Replace a document in place, or redirect the old address to the new version, rather than leaving every version live.
  • A page that presents it, with a summary and an Internal Link to the file, so the file is part of the site rather than an orphan.

Auditing the files on a site

Start with an inventory. A crawl that records content types finds linked files; server logs show files crawlers request that nothing links to any more; Search Console's reports list file URLs alongside pages, so filtering their exports by extension shows which files Google knows about and which bring clicks. For each file, record: the URL, type and owner; where it is linked from; whether an HTML page covers the same content; whether it should rank; the response headers it sends; whether it is gated; and whether a newer version exists.

Then apply the decisions from the table above, in this order: retire outdated versions with a redirect or a not-found response; consolidate duplicates; move substance from files into pages where the page should rank; set noindex headers on files that should not appear; protect gated files; and set canonical and alternate headers where copies and translations remain. Repeat the inventory at each technical audit, because files are uploaded by many people and rarely by the people who think about search.

Illustrative example — not a real result

Lanternmoss Engineering puts its files in order

Lanternmoss Engineering, an invented maker of industrial pumps, finds 640 PDF URLs in its crawl and logs, of which 410 are indexed. For 70 part-number queries, the datasheet appears in results instead of the product page, because the product pages carry a photograph and two sentences while the datasheets hold the full specifications. Nine gated white papers are indexed too, found through links in old confirmation emails that were later posted on a forum. The team adds HTML specification tables to the product pages, sends a noindex header on every datasheet in the downloads folder, redirects 180 superseded versions to their current files, and moves white papers to expiring download links with a noindex header on the folder. Four months later, 95 files remain indexed, all of them forms and published reports meant to rank; the product pages appear for 61 of the 70 part-number queries; and no gated file is reachable from search.

Common mistakes

  • Blocking files in robots.txt to keep them out of results. The header cannot be read, and the URL can still be listed.
  • Noindexing the datasheet before the page carries its content. The searcher then finds neither.
  • Canonicalising a file to a page that only summarises it. The hint describes a duplication that does not exist.
  • Trusting a form to protect a file. A public URL is public; use expiring links and headers.
  • Leaving every version live. Old prices and specifications keep ranking and misleading.
  • Images of text. A file that cannot be read cannot be found for what it says.
  • Hosting files where headers cannot be set. The controls in this guide then do not exist.

Frequently asked questions

How do I stop a PDF appearing in Google?

Send a noindex rule in the X-Robots-Tag HTTP response header for that file, and keep the file crawlable so the header can be read. Google's noindex documentation describes the header for non-HTML resources such as PDFs. Do not rely on robots.txt: Google's robots documentation says not to use it to hide pages, including PDFs, from search results, because a blocked URL can still be listed if other pages link to it.

Can a PDF have a canonical tag?

Not inside the file, but it can have one in its response headers. Google's page on consolidating duplicate URLs describes a rel canonical Link header for non-HTML documents such as PDFs. Use it to consolidate copies of the same document at different addresses, or to point a file at an HTML page that carries the same content in full.

Should I publish content as a PDF or as a web page?

As a web page, in most cases, with the PDF offered as a download for people who want to print or keep it. A page has the site's navigation, links, analytics and conversion paths; a PDF usually has none of them and is harder to update. The exceptions are documents whose value is the file itself, such as a form to print or a formal publication people cite.

Terms used on this page

Tools to try

Frameworks

Also related

Sources

  1. Block Search indexing with noindex — Google Search Central
  2. Consolidate duplicate URLs — Google Search Central
  3. Tell Google about localized versions of your page — Google Search Central
  4. Robots.txt introduction and guide — Google Search Central

Related