Why files end up in search results
A file on a public URL behaves like a page to a crawler. Google's robots.txt documentation refers to PDFs and other text-based formats Google supports alongside web pages, which is to say that they can be crawled and appear in results like pages. Files are found the way pages are found: through links on the site, links from other sites, sitemaps and shared addresses in emails and documents that end up public.
Files are indexed for three common reasons. The site links to them prominently, often with descriptive anchor text, so they look important. They often contain the most specific text the site has, such as part numbers, specifications and figures, which matches specific searches better than a summary page does. And nobody told search engines otherwise, because the meta tags that control pages cannot be placed inside a PDF, and few teams think about response headers.
The result is familiar on technical and business-to-business sites: a datasheet ranks for a product's part number instead of the product page, an old brochure outranks the current service page, a price list from two years ago appears for a branded query, and a report meant to sit behind a sign-up form is one click from the results page.
When the file should rank and when its page should
A page is usually the better result. It carries the site's navigation and internal links, so a visitor can go further; it carries analytics, so the visit is measured; it carries the call to action, so the visit can convert; and it is easier to keep current. A PDF opened from a results page is a dead end on all four counts.
The file should rank when the file itself is what people want:
| Situation | What should rank | Why |
|---|---|---|
| Product datasheet; the product page has the same specifications | The product page | It can sell, link and be measured |
| Datasheet with detail the page does not show | The page, after it gains the detail | Move the substance into HTML; keep the file as a download |
| Form people must print, sign or submit on paper | The file, with a page describing it | The file is the task |
| Formal report or publication people cite by title | Either, if the page presents it fully | The file is the record; the page is the entry point |
| Brochure that repeats the service page | The service page | The brochure adds nothing for a searcher |
| Old version of a current document | Neither; retire the old version | Outdated files mislead |
| Gated white paper or research | The landing page | The file should not be reachable from search |
When the page should rank and the file competes with it, the fix is rarely the file alone. Make the page deserve the position, by carrying the specifications, figures or detail that made the file match, then tell search engines what to do with the file.
Controlling indexing with response headers
Google's noindex documentation describes two ways to set the rule: a robots meta tag in a page's HTML, or an
X-Robots-Tag HTTP response header, which can be used for non-HTML resources such as PDFs, video and image
files. For files, the header is the only option:
HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex
The same documentation states the condition that matters most: the rule works only if crawlers can fetch the file. If robots.txt blocks the file, the crawler never sees the header and the URL can still appear in results if other pages link to it. Google's robots documentation makes the same point from the other side and says not to use robots.txt to hide pages, including PDFs and other text-based formats, from search results. Its main use for robots rules on documents is managing crawl load. The comparison of noindex and robots.txt explains why the two controls do different jobs, and the crawling and indexing guide places both in the wider process.
Headers are set on the server or at the content delivery network, usually by rule rather than file by file: every file under a downloads folder, every file of a given type, or every file in a gated folder. Check that the hosting allows it; some document hosts and site builders do not let you set headers at all, which is a reason to serve important files from your own domain. Verify by requesting the file and reading the response headers, which any command-line HTTP client or browser developer panel shows.
Canonicals and alternates for files
Files are duplicated even more than pages: the same brochure uploaded to three folders, a report on the main
site and on a campaign subdomain, a datasheet at a versioned and an unversioned address. Google's page on
consolidating duplicate URLs describes a rel="canonical" Link header for non-HTML documents such as PDFs,
which names the preferred address:
Link: <https://www.example.com/datasheets/pump-series-a.pdf>; rel="canonical"
A canonical is a statement that two URLs carry the same content. Use it between copies of the same document, or from a file to an HTML page that reproduces it in full. Where the HTML page only summarises the file, a canonical would claim a duplication that does not exist; noindex on the file, with the page made strong enough to rank, is the honest control. Better still, keep one address per document and redirect the others.
Translated documents need the same treatment as translated pages. Google's documentation on localised versions lists HTTP headers as one of the three ways to declare alternates and notes they are useful for non-HTML files such as PDFs. Each language version lists itself and every other version, with the hreflangAnnotation for language/region versions. value for each:
Link: <https://www.example.com/en/guide.pdf>; rel="alternate"; hreflang="en",
<https://www.example.com/de/guide.pdf>; rel="alternate"; hreflang="de"
Gated files and download pages
Gated contentContent a visitor can reach only after giving contact details, usually through a form. sits behind a form, but a form does not protect a file. If the file has a public URL, it can be found: the thank-you page links to it, the confirmation email contains it, the page source reveals it, someone shares it in a forum, and a crawler that finds the link fetches the file without filling in anything. Search results then offer the white paper with no form at all.
Protect gated files in layers. Serve them from a folder that sends X-Robots-Tag: noindex on every file. Better,
deliver them through links that expire or require the visitor's session, so a copied address stops working.
Keep the thank-you page itself out of the index, since it exists to deliver the file. And make the landing page
deserve its ranking: a substantial summary of what the asset contains, which is what can rank. Whether to gate an
asset at all is a lead-generation decision, weighed in the comparison of gated and ungated content
and the business-to-business content guide.
Making a file worth finding
Files that should rank deserve the same care as pages. Practice that makes a document readable and useful as a search result:
- Real text, not images of text. A scanned page or a brochure exported as pictures gives a search engine and a screen reader little to read. Export from the source document with the text intact.
- A meaningful document title in the file's properties, and a descriptive file name.
- Headings and structure carried over from the source, so the document has an outline.
- Links back to the site, so a reader who arrives from search has somewhere to go.
- A stable address. Replace a document in place, or redirect the old address to the new version, rather than leaving every version live.
- A page that presents it, with a summary and an Internal LinkA link from one page to another page on the same website. to the file, so the file is part of the site rather than an orphan.
Auditing the files on a site
Start with an inventory. A crawl that records content types finds linked files; server logs show files crawlers request that nothing links to any more; Search Console's reports list file URLs alongside pages, so filtering their exports by extension shows which files Google knows about and which bring clicks. For each file, record: the URL, type and owner; where it is linked from; whether an HTML page covers the same content; whether it should rank; the response headers it sends; whether it is gated; and whether a newer version exists.
Then apply the decisions from the table above, in this order: retire outdated versions with a redirect or a not-found response; consolidate duplicates; move substance from files into pages where the page should rank; set noindex headers on files that should not appear; protect gated files; and set canonical and alternate headers where copies and translations remain. Repeat the inventory at each technical audit, because files are uploaded by many people and rarely by the people who think about search.
Lanternmoss Engineering puts its files in order
Lanternmoss Engineering, an invented maker of industrial pumps, finds 640 PDF URLs in its crawl and logs, of which 410 are indexed. For 70 part-number queries, the datasheet appears in results instead of the product page, because the product pages carry a photograph and two sentences while the datasheets hold the full specifications. Nine gated white papers are indexed too, found through links in old confirmation emails that were later posted on a forum. The team adds HTML specification tables to the product pages, sends a noindex header on every datasheet in the downloads folder, redirects 180 superseded versions to their current files, and moves white papers to expiring download links with a noindex header on the folder. Four months later, 95 files remain indexed, all of them forms and published reports meant to rank; the product pages appear for 61 of the 70 part-number queries; and no gated file is reachable from search.
Common mistakes
- Blocking files in robots.txt to keep them out of results. The header cannot be read, and the URL can still be listed.
- Noindexing the datasheet before the page carries its content. The searcher then finds neither.
- Canonicalising a file to a page that only summarises it. The hint describes a duplication that does not exist.
- Trusting a form to protect a file. A public URL is public; use expiring links and headers.
- Leaving every version live. Old prices and specifications keep ranking and misleading.
- Images of text. A file that cannot be read cannot be found for what it says.
- Hosting files where headers cannot be set. The controls in this guide then do not exist.
