Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

SEO › Expert

Checkpoint: Server Logs, Parameters and Developer Requirements

Scenario questions set at an invented online auction house test the first expert module: what only server logs can show, which requests to trust, how template labels expose waste and neglect, which rule each class of parameter needs, how a register stays current and what makes a requirement something developers can build and test. Each answer gives the reasoning and the guide behind it.
CheckpointExpert levelModule 1: Crawl evidence at scale4 minUpdated

What this checkpoint covers

Calvenish Auctions is an invented online auction house for antiques and collectables. It publishes a sale catalogue of lots each week, keeps sold lots as an archive, and lets visitors sort catalogues, show them as a grid and price them in other currencies. The questions apply the log file analysis tutorial, the URL parameter audit and the guide to writing SEO requirements for developers.

Illustrative example — not a real result

Calvenish Auctions: a sale day and a week of verified crawler requests

Sale Thursday: requests with a Google crawler user agent 96,000; Search Console Crawl Stats for that day 31,000. On other days the two figures are within a few percent of each other. The hosting provider says 64,000 of Thursday's requests came from addresses belonging to a price-monitoring service.

TemplateURLsVerified requests in the week
Catalogue views with sort, view or currencyUnbounded58,000
Lot pages, past sales140,00022,000
Lot pages, current and upcoming sales12,00019,000
Category pages3004,100

Upcoming lots in the sitemap with no request in the week: 5,200 of 12,000. Most appear only on catalogue pages beyond the first, reached through pagination links that carry the current sort.

Task

  • For each scenario, write what Calvenish should do and the evidence that justifies it, then compare it with the explanation.
  • Where your reasoning differed, reopen the guide section linked in that answer before starting the tyre retailer exercise.

Questions

  1. The marketing manager says Search Console and a weekly site crawl show everything, so the logs can wait. What can only the logs tell Calvenish?
  2. On sale Thursday, logged crawler requests far exceed Crawl Stats. The developer wants to report that sales attract Google. What do you do first?
  3. The developer reads the verified lot page total as proof that lots are well crawled. What do the per-URL view and the sitemap comparison say, and where is the waste?
  4. A developer proposes a robots rule blocking every URL with a query string. The catalogue uses sort, view, currency, page, newsletter campaign tags and a lot parameter that opens one lot inside the catalogue. What goes wrong, and what replaces the proposal?
  5. Some months after the register was written, the logs show a fast-growing set of URLs carrying bidpanel=open, added by a bidding widget release. The register has no row for it. What failed, and what is the fix?
  6. A page for each maker of collectables is planned. The SEO draft says: "Maker pages must be SEO-friendly, indexable and fast, with good titles." Lots load by script after the first response. What does the requirement need?
  7. The developer suggests launching the maker pages now and fixing SEO next sprint, "once we see what Google does". How do you answer?

Answers explained

  1. Every request, URL by URL, including addresses nobody links to. A crawl shows what links reach today, and Crawl Stats aggregates Google's requests with examples. Only logs give per-template counts and reveal old catalogue URLs, stray parameter variants and status codes a browser check misses. See the log tutorial's comparison of sources.
  2. Verify before counting. A user agent is a claim, and an excess on one day against agreement on others points to impostors. Trace the excess to its addresses, have the hosting provider confirm they belong to the price-monitoring service rather than Google, and drop them before counting; verified, the sale day looks like any other. See checking the filter against Search Console.
  3. The total hides the neglect. Per URL, upcoming lots get few requests, and many got none, yet they must be found before the sale. Catalogue views take more requests than every lot page combined: that is the waste, and the cost to Crawl Budget falls on the newest lots. Make pagination links drop the sort first. See counting by template, status and depth.
  4. A blanket block breaks several classes at once. Pagination stops being crawled, so later lots go unfound; tagged newsletter links can no longer show their canonical; and lot changes the content, so it is a page that belongs on a clean path. Give each class of URL Parameter its rule: sort, view and currency unlinked and canonical to the default view, pagination crawlable and self-canonical, tracking never on internal links. See setting the rule for each class.
  5. Release review failed, not the audit. The rule for unknown keys should have caught it: a parameter with no row is handled like a campaign tag, never linked inside the site and canonical to the clean URL, until someone classifies it. Add the row to release review and compare the logs with the register monthly on a catalogue this large. See keeping the register current.
  6. Rules a developer can build and a tester can check. The page type's job and queries, the exact URL pattern, the Indexability and canonical rule for every URL type, lots and links in the server response rather than left to Rendering, title sources with fallbacks, test URLs with expected results, including a maker with no lots, and an owner for each check. See the steps of a requirement.
  7. Agree it before release; it is cheaper. Launched first, bad URLs get crawled, some indexed, the template is rebuilt and the fix competes for the backlog. Ask the developer to mark each line possible, possible with work or not possible, and record any compromise. See why requirements beat fixes after launch.

Where to go next

Misses on logs and parameters send you back to the job board crawl log exercise; misses on requirements, to the search template requirement it ends with. Then start the second module with the tyre retailer title test exercise, or revisit the advanced SEO level for indexing controls.

Pages this lesson applies

Terms used on this page

Also related

Related