Skip to content
Faisal Mahmud TusherDigital marketing, SEO & analytics
with Faisal Tusher

SEO › Expert

Reading Crawl Logs and URL Parameters for a Job Board

You take an invented job board's server logs, keep only the requests that really came from a search engine crawler and read them by template to find where crawling is wasted and which new vacancies are neglected. You finish with a parameter register that gives every class one rule and an owner, an expiry policy for closed vacancies, and a requirement for the search template that developers can build and test before release.
ExerciseExpert levelModule 1: Crawl evidence at scale7 minUpdated

Why this matters

The advanced SEO level chose from demand which filter pages deserve the index; only server logs show where crawlers actually went. The log file analysis tutorial yields waste and neglect, the URL parameter audit rules by class, and the guide to writing SEO requirements for developers behaviour a team can build and test. A job board, whose pages open and close daily, strains all three.

The job board and its week of logs

Ivelsorn Jobs is an invented job board for hospitality and retail vacancies. Each vacancy has its own page, role and area landing pages list open vacancies, employers have profile pages, and a search tool lets visitors filter, sort and search by keyword. The owner's complaint is that new vacancies take days to appear in search while Search Console reports a growing pile of soft errors.

Illustrative example — not a real result

Ivelsorn Jobs: seven days of crawler requests and the parameters behind them

Requests with a Google crawler user agent: 252,400. Of these, 38,900 came from addresses that failed verification, which the hosting company traced to a scraping service; 36,100 of them were to search URLs. Verified requests: 213,500. Search Console Crawl Stats for the same week: 216,000.

TemplatePatternURLsVerified requestsWhat the crawler received
Search tool/jobs/search?…Unbounded104,600200
Closed vacancies/job/[id]/41,00047,300200 and one line: "This job has closed"
Live vacancies/job/[id]/6,80030,700 (9,800 with ?apply=1)200
Old vacancy addresses/vacancy.php?id=5,600 seen11,200301 to /job/[id], then 301 to add the slash
Role and area landing pages/jobs/[role]/[area]/1,2009,400200
Employer pages/employers/[name]/9003,100200
Articles and other pagesvarious3007,200200

Live vacancies not requested at all in the week: 2,450 of 6,800, of which 1,980 were posted in the last ten days. Landing pages list vacancies by relevance, so new ones usually start on page three or later. The vacancy sitemap is rebuilt every Sunday and lists 9,300 URLs, 2,900 of them closed. Search Console page indexing shows 18,700 URLs as soft 404, every example a closed vacancy. Closed vacancies stay linked from each employer's "past vacancies" list. About one closed role in eight is reposted under a new ID.

Parameters found in the crawl, the logs and Search Console examples:

ParameterDistinct URLs in the logsDeveloper's noteInternal links carry it
role, area3,400Narrow search to one role or areaFilter links
contract, pay_min, radius28,900Narrow resultsFilter links
sort22,600Newest, pay or distanceSort links, and pagination links keep it
page9,100PaginationPagination links
q6,300Keyword search"Popular searches" footer block
utm_source, utm_medium, utm_campaign1,850Campaign tagsHomepage banner; job alert emails
ref740Partner widget on employers' own sitesNone
apply6,200Opens the application form over the pageApply button, written as a link
sid410Session from the previous platformNone

Next quarter the product team plans saved searches, which will add a parameter.

Evidence before rules

Verify the crawler first, because impostors inflate the templates they scrape. Read requests per URL beside totals, since a healthy total can hide unvisited pages, and compare the crawled set with the sitemap. Then give rules to classes of parameter, and send every finding with an example URL and the change expected next.

Task

  1. Decide which requests count and say what would have gone wrong without that step.
  2. Read the verified requests by template, as totals and per URL, and list the waste and the neglect, each with an example URL and the change you expect in the logs.
  3. Classify every parameter, give each class its rule and mechanism, and set how any parameter added later is treated, including the planned saved searches.
  4. Decide what happens to a vacancy when it closes, and when the same role is reposted.
  5. Write the requirement for the search template with acceptance checks, test URLs and owners.
  6. Say when and how the fixes are measured again, and who keeps the evidence current.

Worked answer

Which requests count. The requests from unverified addresses are excluded, and the verified total then sits close to Crawl Stats, which covers the whole property. Left in, the scraper's traffic would have made the search tool look even worse than it is and sent the first fixes after a problem that is not the crawler's.

Waste and neglect. The search tool takes the largest share of verified crawling, and closed vacancies the next, returning success with nothing on them, which Search Console already reports as Soft 404 pages. Redirect chains from the old platform add a further slice. On a site that changes this fast, Crawl Budget is a real constraint, and the neglect list shows the cost: new vacancies are the pages the board most needs crawled, and they sit deep in landing pages and appear in a sitemap that is stale for most of the week.

Illustrative example — not a real result

Ivelsorn Jobs: the arithmetic behind the two lists

Shares of verified requests: search tool 49%, closed vacancies 22%, live vacancies 14%, old addresses 5%, landing pages 4%. Requests per URL in the week: landing pages 7.8, live vacancies 4.5, employer pages 3.4, closed vacancies 1.2. The live average hides that 36% of live vacancies were not requested at all, and 81% of those were posted in the last ten days. Example tickets: /jobs/search?sort=pay&contract=seasonal&page=7, a closed vacancy at /job/48213/, and /vacancy.php?id=48213 reaching the vacancy in two hops.

The register. Rules go to classes, with the lightest mechanism that works:

ClassParametersRuleMechanism
Narrows, page existsrole, areaResolve to the landing pageFilter links point at the clean path; a search URL with only these redirects to it
Narrows, no page plannedcontract, pay_min, radiusNot linked; canonical to the page they narrowForm controls, not anchors
Reorders or changes layoutsort, apply (apply only overlays the form)Not linked; canonical to the default viewButtons; the form opens without changing the address
PaginatespageCrawlable, self-canonical, linked on its ownPagination links drop the sort
SearchesqOut of the index and the crawlFooter block replaced by landing page links, then a robots rule
Tracksutm_ tags, refNever on internal links; crawlable; canonical to the clean URLBanner rebuilt without tags
Holds a sessionsidOut of URLsRedirect that strips it

Keyword search URLs lose their footer links first and the robots rule follows. The log tutorial's warning applies: a robots.txt rule stops fetching but does not by itself remove a URL already indexed, so check the indexing report before and after. Tracking tags are not blocked, so tagged links can still pass on their canonical. Old vacancy addresses redirect in a single hop. No new parameter ships without a row, and until classified it is treated as tracking; saved searches get theirs now, as a session held in the visitor's account rather than the URL. The product manager for search owns the register, and the SEO lead checks it against the logs every month.

Closed vacancies. The requirements guide asks every template to say what happens when a page is deleted: a redirect to the new URL or a not-found response. The log tutorial counts pages that return success but say nothing as soft errors. So a closed vacancy leaves the sitemap and the employer page the same day and stops answering with success, and a reposted role redirects permanently to its new vacancy. The expired pages guide sets out the wider lifecycle options for listings.

The search template requirement. It follows the requirements guide's steps, and its opening line settles most arguments: the search tool serves visitors, and landing pages own the queries.

Illustrative example — not a real result

Ivelsorn Jobs: the search template requirement, shortened

AreaRequirementAcceptance check
URLsRole and area alone redirect to /jobs/[role]/[area]/; other parameters lower case in register order/jobs/search?area=x&role=chef returns one 301 to the landing page
LinksFilters, sort and apply are buttons or form fields; pagination links carry only pageA crawl of staging finds no anchor with sort, apply, q, utm_ or sid
SitemapRebuilt within an hour of any vacancy opening or closing; live vacancies onlyNo closed vacancy in the sitemap the morning after closing
Test URLsTwelve, including ?sort=pay&page=2, ?SID=abc, a closed vacancy, a reposted vacancy and an unknown parameterEach has an expected status, canonical and directive
OwnersThe developer automates the URL and link checks, the tester works through the twelve test URLs, the SEO lead reads the staging outputNothing ships while any of the three checks is open

Measuring again. After the fixes have been live a few weeks, repeat the analysis on a window of the same length. The expected change is search tool requests down, live vacancy requests per URL up, fewer new vacancies unvisited, soft errors falling and single-hop redirects. The log review then joins the monthly routine, beside the crawling checks in the technical SEO audit checklist.

Common mistakes

  • Counting every request that claims to be a crawler. The scraper inflates the very template you are about to fix.
  • Reading totals only. The live vacancy total looks healthy while many of its pages go unvisited.
  • A rule for every parameter. Nobody maintains a row of rules per key; a handful of classes stays current.
  • Leaving closed vacancies at success with one line of text. That is the soft error the report already shows.
  • Findings without example URLs or tests. Developers fix what they can reproduce and check.

The crawl evidence checkpoint puts the same calls to an auction house.

Pages this lesson applies

Terms used on this page

Also related

Related