Why this matters
The advanced SEO level chose from demand which filter pages deserve the index; only server logs show where crawlers actually went. The log file analysis tutorial yields waste and neglect, the URL parameter audit rules by class, and the guide to writing SEO requirements for developers behaviour a team can build and test. A job board, whose pages open and close daily, strains all three.
The job board and its week of logs
Ivelsorn Jobs is an invented job board for hospitality and retail vacancies. Each vacancy has its own page, role and area landing pages list open vacancies, employers have profile pages, and a search tool lets visitors filter, sort and search by keyword. The owner's complaint is that new vacancies take days to appear in search while Search Console reports a growing pile of soft errors.
Ivelsorn Jobs: seven days of crawler requests and the parameters behind them
Requests with a Google crawler user agent: 252,400. Of these, 38,900 came from addresses that failed verification, which the hosting company traced to a scraping service; 36,100 of them were to search URLs. Verified requests: 213,500. Search Console Crawl Stats for the same week: 216,000.
| Template | Pattern | URLs | Verified requests | What the crawler received |
|---|---|---|---|---|
| Search tool | /jobs/search?… | Unbounded | 104,600 | 200 |
| Closed vacancies | /job/[id]/ | 41,000 | 47,300 | 200 and one line: "This job has closed" |
| Live vacancies | /job/[id]/ | 6,800 | 30,700 (9,800 with ?apply=1) | 200 |
| Old vacancy addresses | /vacancy.php?id= | 5,600 seen | 11,200 | 301 to /job/[id], then 301 to add the slash |
| Role and area landing pages | /jobs/[role]/[area]/ | 1,200 | 9,400 | 200 |
| Employer pages | /employers/[name]/ | 900 | 3,100 | 200 |
| Articles and other pages | various | 300 | 7,200 | 200 |
Live vacancies not requested at all in the week: 2,450 of 6,800, of which 1,980 were posted in the last ten days. Landing pages list vacancies by relevance, so new ones usually start on page three or later. The vacancy sitemap is rebuilt every Sunday and lists 9,300 URLs, 2,900 of them closed. Search Console page indexing shows 18,700 URLs as soft 404, every example a closed vacancy. Closed vacancies stay linked from each employer's "past vacancies" list. About one closed role in eight is reposted under a new ID.
Parameters found in the crawl, the logs and Search Console examples:
| Parameter | Distinct URLs in the logs | Developer's note | Internal links carry it |
|---|---|---|---|
role, area | 3,400 | Narrow search to one role or area | Filter links |
contract, pay_min, radius | 28,900 | Narrow results | Filter links |
sort | 22,600 | Newest, pay or distance | Sort links, and pagination links keep it |
page | 9,100 | Pagination | Pagination links |
q | 6,300 | Keyword search | "Popular searches" footer block |
utm_source, utm_medium, utm_campaign | 1,850 | Campaign tags | Homepage banner; job alert emails |
ref | 740 | Partner widget on employers' own sites | None |
apply | 6,200 | Opens the application form over the page | Apply button, written as a link |
sid | 410 | Session from the previous platform | None |
Next quarter the product team plans saved searches, which will add a parameter.
Evidence before rules
Verify the crawler first, because impostors inflate the templates they scrape. Read requests per URL beside totals, since a healthy total can hide unvisited pages, and compare the crawled set with the sitemap. Then give rules to classes of parameter, and send every finding with an example URL and the change expected next.
Task
- Decide which requests count and say what would have gone wrong without that step.
- Read the verified requests by template, as totals and per URL, and list the waste and the neglect, each with an example URL and the change you expect in the logs.
- Classify every parameter, give each class its rule and mechanism, and set how any parameter added later is treated, including the planned saved searches.
- Decide what happens to a vacancy when it closes, and when the same role is reposted.
- Write the requirement for the search template with acceptance checks, test URLs and owners.
- Say when and how the fixes are measured again, and who keeps the evidence current.
Worked answer
Which requests count. The requests from unverified addresses are excluded, and the verified total then sits close to Crawl Stats, which covers the whole property. Left in, the scraper's traffic would have made the search tool look even worse than it is and sent the first fixes after a problem that is not the crawler's.
Waste and neglect. The search tool takes the largest share of verified crawling, and closed vacancies the next, returning success with nothing on them, which Search Console already reports as Soft 404A page that returns a success status code but shows an error, empty or not-found message. pages. Redirect chains from the old platform add a further slice. On a site that changes this fast, Crawl BudgetHow much a search engine crawls a site. is a real constraint, and the neglect list shows the cost: new vacancies are the pages the board most needs crawled, and they sit deep in landing pages and appear in a sitemap that is stale for most of the week.
Ivelsorn Jobs: the arithmetic behind the two lists
Shares of verified requests: search tool 49%, closed vacancies 22%, live vacancies 14%, old addresses 5%, landing
pages 4%. Requests per URL in the week: landing pages 7.8, live vacancies 4.5, employer pages 3.4, closed vacancies
1.2. The live average hides that 36% of live vacancies were not requested at all, and 81% of those were posted in
the last ten days. Example tickets: /jobs/search?sort=pay&contract=seasonal&page=7, a closed vacancy at
/job/48213/, and /vacancy.php?id=48213 reaching the vacancy in two hops.
The register. Rules go to classes, with the lightest mechanism that works:
| Class | Parameters | Rule | Mechanism |
|---|---|---|---|
| Narrows, page exists | role, area | Resolve to the landing page | Filter links point at the clean path; a search URL with only these redirects to it |
| Narrows, no page planned | contract, pay_min, radius | Not linked; canonical to the page they narrow | Form controls, not anchors |
| Reorders or changes layout | sort, apply (apply only overlays the form) | Not linked; canonical to the default view | Buttons; the form opens without changing the address |
| Paginates | page | Crawlable, self-canonical, linked on its own | Pagination links drop the sort |
| Searches | q | Out of the index and the crawl | Footer block replaced by landing page links, then a robots rule |
| Tracks | utm_ tags, ref | Never on internal links; crawlable; canonical to the clean URL | Banner rebuilt without tags |
| Holds a session | sid | Out of URLs | Redirect that strips it |
Keyword search URLs lose their footer links first and the robots rule follows. The log tutorial's warning applies: a robots.txtA file at a site's root telling crawlers which URLs they may and may not fetch. rule stops fetching but does not by itself remove a URL already indexed, so check the indexing report before and after. Tracking tags are not blocked, so tagged links can still pass on their canonical. Old vacancy addresses redirect in a single hop. No new parameter ships without a row, and until classified it is treated as tracking; saved searches get theirs now, as a session held in the visitor's account rather than the URL. The product manager for search owns the register, and the SEO lead checks it against the logs every month.
Closed vacancies. The requirements guide asks every template to say what happens when a page is deleted: a redirect to the new URL or a not-found response. The log tutorial counts pages that return success but say nothing as soft errors. So a closed vacancy leaves the sitemap and the employer page the same day and stops answering with success, and a reposted role redirects permanently to its new vacancy. The expired pages guide sets out the wider lifecycle options for listings.
The search template requirement. It follows the requirements guide's steps, and its opening line settles most arguments: the search tool serves visitors, and landing pages own the queries.
Ivelsorn Jobs: the search template requirement, shortened
| Area | Requirement | Acceptance check |
|---|---|---|
| URLs | Role and area alone redirect to /jobs/[role]/[area]/; other parameters lower case in register order | /jobs/search?area=x&role=chef returns one 301 to the landing page |
| Links | Filters, sort and apply are buttons or form fields; pagination links carry only page | A crawl of staging finds no anchor with sort, apply, q, utm_ or sid |
| Sitemap | Rebuilt within an hour of any vacancy opening or closing; live vacancies only | No closed vacancy in the sitemap the morning after closing |
| Test URLs | Twelve, including ?sort=pay&page=2, ?SID=abc, a closed vacancy, a reposted vacancy and an unknown parameter | Each has an expected status, canonical and directive |
| Owners | The developer automates the URL and link checks, the tester works through the twelve test URLs, the SEO lead reads the staging output | Nothing ships while any of the three checks is open |
Measuring again. After the fixes have been live a few weeks, repeat the analysis on a window of the same length. The expected change is search tool requests down, live vacancy requests per URL up, fewer new vacancies unvisited, soft errors falling and single-hop redirects. The log review then joins the monthly routine, beside the crawling checks in the technical SEO audit checklist.
Common mistakes
- Counting every request that claims to be a crawler. The scraper inflates the very template you are about to fix.
- Reading totals only. The live vacancy total looks healthy while many of its pages go unvisited.
- A rule for every parameter. Nobody maintains a row of rules per key; a handful of classes stays current.
- Leaving closed vacancies at success with one line of text. That is the soft error the report already shows.
- Findings without example URLs or tests. Developers fix what they can reproduce and check.
The crawl evidence checkpoint puts the same calls to an auction house.
