A 141 page crawl of a 137 page site, and what it found

A link crawl left our own site at 97 out of 100. This is what a crawl of 141 pages on a 137 URL site actually returns, what the one broken link turned out to be, and why the number is worth more than the score.

The crawler is the one behind the broken link checker. It starts at the sitemap, follows every internal link it finds, and requests each outbound one once, at twenty pages a minute, until a page cap or a clock stops it.

What it measured

Two numbers are doing the work there. 141 pages from a sitemap of 137 means the crawl found four pages that the sitemap does not list, which is normal and worth seeing. And one broken link in 141 pages is not a good score, it is a single fault with a name attached.

The broken link was a promise with nothing behind it

The fault was one URL, /support/articles/, returning 404. It was linked from the server check page, on a button that says read what each finding means.

The reason is worth understanding, because it is a shape that recurs. Every support article is one document served by a single viewer, addressed as /support/articles/<category>/<slug>/. The viewer needs both halves to know which article to show, so the bare categoryless path had nothing to resolve and fell through to the 404.

Nothing was wrong with the article it promised. The button simply pointed one level too high.

It now points at the guide it meant, the server check walkthrough, and the bare path redirects to the support index rather than answering 404. Both are better than a green tick over a dead end.

Why 97 rather than 100

The score weights a broken internal link heavily, because an internal dead end wastes a crawl budget and strands a reader. One fault out of 141 pages is one per cent of the pages and a three point deduction, which is a reasonable shape for a rule: it is loud enough to notice and it does not bury the ninety nine per cent that passed.

What it cannot tell you is whether the pages that passed are any good. A site of 141 empty pages and no links scores 100.

What to do with this if you run a site

Run it on a schedule rather than when something feels wrong, because the useful number is the difference between two runs. A site at 97 that was 100 last month has a new fault and it is one page old. A site at 97 that has been 97 for a year has a habit.

The other thing a crawl gives you is the list of pages nothing links to. Those are the pages that exist, answer 200, and are invisible to a reader who did not arrive from somewhere else. That list is usually the interesting one.

The website audit runs a shorter version of this crawl alongside the performance and security checks, up to twenty pages, and reports what a crawler found on each one.