Crawlhive — free SEO audit, site crawler & technical SEO checker

Start with one question: can this page be indexed?

Before anything else, settle whether the page is eligible to appear in search at all. Everything else is wasted effort if the answer is no, and the answer is not visible by looking at the page.

Three separate mechanisms can keep a page out of the index, and only one of them is in the HTML you can view:

• The robots meta tag. The one every SEO extension reads. A noindex here is a direct instruction not to index.
• The X-Robots-Tag response header. Does exactly the same job, but travels in the HTTP response — invisible in the page source, and routinely missed.
• The canonical tag. Not a block, but a page whose canonical points elsewhere is asking not to be indexed under its own URL.

A noindex that moves from staging to production is the most expensive bug in SEO for exactly this reason: the page looks completely normal, traffic drains over weeks, and nobody thinks to check the response headers.

Then robots.txt — and read it properly

robots.txt controls crawling rather than indexing, and its rules are less obvious than they look. When several rules match the same URL, the longest match wins, and when two rules of equal length disagree, Allow beats Disallow. Wildcards make this harder to reason about by eye.

A blocked page cannot be crawled, so a noindex on it will never be seen — which is why blocking a page in robots.txt is the wrong way to remove it from search results. The two mechanisms solve different problems and are regularly confused.

Check the title and description in pixels, not characters

The familiar advice — keep titles to 50-60 characters — is a rough proxy for something Google measures differently. Results are truncated by pixel width, and characters are not equal: a title in capitals or full of wide letters runs out far sooner than the character count suggests, while a title of narrow letters survives well past 60.

Measuring the real width in the font Google actually uses is the only way to know where the cut falls. It is a small detail that decides whether your call to action survives into the search result or gets replaced by an ellipsis.

Read the heading structure, not just the H1

The common check is "does the page have exactly one H1", and that is worth knowing, but the more useful signal is the outline. Headings that skip levels — an H2 followed directly by an H4 — usually mean the page is being styled rather than structured, and the document no longer describes its own hierarchy.

Duplicate headings matter too. Two identical H2s often mean a template is repeating a section that should have been merged, and it is a reliable sign that the page grew by accretion rather than design.

Structured data: "found" is not the same as "valid"

Most tools tell you a schema type was detected. That is the easy part. What determines whether you get a rich result is whether the required properties are present — a Product with no offers produces no rich result at all, however correct the markup looks.

So the useful output is not "Product found" but "Product found, but no offers, so no rich result". The difference is the entire point of checking.

Compare the raw HTML with the rendered page

A crawler's first pass reads the HTML as it arrives from the server. If your title, your H1 or your canonical only appear after JavaScript runs, they are not there for that first read. Google does render pages eventually, but rendering is a second, slower, less certain step.

Comparing what the server sent against what the browser ended up with tells you which parts of your page depend on that second pass. Anything critical to indexing belongs in the first one.

Now zoom out: the problems one page cannot show

Everything above is about a single URL. The findings that change rankings most often are structural, and they only appear when you look at the site as a graph:

• Duplicate titles. Two pages with the same title compete with each other. It is the most common finding in any site audit, and it is invisible from either page alone.
• Click depth. How many clicks a page sits from your entry point. Pages four or five levels deep get crawled rarely, no matter how good they are.
• Incoming internal links. A page with one internal link, or none, is telling search engines it does not matter. This is the quiet answer to most "why doesn't this page rank" questions.
• Orphan pages. In the sitemap, but no link anywhere leads to them. Both halves of that check exist already — the sitemap and the crawl — they just have to be compared.
• Internal links landing on redirects. Every visitor and every crawler pays for an extra hop that one href edit would remove.

Speed, in the terms Google actually uses

Core Web Vitals narrowed a vague topic into three measurable things, and two of them are usually fixable without touching the server. LCP is how long the largest visible element takes to paint — most often a hero image or a heading held up by a render-blocking stylesheet. CLS is how much the layout jumps while loading, nearly always because an image or an ad has no reserved space.

TTFB is the one that points at the server, and it is worth separating from the rest: a slow first byte makes every other metric worse and no amount of front-end work will fix it.

One specific waste is worth checking on every site: images served far larger than the box they are drawn in. A 128-pixel file displayed at 24 pixels is bytes the visitor downloads and never sees. Screen density matters here — on a 2× display a 400-pixel file in a 200-pixel box is correct, not wasteful — and vectors are exempt, since an SVG has no resolution to waste.

Where an audit fits with everything else

A technical audit answers "can this page compete", not "is this page good". Once indexing and structure are clean, the work moves to the things a crawler cannot judge: whether the page actually answers the query, and whether anyone links to it. Our guide to checking on-page SEO in the browser walks through the page-level half in more detail.

Two related checks often come up in the same session. Knowing which platform and technologies a site runs on explains a surprising number of technical findings — a CMS that ships duplicate tag pages, a framework that renders headings only after JavaScript. And when the audit points at speed rather than structure, the fixes usually live in the render-blocking files and oversized images the Speed panel lists.

How often, and what to do with the output

A full site audit is a quarterly job for most sites. The indexability check is not: run it after every deploy that touches templates, and on any page that suddenly loses traffic. It takes seconds and it catches the failure mode that costs the most.

When you do run the full pass, fix in this order: things that block indexing, then structural problems (duplicates, orphans, depth), then on-page details, then speed. That order is not a matter of taste — a page that cannot be indexed gains nothing from a faster LCP.