HTTP Header Checker

Inspect the response headers behind any URL. The SEO-relevant ones are called out first, including the X-Robots-Tag that can keep a page out of Google without leaving any trace in the HTML.

URLs to check (one per line, up to 25)

Paste a full URL to check that page, or a bare domain to check its home page. Every response header comes back, and the ones that decide how the page is indexed, cached, and compressed are called out first with what each value means.

When a page disappears and the HTML looks fine

The usual diagnosis path for a deindexed page runs through the meta robots tag, the canonical, and robots.txt. When all three come back clean and the page is still gone, the answer is almost always in the response headers, because a header can carry the same directives and nothing in the page source reveals it.

This happens most often when a rule was added at the CDN or load balancer rather than in the application, which is a common way to keep a staging environment out of search. The rule outlives the environment, and a path pattern that once matched only staging starts matching production paths after a restructure.

An X-Robots-Tag is read one line at a time, and a line may name the crawler it addresses before its directives. A response carrying all on one line and otherbot: noindex on the next is indexed by Google normally, so this tool reports a noindex as blocking only where a Google crawler reads it, and names the crawler where one is named.

The Link header is worth the same attention. A canonical declared there competes with the one in the HTML, and when they disagree the outcome is not something you can reason about from the page alone.

One thing this check cannot do for you is arrive as Googlebot. A rule at the CDN may serve a header to Google and to nobody else, which is exactly the rule that hides best. When the headers here look right and the page is still missing, the URL Inspection tool in Search Console fetches it as Googlebot and settles the question.

Frequently Asked Questions

What is an X-Robots-Tag and why does it matter?

It is a way of sending robots directives in the HTTP response rather than in the HTML. A header carrying noindex removes a page from search exactly as a meta tag would, but nothing in the page source shows it, so it is the hardest deindexing cause to find. If a page has vanished from Google and its HTML looks correct, check this header first.

Which response headers affect SEO?

Three carry real weight, and two of them only for certain values. X-Robots-Tag controls indexing, though only through the lines a Google crawler reads. Link matters when it declares a canonical or an hreflang alternate, and not when it carries a preload hint. Vary matters when the response changes by user agent, language, or cookie, and not when it names only Accept-Encoding, which is compression rather than search. The rest are performance and security concerns.

Why does a noindex header sometimes not count?

An X-Robots-Tag may open with the name of the crawler it addresses, and that name scopes every directive on the same line. So a response sending all on one line and otherbot: noindex on the next is indexed by Google exactly as normal, because the noindex was never addressed to a Google crawler. This tool reads each line the response sent rather than the lines run together, and reports a directive as blocking only where a Google crawler reads it. A line addressed to another crawler is still shown, with a note saying who it speaks to.

What does the "Affects search" label mean?

It marks a header whose value on this page changes how search engines treat it, and the line underneath says why. The label reads the value rather than the header name, so Vary: User-Agent is flagged while Vary: Accept-Encoding is not, and an X-Robots-Tag addressed to a crawler other than Google is not. Every other header still shows what it does, without the label.

Which security headers should a response send?

Four carry their weight today. Strict-Transport-Security tells the browser to reach the host over https on every later visit. Content-Security-Policy restricts what the page may load, and its frame-ancestors directive is the rule current browsers read to decide who may frame the page, which is the job X-Frame-Options once held alone; you should send both while you still support browsers that predate the directive. Referrer-Policy decides how much of the current URL travels with an outgoing request, and Permissions-Policy decides which browser features the page may use. Two headers no longer earn their place: X-XSS-Protection, whose filter every current browser removed, and Feature-Policy, which Permissions-Policy replaced.

What does Cache-Control do for SEO?

Nothing directly, and a good deal indirectly. Caching headers decide how much of a repeat visit is served from disk rather than fetched again, which shows up in the Core Web Vitals that Google does measure. A response with no Cache-Control leaves every cache guessing.

Why is compression worth checking?

Because HTML compresses extremely well and the saving is free. A page served without gzip or brotli typically transfers three times the bytes it needs to, on every request, for every visitor. If content-encoding is absent from the table above, that is usually one line of server configuration away from being fixed.

Which crawler do these checks arrive as?

Every check is sent from a RankYak server under RankYak's own crawler user agent, RankYakTools. A site that varies its response by user agent may answer Googlebot with headers it does not send here, and a rule that serves an X-Robots-Tag to Googlebot alone is the case worth knowing about. So when a page looks correct in this tool and is still missing from Google, fetch it as Googlebot with the URL Inspection tool in Search Console, which is the only fetch Google treats as its own.

Can I check several URLs at once?

Yes, paste up to 25 URLs. The export includes every header found across all of them as columns, which makes it straightforward to compare a template against itself across pages, or a staging environment against production.