SEOSTUDIO™ · SEO & WEBMASTER

How do I check meta robots and X-Robots-Tag together?

Paste robots meta elements and X-Robots-Tag lines to see their scopes, indexing restrictions, and simple conflicts.

Compare robots directives

This browser-only check reads meta and X-Robots-Tag lines from the same response. It does not assess search results, time-based rules, or a crawler’s full syntax.

What problem does this tool solve?

A CDN or server header can add a restriction that the page template never shows. Compare the directives side by side before changing the release.

Ten practical questions answered

Why can Chrome show a page whose response says noindex?

The directive is an instruction for indexing consumers, not an access control. The page can still return 200 and render normally in a browser.

Where should I look for a CDN-level restriction?

Find the final document request in the network panel and copy the exact response line. Compare the CDN and origin rules if your team can inspect both deployments.

How should I check a served PDF separately from its summary page?

Use the PDF resource's own response header. Keep the summary page's metadata and the PDF response as two separate checks because they are separate URLs.

What does a crawler prefix before the colon mean?

It scopes the provided header to the specified bot. Enter separate header lines when reviewing different agents and read the provider's supported names.

What should I do with an index versus noindex conflict?

Trace each row to its controlling template or server rule. Google documents using the more restrictive directive when rules conflict. Your pasted sample is a starting point for the live check.

How does a crawl block affect a metadata check?

A crawler typically needs permission to retrieve the URL before it can read directives in the response. Review the relevant robots.txt rules separately.

What does none mean for an HTML document?

For Google, none is equivalent to noindex and nofollow. This checker highlights the indexing restriction when the supplied value is none.

Can I paste response headers from different URLs together?

Keep the pasted headers tied to one final resource and one request. Mixing two responses can manufacture a conflict that no user or crawler receives.

Where do I verify a template change that runs after client hydration?

Inspect the initial document and the rendered head separately. Confirm the Google-specific behavior with a supported inspection tool because this parser only sees the text you supply.

Can a successful check confirm that a URL was removed from search?

Check the live response and the processing status using an authorized search console. The pasted directives are evidence of declared behavior, while indexing status is a separate observation.

How it works, with an example

1. Copy relevant meta elements from the final document. 2. Paste each X-Robots-Tag response header on its own line. 3. Review which scope owns each directive. 4. Retrieve a new response after any config change.

Practical example

A PDF returned by a content server includes X-Robots-Tag: noindex. Its landing page has a different response. Paste the PDF header on its own and review the general scope before changing the PDF delivery rule.

Additional use cases

  • Review an index restriction on a PDF.
  • Compare a bot-specific header and a general meta element.
  • Record where a post-release noindex originated.

Technical reference: Google Search Central · Robots meta and X-Robots-Tag

Frequently asked questions

Does this send HTTP requests?

It reads the text you paste locally. Retrieve the actual response in a separate check.

Should I paste the entire page source?

Paste one quoted meta element per line, without the surrounding head or body.

Will it modify headers or tags for me?

Change your template, server, or CDN configuration separately and verify the new deployment.

Are HTTP header names case-sensitive in the form?

The check accepts case variations of X-Robots-Tag. Paste each full header on its own line.

Does the checker interpret dates and snippet limits?

The listing helps locate the source of a directive. Review date and snippet values separately against the crawler documentation.

Related tools for your next step

Review indexing across your website

If the pattern spans many URLs, we can review the affected templates and HTTP responses with you.

Request a free consultation →

SEOSTUDIO editorial team · Updated