Noindex and robots.txt do different jobs
`noindex` controls page indexing. robots.txt controls crawler access to a URL. If crawling is blocked completely, a bot may not see the meta directive intended to remove the page. Diagnosis therefore starts with the actual response, HTML and rules for a specific user agent.
Where the blocker can hide
Check meta robots in server HTML, X-Robots-Tag response headers, canonical, robots.txt, sitemap, language versions, CDN rules and middleware. Also verify that production did not inherit a staging or CMS-template setting.
Search and training are separate choices
OpenAI documents OAI-SearchBot for surfacing public content in ChatGPT search and GPTBot for potential model training. Anthropic also publishes crawler controls. A business can allow search use while restricting other use cases.
What the technical test proves
It proves whether a page can be fetched, read and indexed in the relevant layer. It does not prove that a particular system will retrieve the page, cite it or recommend the brand. Those claims require observed-answer testing.
What to do after the fix
Update the sitemap, verify canonical and language alternates, request recrawling in Search Console, allow processing time and capture a new baseline. Do not compare before and after runs if prompts, language or search conditions changed.
Save the HTML, response headers, robots.txt and test date before changing anything. Repeat the same technical checks and prompts afterwards. This separates a real effect from answer variance and prevents a content project being sold where the problem was one header.
Primary sources
These links point to platform documentation and research supporting the claims in this article.