What Is Noindex?
Noindex is a directive that tells search engines to exclude a page from their index. It can be implemented via a <meta name="robots" content="noindex"> tag in the HTML head or an X-Robots-Tag: noindex HTTP header. Unlike robots.txt (which blocks crawling), noindex allows crawling but prevents indexing.
Why Noindex Matters
Use noindex for pages that should be accessible to users but not appear in search results: internal search pages, thank-you pages, admin pages, or thin content. Noindex is the only reliable way to prevent indexing — robots.txt Disallow does not prevent indexing if other sites link to the blocked URLs.
How to Implement Noindex
Add <meta name="robots" content="noindex"> to the page's <head>. For non-HTML resources, use the X-Robots-Tag: noindex HTTP header. Do not block noindexed pages in robots.txt. Audit with CrawlBeast to ensure important pages are not accidentally noindexed.
📖 Related Article: Diagnosing Indexation Issues — Read our in-depth guide for practical examples and advanced techniques.
Crawl Your Site Like a Search Engine
CrawlBeast finds SEO issues — broken links, redirect chains, missing tags, and indexation problems — before Google does.
Try CrawlBeast Free