What Is the Meta Robots Tag?
The meta robots tag is placed in a page's <head> and provides instructions to crawlers. Common values include noindex, nofollow, noarchive, nosnippet, and noimageindex. Multiple values can be combined: <meta name="robots" content="noindex, nofollow">.
Why Meta Robots Matters
Meta robots gives you page-level control over search engine behavior that robots.txt cannot provide. While robots.txt controls crawling at the URL pattern level, meta robots controls indexing and behavior on a per-page basis.
How to Use Meta Robots
Add the tag to your page's <head>. Use noindex for pages that should not appear in search. Use nofollow on pages with untrusted outbound links. The default (no tag) is index, follow. For non-HTML files, use the X-Robots-Tag HTTP header.
Crawl Your Site Like a Search Engine
CrawlBeast finds SEO issues — broken links, redirect chains, missing tags, and indexation problems — before Google does.
Try CrawlBeast Free