The robots meta tag is a small piece of HTML with large effects. Google’s specification (last updated 24 March 2026 as we read it) is the reference for what each rule does. This post turns it into a cheat sheet and a safe rollout routine.
Best for: SEO leads, developers and CMS owners who add or audit page-level rules.
What Google says
- Settings can be set on HTML pages with a meta tag or in an HTTP header, and at text level with the
data-nosnippetattribute. They can be read and followed only if crawlers are allowed to access the page. - The
namevaluerobotsapplies to all crawlers. Google supports two user agent tokens in the meta tag:googlebot(text results) andgooglebot-news(news results). Other values are ignored. - Google does not enforce placement in the
headand will respect meta tags in the body as well. - For non-HTML files such as PDFs, video and images, use the
X-Robots-Tagresponse header. Any rule usable in a meta tag can be used there, and a user agent can be named before the rules. - Conflicts: the more restrictive rule applies. For example
nosnippetbeatsmax-snippet:50. - Rules:
noindex(do not show the page),nofollow,none(noindex plus nofollow),nosnippet,indexifembedded(allows indexing of an embedded page despite noindex, and only works with noindex),max-snippet,max-image-preview(none, standard or large),max-video-preview,notranslate,noimageindexandunavailable_after. - AI features:
nosnippetapplies to all forms of results at Google, including AI Overviews and AI Mode, and prevents the content from being used as a direct input for them.max-snippetlimits how much may be used as direct input, though it does not override content a publisher has separately permitted, such as in-page structured data or a licence.
Cheat sheet (our summary)
| You want | Rule | Watch out for |
|---|---|---|
| Keep a page out of results | noindex | The page must be crawlable so Google sees it |
| Retire a page on a date | unavailable_after: [date] | Use a standard date format |
| No text snippet anywhere | nosnippet | Also limits AI features |
| Cap the snippet length | max-snippet:[n] | 0 equals nosnippet, -1 leaves it to Google |
| Allow big image previews | max-image-preview:large | Helps Discover images, see our checklist |
| Limit video previews | max-video-preview:[n] | 0 allows at most a static image |
| Keep a PDF out | X-Robots-Tag: noindex | Needs a server header |
What to do: step by step
| Step | Owner | Time | How to check | Done when |
|---|---|---|---|---|
| 1. Inventory current rules | SEO | 1 hour | Crawl and export meta robots and X-Robots-Tag values | A table of rule per template |
| 2. Remove accidental rules | SEO, developer | 1 hour | Look for site-wide noindex or nosnippet | None on pages meant to be found |
| 3. Check crawl access | SEO | 20 min | Is any noindex page also blocked in robots.txt? | No conflicts |
| 4. Set intended rules | Developer | 2 hours | View source and headers | Output matches the plan |
| 5. Test five pages | SEO | 30 min | URL Inspection shows the rule | Samples confirm it |
| 6. Record AI decisions | Leadership, SEO | 30 min | Which pages may be used by AI features? | A written decision |
Common mistakes
- Blocking a page in robots.txt and expecting its noindex to be read. See noindex vs robots.txt.
- Adding
nosnippetfor a legal reason without knowing it limits AI features. - Forgetting the header for non-HTML files.
- Assuming other search engines treat every rule the same way. Google notes they may not.
Related: featured snippets and opting out, control what you share and Discover images. Rules do not guarantee a result.
