OpenAI has introduced a clearer system for identifying and controlling the bots that access websites. These bots appear in server logs the same way Googlebot, Bingbot, or Pinterestbot do. The difference now is simple transparency. You see the name. You know its purpose. You decide whether to allow it or block it.
Here is the complete list from the official documentation, rewritten in easy language.
1. OpenAI-User
This is the main crawling bot used when ChatGPT or other OpenAI tools fetch pages on behalf of a user.
Example. When a user says, “Visit this link and summarise the page,” this is the bot that fetches the content.
Purpose
User-triggered browsing.
Acts only when a human request requires loading a webpage.
How to block
User-agent: OpenAI-User
Disallow: /
2. OpenAI-Assistant
This bot fetches content as part of AI agent workflows.
When an AI agent needs external information to complete a task, this bot fetches it.
Purpose
Autonomous or semi-autonomous agent behaviours.
How to block
User-agent: OpenAI-Assistant
Disallow: /
3. OpenAI-SearchIndexer
This bot indexes content for OpenAI’s search capabilities.
This is not general web crawling like Google Search.
It is used only to support OpenAI’s internal semantic search tools.
Purpose
Indexing pages for retrieval and ranking inside OpenAI systems.
How to block
User-agent: OpenAI-SearchIndexer
Disallow: /
4. GPTBot
This is the older, well-known bot used earlier for training data collection.
It is still part of the bot list but now operates under stricter rules and transparency.
Purpose
Collects publicly available data for model improvement, with restrictions.
How to block
User-agent: GPTBot
Disallow: /
5. The Full Robots.txt Table (Reconstructed Cleanly)
| Bot Name | What It Does | robots.txt Rule To Block |
|---|---|---|
| OpenAI-User | Fetches pages when a user asks ChatGPT to browse | User-agent: OpenAI-User + Disallow: / |
| OpenAI-Assistant | Allows AI agents to fetch data for tasks | User-agent: OpenAI-Assistant + Disallow: / |
| OpenAI-SearchIndexer | Indexes pages for OpenAI search features | User-agent: OpenAI-SearchIndexer + Disallow: / |
| GPTBot | Crawls public data for model training | User-agent: GPTBot + Disallow: / |
Allowing them works the same way by replacing Disallow with Allow.
Why This Matters For Website Owners
Until now, many AI systems crawled the web without clear disclosure.
Now:
• Every bot has a name.
• Every bot has a purpose.
• Every bot can be allowed or blocked.
• Every action is controlled by the website owner.
This gives publishers, newsrooms, and creators real control over how their content is used.
How To Block All OpenAI Bots At Once
If you want a single rule that blocks all four bots:
User-agent: OpenAI-User
Disallow: /
User-agent: OpenAI-Assistant
Disallow: /
User-agent: OpenAI-SearchIndexer
Disallow: /
User-agent: GPTBot
Disallow: /
If You Want To Allow All OpenAI Bots
User-agent: OpenAI-User
Allow: /
User-agent: OpenAI-Assistant
Allow: /
User-agent: OpenAI-SearchIndexer
Allow: /
User-agent: GPTBot
Allow: /
Discover more from Rudra Kasturi
Subscribe to get the latest posts sent to your email.