What GPTBot means
GPTBot is one of several OpenAI user agents. It's documented as collecting data used to improve models. OAI-SearchBot is separate and serves search and citation, while ChatGPT-User handles fetches triggered directly by a user's request.
The distinction matters commercially: you can allow search retrieval while disallowing training crawls, because they're controlled independently in robots.txt.
Why it matters for AI search
Blocking GPTBot is one of the most common unintentional AI visibility failures we find. It's often added years ago as a blanket AI-crawler rule, or by a security tool nobody associated with marketing.
Whether to allow it is a genuine business decision rather than a technical default — but it should be a decision, not an inheritance.
See it in action
Three OpenAI agents, three different jobs. Here's what each one is for.
Allowing search while disallowing training is a valid position.
Many publishers take exactly this route: opt out of training, stay eligible for citation. Most brands selling a product benefit from allowing all three.
How to get it right
Configuring OpenAI crawlers
- Check robots.txt for all three agents rather than assuming one rule covers them
- Check your WAF and bot-management tool separately — they block by default more often than robots.txt does
- Decide training and search access independently; they're different commercial questions
- Verify with an actual fetch as the user agent, not just by reading the file
- Re-check after deploys; security rule changes reintroduce blocks silently
Common questions
Should we allow GPTBot?
It's a business decision. Allowing it is a precondition for being cited when ChatGPT searches. Blocking protects content from being used for training without compensation.
Most brands selling a product or service benefit from being findable. Publishers monetising content directly have a stronger case for restricting.
Does blocking GPTBot remove us from ChatGPT?
Not entirely. GPTBot governs training crawls; OAI-SearchBot governs search retrieval. Blocking only GPTBot leaves you eligible for citation when ChatGPT searches.
Blocking both removes you from the retrieval pathway completely.
How do I check if we're blocking it?
Read your robots.txt for a GPTBot user-agent block, then fetch a page with the GPTBot user-agent string and check the status code and response size.
A 403, a challenge page, or a suspiciously small response all indicate a block.
Related concepts
These come up alongside GPTBot constantly.