What robots.txt means
robots.txt sits at yourdomain.com/robots.txt and lists user agents with allow and disallow rules. Compliant crawlers read it before fetching.
It controls crawling, not indexing. A disallowed page can still appear in results if other sites link to it, because the engine knows it exists without having fetched it. Use noindex for exclusion from results.
Why it matters for AI search
For AI search it's the single highest-leverage file on your site. One misplaced disallow can remove you from an entire platform, and it's frequently added years ago by someone no longer at the company.
It's also incomplete as a control. WAF rules and bot-management tools block crawlers independently of robots.txt, which is why testing with an actual fetch matters more than reading the file.
See it in action
Building an AI-aware robots.txt line by line.
Explicit beats implicit.
A blanket disallow with no named AI agents is how most sites end up invisible. Naming each one makes the decision visible and auditable.
How to get it right
Auditing robots.txt properly
- List every AI user agent explicitly rather than relying on a wildcard
- Check for legacy blanket disallows added during earlier AI caution
- Test with a real fetch as each user agent — the file isn't the whole story
- Check WAF and bot-management rules separately; they block independently
- Include your sitemap directive and keep it accurate
Common questions
Does robots.txt stop pages being indexed?
No. It controls crawling, not indexing. A disallowed URL can still appear in results if it's linked from elsewhere.
Use a noindex meta tag or header to keep something out of results — but the page must be crawlable for the directive to be read.
Is a wildcard disallow enough to block AI crawlers?
Compliant crawlers respect a wildcard, but naming agents explicitly is safer and much easier to audit later.
It also lets you make different decisions for training versus search retrieval, which a wildcard can't express.
Can robots.txt block a crawler that ignores it?
No. It's a convention, not enforcement. Well-behaved crawlers comply; others don't.
For genuine blocking you need server-level or WAF rules — which is also how crawlers get blocked accidentally.
Related concepts
These come up alongside robots.txt constantly.