We audit a lot of sites. When a brand tells us they've invested eighteen months in content and see nothing in AI search, the cause is almost never the content. It's usually one line in a file nobody has opened since 2023.
This is the most common conversation we have. A marketing team has done the work — published consistently, hired good writers, built genuine expertise into the pages — and their brand still doesn't appear when someone asks ChatGPT or Perplexity about their category. The natural conclusion is that AI search doesn't work for them, or that the content isn't good enough.
Then we run a crawl and find GPTBot has been receiving a 403 the entire time.
The four layers, in the order they break
AI visibility fails at four distinct layers, and they're worth diagnosing separately because the fix for each is completely different. Most teams jump straight to the fourth and wonder why nothing moves.
Four failures, and only the fourth is about content.
Content investment behind a blocked crawler produces nothing. This is why we benchmark before recommending anything — the honest answer to 'what should we do' depends entirely on which layer is broken.
Layer one: retrieval
Whether AI crawlers can reach your pages at all. The failure is usually a robots.txt directive added during the initial wave of AI-crawler caution, or a WAF rule blocking unfamiliar user agents by default. Neither gets reviewed, because neither is anyone's job.
This is the cheapest thing in AI SEO to fix and the most expensive to leave in place, because everything downstream assumes the page can be fetched.
Layer two: rendering
Whether crawlers that can reach you actually receive content. This is where most modern sites fail, and it's harder to spot because every response is a 200.
If your site renders client-side, the HTML response is a shell and the content arrives via JavaScript. Googlebot renders JavaScript reasonably well, so teams check Google, see everything indexed, and assume all crawlers behave the same way. Several major AI crawlers don't execute JavaScript at all.
A site can rank in the top three on Google and be completely invisible to ChatGPT and Perplexity, for exactly this reason.
Layer three: entity resolution
Whether models can work out who you are. Your brand name is a string until something resolves it to a specific organisation with defined attributes. Without Organization schema, a sameAs graph, and consistent naming, your content is attributed to an ambiguous entity — which carries far less weight than an identified one.
Layer four: corroboration
Whether anyone else says what you say. Models weight independent agreement heavily. A claim appearing only on your own domain reads as an assertion; the same claim across three credible independent sources reads as fact.
This is the slowest layer, usually two to four months, because it depends on third parties publishing and being re-crawled.
Why the order matters more than the effort
The sequencing isn't arbitrary. Each layer depends on the one before it. Citation building aimed at an entity models can't resolve mostly evaporates. Beautiful answer-first content behind a blocked crawler earns nothing at all.
Same total effort, sequenced correctly.
The left column is what happens when teams start with content, which is the intuitive place to start. The right column is what happens when the technical layers are fixed first.
What to do this week
- Open your robots.txt and look for GPTBot, ClaudeBot, PerplexityBot and Google-Extended. If any are disallowed, decide whether that was deliberate.
- Fetch one of your key pages with a crawler user-agent string and compare the response size to what a browser sees. A 4kb versus 80kb gap is a render parity failure.
- Ask three different models to describe your company. Note what they get wrong — that's your entity and model memory problem, made visible.
- Count how many independent sources state your main positioning claim. If the answer is one, corroboration is your ceiling.
None of that requires an agency. If you'd rather have it done properly with a benchmarked prompt set and a written roadmap, that's what our AI visibility audit is for — but the four checks above will tell you which layer you're stuck on in about an hour.
The uncomfortable version
A meaningful share of the brands convinced that AI search is overhyped are brands that have been technically excluded from it. They concluded the channel doesn't work when what actually happened is they never entered it.
That's a good problem to have, because it's the cheapest one on the list.