Most AI visibility failures are technical, not editorial. We fix the crawler access, rendering, and architecture problems that stop AI systems from reading your site at all — before anyone spends a dollar on content.
We audit a lot of sites, and the pattern repeats: a brand invests heavily in content for eighteen months, sees no AI visibility, and concludes AI search doesn't work for their category. Then a crawl test shows GPTBot has been receiving a 403 the entire time, or that every page renders as an empty shell because the content only appears after JavaScript executes.
This is the cheapest problem in AI SEO to fix and the most expensive to ignore, because everything else depends on it. Entity work, content restructuring, and citation building all assume the page can actually be retrieved. When it can't, that investment produces nothing.
Technical SEO for AI search overlaps heavily with classic technical SEO but isn't identical. Different user agents, different rendering tolerances, different crawl behaviour. We test per agent rather than assuming that because Googlebot is fine, everything else must be.
When a site fails crawler rendering tests, there are two routes. The right one depends on your stack and how much engineering capacity you have.
Best for: Sites on modern JS frameworks where content exists but isn't served to bots.
Best for: Sites where rendering workarounds keep breaking, or where a replatform was already on the roadmap.
Four layers, tested independently. Most sites pass the first and fail the second, which is why browser-based testing gives false confidence.
Whether each AI user agent receives a 200 rather than a 403, 429, or a challenge page. robots.txt directives, WAF rules, rate limiting, and bot-management tooling all block AI crawlers routinely and silently.
Whether the content a bot receives matches what a browser sees. This is where most modern sites fail — the HTML response is a shell and the actual content arrives via JavaScript the crawler never executes.
Whether your URL structure, internal linking, and site hierarchy make topical relationships legible, and whether crawl budget is being consumed by low-value URLs.
Whether pages respond fast enough under bot traffic. Slow or timing-out responses get retried less and retrieved less, independent of Core Web Vitals scores.
Diagnosis, developer-ready specification, and implementation support — plus the ongoing monitoring that catches when a deploy silently breaks retrieval again.
Crawling your site as GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended and others, comparing what each receives against browser output.
Auditing every directive, meta robots tag, and header for rules that block or restrict AI crawlers — often added years ago by someone no longer at the company.
Specifying and implementing server-side rendering, static generation, or dynamic rendering so bots receive complete content without executing JavaScript.
Restructuring URL hierarchy, internal linking, and canonicalisation so topical relationships are explicit and crawl budget goes to pages that matter.
Improving response times, caching strategy, and payload size — measured under bot conditions, not just lab-tested Core Web Vitals scores.
Ongoing crawler access monitoring, because the most common failure mode is a deploy or WAF rule change silently re-blocking a crawler months later.
The failure pattern is usually predictable from the technology stack. Here's what we find most often.
The most common AI visibility blocker we encounter. Client-side rendering serves an empty shell to any crawler that doesn't execute JavaScript, which includes several major AI agents.
Usually renders fine but suffers from directive chaos — multiple SEO and security plugins writing conflicting robots rules and headers.
Faceted navigation and parameter URLs consume crawl budget at enormous scale, so product pages get retrieved infrequently and stale data persists.
Enterprise WAF and bot-management tooling frequently blocks AI crawlers by default, and nobody notices because the rules were configured for security rather than visibility.
Technical work is the fastest-moving service we offer, because fixes register on the next crawl cycle rather than depending on third parties.
We crawl as every major AI user agent and diff what each receives against browser-rendered output, per template rather than per page.
Every failure is traced to its cause — directive, WAF rule, rendering path, or architecture — and written up as a developer-ready specification.
Your team or ours implements, then we re-crawl to verify the fix actually worked rather than assuming the deploy did what it was supposed to.
Ongoing crawler access monitoring, since the most common failure is a later deploy silently reintroducing the block.
There's almost no tension here. Faster, cleaner, better-architected sites serve search engines and language models identically — the only genuinely AI-specific work is the user agent testing and directive auditing.
The questions clients ask us most before starting. If yours isn't here, ask us directly on a consultation call.
Because the failure is total rather than gradual. In classic search, a slow or partially-rendered page might rank a bit lower. In AI search, a blocked crawler or empty render means you're absent from the surface entirely — there's no degraded position to fall back to.
It's also the most common problem we find. A large share of brands with poor AI visibility have a purely technical cause, not a content one.
Check your robots.txt for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended directives, and check whether your WAF or bot-management tool blocks unknown agents by default. Then fetch a page as one of those agents and compare the response to what a browser shows.
That last step catches the rendering failures, which are far more common than outright blocks and much easier to miss.
It's a genuine business decision, not a technical default. Blocking them protects content from being used without attribution; allowing them is a precondition for being cited at all.
Most brands selling something benefit from being found. Publishers monetising content directly have a real case for restricting. We'll walk through the tradeoff rather than assuming — and if you decide to block, that's a legitimate choice, but you should make it deliberately.
Only if it renders client-side. React itself is fine; serving an empty HTML shell that populates via JavaScript is the problem, because several AI crawlers don't execute JavaScript.
Server-side rendering, static generation, or pre-rendering all solve it. Next.js, Nuxt, and similar frameworks support this natively — it's usually a configuration decision rather than a rewrite.
Access fixes can register within days to a couple of weeks, on the next crawl cycle. Rendering changes typically take two to six weeks to be fully reflected as pages are re-crawled.
It's the fastest-moving work we do, which is why we sequence it first in nearly every engagement.
Both, depending on what you need. We write developer-ready specifications your team can implement, and we can implement directly where you'd rather we did and access allows.
For WordPress and most common stacks we can usually handle implementation. For complex enterprise environments, we typically specify and support your engineering team rather than working in production ourselves.
We implement it because it's trivial and harmless, but we won't oversell it. Adoption is genuinely uncertain and it currently does far less than robots.txt directives and rendering strategy.
Treat it as a cheap hedge, not a strategy. Any agency positioning llms.txt as a core service line is dressing up a fifteen-minute task.
Where it's the primary need, we scope it as a fixed-fee project, typically in the low five figures for diagnosis plus specification. Implementation cost depends on whether we're doing it or your team is.
Within a broader retainer it's included as the first phase, since almost nothing else works until it's resolved.
We'll crawl your site as every major AI user agent and show you, per template, exactly how much of your content each one gets. It's usually a lot less than you'd expect.
No commitment · 45 minutes · Immediate value
Two fields. We’ll crawl your site as GPTBot, ClaudeBot and PerplexityBot, benchmark a sample of your category’s prompts, and send you what we find.
No commitment, no sales sequence. If we’re not a fit, we’ll say so.