What Log File Analysis means
Every request to your server is recorded with a timestamp, URL, user agent and status code. That's the only unfiltered record of what crawlers actually did.
Everything else — Search Console, crawl simulators, third-party tools — is sampled, delayed, or inferred. Logs are ground truth.
Why it matters for AI search
For AI search it's the only way to confirm whether GPTBot, ClaudeBot and PerplexityBot are visiting at all, how often, and what they receive.
It also reveals crawl budget waste with a precision nothing else offers — you can see exactly how many fetches went to facet URLs instead of products.
See it in action
What server logs reveal that nothing else does.
GPTBot never came. PerplexityBot was blocked 1,240 times.
Neither of those facts appears in any SEO tool. The 403s point at a WAF rule, and the total absence suggests robots.txt — two different fixes for two different problems.
How to get it right
Doing log analysis usefully
- Get access to raw server or CDN logs covering at least 30 days
- Segment by user agent to see each crawler's behaviour separately
- Check status codes per agent; 403s and 429s indicate blocking or rate limits
- Map requests to URL type to find where crawl budget actually goes
- Re-run after fixes to verify the change actually landed
Common questions
Where do I get log files?
From your hosting control panel, server, or CDN. Cloudflare and similar providers expose them, sometimes on higher plans.
How much log data do I need?
At least 30 days for meaningful patterns. Large sites can work with less; smaller sites may need longer to accumulate enough crawler visits.
Isn't Search Console enough?
For Google, partly — it's sampled and delayed. For AI crawlers it tells you nothing at all, which is the gap logs fill.
Related concepts
These come up alongside Log File Analysis constantly.