AI Crawlers Fail on Sites That Look Healthy in a Browser: Notes from Hundreds of Probes

There is a class of network failure that produces no error message, no monitoring alert, and no unhappy user, and it quietly decides which businesses exist in AI answers. A site serves browsers flawlessly, passes every uptime check, and returns 403 to GPTBot, ClaudeBot, and PerplexityBot at the network edge. The owner never learns, because nothing they watch ever shows it. The engines stop retrieving the site, and the answers about its category get built from whatever sources remain reachable.

Across several hundred domain checks this year, this condition appeared repeatedly. The pattern in where these failures occur is worth documenting for a technical audience because most originate within infrastructure that site owners and technical teams directly manage.

The failure taxonomy

Default-on bot management

The most common case by a wide margin. Several CDN and security vendors now ship AI-crawler blocking as a toggle that arrives enabled, and rule updates can enable it on existing configurations. Nobody at the affected company decided to block AI retrieval; a default did. One probed fintech was preparing a product launch while its edge configuration returned 403 to every major AI crawler.

TLS fingerprinting

WAF policies that fingerprint client handshakes drop non-browser clients before any HTTP status exists, which makes the failure invisible even to status-code monitoring. The symptom is a connection that dies mid-handshake for exactly one class of visitor.

Rendering budgets

A 200 response whose meaningful content only materializes after client-side JavaScript is, to a crawler on a time budget, an empty page served successfully. The site is reachable and effectively blank.

Agent-keyed caching

Cache rules keyed to browser user agents can serve crawlers stale or divergent content indefinitely, a subtler corruption than blocking and harder to notice.

Which bots matter, and the distinction most sites miss

Training and retrieval are separate decisions wearing similar user agents. Blocking GPTBot keeps content out of future training runs, a legitimate policy choice; blocking OAI-SearchBot removes the site from ChatGPT's live, cited answers, which is rarely what anyone intends. A large share of probed sites blocked both while clearly wanting only the first, usually via an inherited rule that treats every AI user agent as one category.

The five-minute diagnostic

for UA in "GPTBot/1.1" "OAI-SearchBot/1.0" "ClaudeBot/1.0" "PerplexityBot/1.0"; do
  printf "%-22s" "$UA"
  curl -s -o /dev/null -w "%{http_code}\n" \
    -A "Mozilla/5.0 (compatible; $UA)" https://yourdomain.com/
done
curl -s -o /dev/null -w "browser control:  %{http_code}\n" \
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)" https://yourdomain.com/

A 200 for the browser control and anything else for the crawlers is the whole diagnosis. Run it against the homepage and the three most important pages, because bot rules are frequently path-scoped, and repeat it monthly, because vendor defaults can change between releases. An AI visibility checker can also be used to test crawler reachability and identify changes that may affect how a site appears in AI-generated answers. Honeyb is one example of a tool that provides this type of visibility check.

Notes on verification, for the sceptical

Two refinements harden the diagnostic. First, spoofed user agents: the curl test uses self-declared identity, which any scraper can fake, so serious bot management verifies crawlers by reverse DNS against the operators' published IP ranges rather than by string matching, and your allowlists should too. Second, robots.txt interactions: a bot allowed at the edge but disallowed in robots.txt will connect and then decline to index, which produces a confusingly healthy curl result on a still-invisible site. Check both layers, in that order, and log the results somewhere a future engineer will find them before they re-enable the vendor default that started the problem.

Why the stakes keep rising

The answers being built from reachable sources are volatile: SparkToro measured the same question changing its answer roughly 70% of the time. That volatility means slots in AI answers are perpetually being re-decided, and a site the crawlers cannot reach forfeits every one of those re-decisions silently. For the people who run the infrastructure, this is a rare situation where a five-minute check and usually a one-line allowlist change protect a channel the marketing team cannot even see failing. Few fixes in this field have a better effort-to-consequence ratio.