Is Your Website Blocking AI Search? How to Check robots.txt and Your Firewall
Open yoursite.com/robots.txt and look for Disallow rules under OAI-SearchBot, PerplexityBot, Claude-SearchBot or the catch-all "User-agent: *" group, then ask whoever runs your hosting whether a firewall or bot-protection setting blocks those crawlers. A clean robots.txt is not enough on its own: OpenAI and Perplexity both tell site owners to allow their published IP ranges at the host or firewall as well.
Which crawler does what
| Crawler | Company | What it is for | If you block it |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search | OpenAI says your site will not be shown in ChatGPT search answers, though it can still appear as a navigational link |
| GPTBot | OpenAI | Content that may be used to train OpenAI's models | Signals your content should not be used for training; separate from search |
| ChatGPT-User | OpenAI | Visits a page when a user asks ChatGPT | OpenAI says robots.txt rules may not apply to these user-initiated visits |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search; Perplexity says it is not used for model training | Your site cannot be a Perplexity search source |
| Perplexity-User | Perplexity | Fetches a page when a user asks | Perplexity says it generally ignores robots.txt because a user asked |
| Claude-SearchBot | Anthropic | Indexes content to improve Claude's search results | Anthropic says this may reduce your visibility in Claude's search results |
| Claude-User | Anthropic | Fetches a page when a Claude user asks | Anthropic says this may reduce your visibility for user-directed search |
| ClaudeBot | Anthropic | Content that could contribute to training Anthropic's models | Signals future content should be excluded from training |
| Google-Extended | Controls use of your content for Gemini training and grounding | Google says it does not affect inclusion or ranking in Google Search | |
| Applebot-Extended | Apple | Controls use of Applebot data to train Apple's models | Apple says pages that disallow it can still appear in its search results |
The pattern: each company now separates search crawlers from training crawlers. You can block training and still allow search.
How to check, step by step
- Open your robots.txt in a browser. Look for any group that names one of the search crawlers above with "Disallow: /", and check what the "User-agent: *" group disallows, because it applies to every crawler not named elsewhere.
- Ask your web host or developer whether a firewall, content delivery network or bot-protection product is on, and whether it blocks or challenges AI crawlers. OpenAI's help page asks site owners to confirm their host or network "allows traffic from OpenAI's published searchbot IP addresses."
- If you use Cloudflare, note Perplexity's warning that Cloudflare's Bot Fight Mode "can't be skipped with WAF custom rules" and can still challenge its crawlers.
- After any change, allow time. OpenAI says it can take about 24 hours for its systems to adjust to a robots.txt change.
What we recommend for a typical small business site
Allow the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot) and the user-request fetchers. Decide on the training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) as a separate business choice; blocking them does not remove you from those companies' search features. Our own robots.txt allows all of them.
How Schkovl can help
Crawler access is one of the first checks in our Gap Analysis and part of our SEO & Answer Engine Optimization work. Ask us to check your site.
Sources
- OpenAI: Overview of OpenAI crawlers
- OpenAI Help Center: Searching the web with ChatGPT
- Perplexity docs: Perplexity crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Google Search Central: Google's common crawlers (Google-Extended)
- Apple: About Applebot