As of 30 October 2026, the crawlers to allow for AI citations are the search crawlers: OAI-SearchBot for ChatGPT, PerplexityBot for Perplexity, Claude-SearchBot for Claude and Googlebot for Google's AI features. No vendor page we read says a training crawler such as GPTBot or ClaudeBot must be allowed to be cited.
- OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers (OpenAI, Overview of OpenAI Crawlers, read 30 October 2026).
- OpenAI says a robots.txt update can take about 24 hours to take effect for search, and Perplexity says up to 24 hours (OpenAI and Perplexity, crawler documentation, read 30 October 2026).
- Anthropic says disabling Claude-SearchBot may reduce a site's visibility and accuracy in user search results (Anthropic, Claude Help Center, read 30 October 2026).
- Google says Google-Extended does not impact a site's inclusion in Google Search (Google, Google's common crawlers, read 30 October 2026).
- Penlid's methodology reports two figures as significantly different only when their 95% intervals do not overlap (Penlid, Methodology, read 30 October 2026).
Which crawlers does each AI vendor run?
The table sorts by job the search, training and user-request bots that OpenAI, Perplexity, Anthropic, Google and Microsoft name on their own pages; it leaves out other bots, such as OpenAI's OAI-AdsBot. The sorting follows Cloudflare's documentation, which divides AI bots into Search, for crawlers that collect or index your content to answer questions about it later
; Agent, for automated activity acting in real time on a person's behalf
; and Training, for crawlers taking your content to train or fine-tune a model
. We label the Agent column User request, matching how the vendors describe those bots.
| Vendor | Search | Training | User request |
|---|---|---|---|
| OpenAI | OAI-SearchBot | GPTBot | ChatGPT-User |
| Perplexity | PerplexityBot | None named | Perplexity-User |
| Anthropic | Claude-SearchBot | ClaudeBot | Claude-User |
| Googlebot | Google-Extended, a product token for Gemini training and grounding | Not covered here | |
| Microsoft | Bingbot, Bing's standard crawler | None named | None named |
Which crawler should you allow so each engine can cite your site?
Allow the search crawler each vendor names for its own search results: OAI-SearchBot, PerplexityBot and Claude-SearchBot, plus Googlebot for Google. OpenAI says we recommend allowing OAI-SearchBot in your site's robots.txt file
and that sites opted out of it will not be shown in ChatGPT search answers, though can still appear as navigational links
. Perplexity says we recommend allowing PerplexityBot in your site's robots.txt file
. Anthropic says disabling Claude-SearchBot may reduce your site's visibility and accuracy in user search results
.
To be eligible to be shown as a supporting link in AI Overviews or AI Mode, Google says a page must be indexed and eligible to be shown in Google Search with a snippet
and adds There are no additional technical requirements.
Google names robots.txt directives for Googlebot as the control for how a site is crawled for Search, so our reading is that Googlebot is the crawler to allow. Search Console has a setting that includes or excludes a site from Search generative AI features, and Google says Sites that opt out will not receive traffic or impressions from our generative AI features.
The Generative AI performance report counts impressions in those features.
Do you have to allow GPTBot, ClaudeBot or Google-Extended to be cited?
No: none of the vendor pages we read says GPTBot, ClaudeBot or Google-Extended must be allowed for a site to be cited in search results. OpenAI says a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot
, and Google says Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.
Anthropic says that when a site restricts ClaudeBot, it signals that the site's future materials should be excluded from our AI model training datasets
. For OpenAI and Anthropic, our reading is that training access is a decision about model training, not about search citations. Google's common crawlers page adds that Google-Extended also manages whether content may be used for grounding in Gemini Apps, so for Gemini it is not only a training choice.
Does robots.txt control the crawlers that act when a user asks?
Not always: for ChatGPT-User, OpenAI says robots.txt rules may not apply
, and for Perplexity-User, Perplexity says this fetcher generally ignores robots.txt rules
, while Anthropic says its bots honor industry-standard robots.txt directives. OpenAI adds that ChatGPT-User is not used to determine whether content may appear in Search.
Anthropic says disabling Claude-User may reduce your site's visibility for user-directed web search
.
Cloudflare's documentation says robots.txt compliance is voluntary.
Our reasoning: a Disallow line is a request, so where a vendor says its user-request fetcher may ignore it, refusing that fetcher needs a server or CDN rule instead, and welcoming one means checking that no such rule blocks it.
Can a firewall or bot setting block a crawler that robots.txt allows?
Yes: a firewall or bot-management rule can stop a crawler that your robots.txt allows, and Perplexity warns you may need to explicitly allow Perplexity's bots to ensure they can access your content.
Perplexity's page adds Cloudflare's Bot Fight Mode can't be skipped with WAF custom rules.
OpenAI and Perplexity both recommend allowing requests from their published IP ranges, and Anthropic links a list of its IP addresses, saying If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.
Comparing your server logs with each vendor's list shows whether its requests arrive (our reasoning).
Cloudflare's own documentation says it will set updated defaults for new domains on 15 September 2026, under which bots classified as Training or as Agent will be blocked on pages that display ads, and Search will remain allowed
. The page still used the future tense when we read it on 30 October 2026, so we cannot say whether those defaults are live.
How do you check that a crawler change made a difference?
Change one thing, note the date, wait at least 24 hours, then ask the same unbranded questions (ones that do not name your brand) repeatedly and compare the citation rate before and after, each with its 95% interval. OpenAI says a robots.txt update can take about 24 hours to take effect for search and Perplexity says up to 24 hours; neither page says how long crawling your pages takes afterward, so treat 24 hours as a minimum (our reasoning).
One answer is not a measurement (checking whether ChatGPT recommends your brand explains why), so our methodology samples each tracked prompt repeatedly and records the model version with every sample. A rate that spans a model change is two measurements, not one, so split your windows at a version change (why AI visibility changes when you change nothing). Report each window as k answers citing you out of n, with a Wilson 95% interval (how to measure AI visibility), and call two rates different only when their intervals do not overlap. The rule is conservative: a pair it does not name is not shown to differ, which is not the same as shown to be the same.
Example (illustrative numbers)
These numbers are invented and are not data from any site. A site removes a robots.txt rule that blocked OAI-SearchBot from its guides folder. It tracks 12 unbranded buying questions and asks each one 5 times per window, so a window holds 12 × 5 = 60 answers. An answer counts when it cites any page on the site, and every answer came from the same model version. The before window is the 7 days before the change, and the after window starts 72 hours later (our margin, three times the 24 hours OpenAI mentions).
| Window | Answers citing the site | Answers | Rate | 95% interval |
|---|---|---|---|---|
| Before | 3 | 60 | 5.0% | 1.7% to 13.7% |
| After | 12 | 60 | 20.0% | 11.8% to 31.8% |
| Before, if 120 answers | 6 | 120 | 5.0% | 2.3% to 10.5% |
| After, if 120 answers | 24 | 120 | 20.0% | 13.8% to 28.0% |
The intervals. The Wilson interval for k of n answers uses p = k ÷ n, z = 1.96 and z² = 3.8416. For the before window, p = 3 ÷ 60 = 0.05 and z² ÷ n = 3.8416 ÷ 60 = 0.0640. The center is (0.05 + 0.0640 ÷ 2) ÷ (1 + 0.0640) = 0.0820 ÷ 1.0640 = 0.0771. The half-width is z × √(p × (1 − p) ÷ n + z² ÷ (4 × n²)) ÷ 1.0640 = 1.96 × √(0.05 × 0.95 ÷ 60 + 3.8416 ÷ 14,400) ÷ 1.0640 = 1.96 × √0.001058 ÷ 1.0640 = 1.96 × 0.03253 ÷ 1.0640 = 0.0599. So the interval runs from 0.0771 − 0.0599 = 0.0172, or 1.7%, to 0.0771 + 0.0599 = 0.1370, or 13.7%. The after window, with p = 12 ÷ 60 = 0.20, gives 11.8% to 31.8% by the same steps.
The verdict. The first interval ends at 13.7% and the second begins at 11.8%, so they overlap by 1.9 points. Under the rule in our methodology (the dashboard applies it to pairs of engines; here it compares two windows) the verdict is no significant change, even though the rate rose from 5.0% to 20.0%. With the same rates in 120 answers a window, the intervals are 2.3% to 10.5% and 13.8% to 28.0%; 13.8% is above 10.5%, so they do not overlap and the rule would name the difference. Either way the comparison is timing evidence, not proof that the robots.txt change caused the rise; the limits below list what else can move a rate.
What this cannot tell you
A before and after comparison cannot show that a crawler change caused a citation change, and our reading is that allowing a crawler is a condition for appearing, not a promise of a citation: OpenAI's own wording is To help ensure your site appears in search results
. We observe what an engine answered, not why, and a change in a citation rate may come from your content, a model update, a change in the engine's retrieval or a competitor (what can and cannot be attributed to a content change).
- It cannot tell you what the vendors do after 30 October 2026, the day we read their pages, so read each page again before you act.
- It cannot tell you whether blocking a training crawler changes how a model describes your brand, because none of the pages we read covers that.
- It cannot cover every route into an answer: OpenAI's help page says ChatGPT search
sometimes partners with other search providers
, and the pages we read do not say which crawlers they use (see Bing Webmaster Tools and ChatGPT search). - The interval measures sampling noise for the questions you asked. It does not cover your choice of questions, personalization or model changes.
Questions people also ask
What is the difference between AI crawlers and search crawlers?
A search crawler collects pages so an engine can find them and show them in answers. A training crawler collects pages to build or improve a model, and a third kind fetches a single page because a person asked for it. OpenAI, Perplexity and Anthropic give these jobs separate bot names, which is why a site can allow one job and refuse another.
What is an AI crawler?
An AI crawler is an automated program, run by an AI company, that requests pages from your site. The vendors we read describe three purposes: gathering content that may be used to train models, finding and indexing pages for AI search, and fetching a page for a user in real time. Each bot has its own name, such as GPTBot or PerplexityBot, so one robots.txt file can treat them differently.
How do you keep your site out of AI searches?
You opt out of each engine separately. OpenAI says sites opted out of OAI-SearchBot are not shown in ChatGPT search answers, Perplexity recommends allowing PerplexityBot to appear in its results, and Anthropic says disabling Claude-SearchBot may lower visibility in user search results, so disallowing each search crawler in robots.txt is the matching step (our reading). For Google's AI features, use the Search Console setting that excludes a site from Search generative AI features. Opting out may cost you citations in that engine, and a robots.txt line does not bind every fetcher.
Do AI agent crawlers follow robots.txt?
It depends on the vendor, going by their own pages. OpenAI says the rules may not apply when a user starts the action, and Perplexity says its user fetcher generally ignores them, while Anthropic says its bots honor standard directives. Cloudflare calls compliance voluntary. Because the answer differs by vendor, check which requests reach your server rather than relying on one file.
Sources
- Overview of OpenAI Crawlers, OpenAI, read 30 October 2026.
To help ensure your site appears in search results, we recommend allowing OAI-SearchBot in your site's robots.txt file and allowing requests from our published IP ranges below.
Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.
a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot
Because these actions are initiated by a user, robots.txt rules may not apply.
ChatGPT-User is not used to determine whether content may appear in Search.
- Perplexity Crawlers, Perplexity documentation, Perplexity, read 30 October 2026.
To ensure your site appears in search results, we recommend allowing PerplexityBot in your site's robots.txt file and permitting requests from our published IP ranges listed below.
Since a user requested the fetch, this fetcher generally ignores robots.txt rules.
you may need to explicitly allow Perplexity's bots to ensure they can access your content.
Cloudflare's Bot Fight Mode can't be skipped with WAF custom rules.
- Does Anthropic crawl data from the web, and how can site owners block the crawler?, Claude Help Center, Anthropic, read 30 October 2026.
Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.
Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search.
When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets.
If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.
- Google's common crawlers, Google Crawling Infrastructure documentation, Google, read 30 October 2026.
Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.
- AI features and your website, Google Search Central documentation, Google, read 30 October 2026.
To be eligible to be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet, fulfilling the Search technical requirements.
There are no additional technical requirements.
- New opportunities, control and insights for website owners, The Keyword (blog.google), Google, read 30 October 2026.
Sites that opt out will not receive traffic or impressions from our generative AI features.
- Block AI Bots, Cloudflare bot solutions documentation, Cloudflare, read 30 October 2026.
Search: crawlers that collect or index your content to answer questions about it later.
Agent: automated activity acting in real time on a person's behalf, such as chat fetch bots and browser-use agents.
Training: crawlers taking your content to train or fine-tune a model, including mixed-purpose crawlers that are used both for Training and for Search.
bots classified as Training or as Agent will be blocked on pages that display ads, and Search will remain allowed.
- robots.txt setting, Cloudflare bot solutions documentation, Cloudflare, read 30 October 2026.
robots.txt compliance is voluntary.
- Which Crawlers Does Bing Use?, Bing Webmaster Tools Help, Microsoft, read 30 October 2026.
- Searching the web with ChatGPT, OpenAI Help Center, OpenAI, read 30 October 2026.
ChatGPT search sometimes partners with other search providers.
- Methodology: how we measure, and what we can't, Penlid, read 30 October 2026.