← All articles
AI search11 min read

AI crawlers: which to allow, which to block

Your rules for AI crawlers decide if ChatGPT, Claude, Perplexity and Google can read and cite your site. Which bots train, which search, what a block costs.

FameScan EditorialResearch & practical guides
Visitor ID badges on lanyards, one of them green: an AI crawler identifies itself by its name, and your access rules read exactly that name.
The short answer

What matters first.

  • AI crawlers fetch your pages for one of three jobs: training models, building the search index an assistant cites from, or fetching a page live because a user asked.
  • Blocking a search or user-request crawler such as OAI-SearchBot or PerplexityBot can cost you the citation. A training crawler such as GPTBot is a separate decision; OpenAI writes: “Each setting is independent of the others.”
  • robots.txt is only one layer. A CDN or firewall rule can turn away a crawler your robots.txt allows. FameScan’s AI readiness audit checks both for your homepage.

AI crawlers are the bots AI companies send to read websites, and each one announces itself by name, so your robots.txt can treat it on its own. Let the search and user-request crawlers in if you want ChatGPT, Claude, Perplexity and Google to be able to cite your pages, and decide about training crawlers such as GPTBot separately. Then check that your firewall agrees with your robots.txt.

Run an AI readiness audit

FameScan’s AI readiness audit reads your robots.txt for each AI crawler, sorts the crawlers into training, search and user request, and loads your homepage with the user agents of five of them to show whether a firewall turns such requests away. FameScan also measures how your brand appears in answers from ChatGPT, Perplexity, Gemini, Claude and Google AI Overviews, so the same workspace shows whether your pages get cited afterwards. We opened every vendor page linked in this guide on November 10, 2026.

What is an AI crawler?

An AI crawler is a bot that fetches web pages for an AI company for one of three jobs: collecting training data, building the search index an assistant cites from, or fetching a page live because a user asked. Most vendors run more than one. OpenAI alone documents GPTBot, OAI-SearchBot and ChatGPT-User next to further bots, each for a different purpose.

A crawler’s name is its user agent, the label it sends with every request, and a robots.txt rule addresses exactly that label. “User-agent: GPTBot” speaks to GPTBot and to nobody else. So the useful question is never “do we allow AI?” but “which of these names may read which pages?”.

Some names are switches without a crawler of their own. Google says Google-Extended has no separate HTTP user agent string, and Apple says Applebot-Extended does not crawl webpages. Both decide how content that Googlebot or Applebot already fetched may be used for AI: Google-Extended covers Gemini training and grounding, Applebot-Extended covers training. For how OpenAI’s crawlers fit into diagnosing ChatGPT, see our guide to ChatGPT visibility.

Training, search, user request: three jobs

The job decides what a block costs you. FameScan’s audit sorts crawlers into the same three groups for that reason: Training, Search and User request.

  • Training. The crawler collects content that may be used to train future models. A block is a statement about whether your content should be used that way. OpenAI, Google and Apple each keep this switch separate from their search.
  • Search. The crawler builds the index an assistant pulls sources from when it answers. A block can take your pages out of those answers. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.
  • User request. The crawler fetches one page because a person asked about it, for example by pasting a link. It does not crawl the web on its own. Anthropic says disabling Claude-User prevents Claude from retrieving your content in response to a user query.
OpenAI
GPTBot
Training
future models
OAI-SearchBot
Search
ChatGPT-User
User request
citable
Three of OpenAI’s crawlers and where each one can lead. Source: OpenAI crawler page, opened November 10, 2026.

Only the last two can put your page into an answer as a cited source. That is why the audit fails its check “AI search assistants allowed” when a major search or user agent is blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Googlebot or Bingbot. A blocked training crawler shows up under “AI training crawlers” instead and never lowers the score.

Which AI crawlers visit your site?

The table lists the AI crawlers you are most likely to meet in a robots.txt file, with what each vendor says about a block. Every cell comes from the vendor’s own page, opened on November 10, 2026. Vendors update these pages, so check the source before you change a rule.

Crawler and jobIf you block it, per the vendor
OpenAI
GPTBotTrainingSignals that your content should not be used to train OpenAI’s foundation models
OAI-SearchBotSearchNot shown in ChatGPT search answers; navigational links can still appear
ChatGPT-UserUser requestrobots.txt rules may not apply, because a user starts the visit
Anthropic
ClaudeBotTrainingSignals that future material should be excluded from Anthropic’s training datasets
Claude-SearchBotSearchMay reduce your visibility and accuracy in user search results
Claude-UserUser requestClaude cannot retrieve your page for a user’s question
Perplexity
PerplexityBotSearchPerplexity recommends allowing it so your site appears in its search results
Perplexity-UserUser requestGenerally ignores robots.txt, since a user requested the fetch
Google
GooglebotSearchAffects Google Search; AI Overviews and AI Mode link only to pages indexed there
Google-ExtendedTraining and grounding, a token onlyNo Gemini training and no grounding in Gemini Apps; no effect on inclusion or ranking in Search
Apple
ApplebotSearch in Spotlight, Siri, SafariApple says allowing it lets your content appear in search results; without its own group it follows your Googlebot group
Applebot-ExtendedTraining, a token onlyNo training of Apple’s foundation models; pages can still appear in search results
Common Crawl
CCBotOpen archiveNo crawling for Common Crawl’s open repository, which anyone can access and analyze
Meta
Meta-ExternalAgentTraining and indexingMeta names training foundation AI models and indexing content as its uses
Meta-ExternalFetcherUser requestMeta says it may bypass robots.txt rules

FameScan’s audit reads your rules for 22 AI crawlers: the ones above plus Bingbot, Amazonbot, DuckAssistBot and a few more. It files CCBot under Training. Whatever list you work from, keep it dated, because new names keep appearing.

Should you block AI crawlers?

Block training crawlers if you do not want your content used for model training, and keep search and user-request crawlers open if you want your pages to be eligible as sources. Those are two separate decisions, and the vendors built them that way. OpenAI’s crawler page puts it in one line: “Each setting is independent of the others.”

A mention is not a citation, and a citation is not a recommendation. They are three different wins, and crawler access mainly touches the second. Your name can still come up in answers through other people’s pages while your own page is blocked. What you lose is the link to your page, the one source whose wording you control.

So decide per job, not per vendor:

  • Search and user request: allow, unless a page must stay out of AI answers entirely. A blocked OAI-SearchBot, Claude-SearchBot or PerplexityBot is the most expensive mistake on this page, because it is silent: nothing breaks on your site.
  • Training: your call. A publisher with paid content might block it. A software company that wants future models to describe it accurately might allow it. Both are legitimate.
  • Unknown names in your logs: look them up on the vendor’s page before you block them. A crawler you have never heard of may feed an assistant your buyers use.

No setting is right for every site, and a crawl setting is a technical choice, not legal advice. If licensing or copyright drives the decision, bring in the person who owns those questions.

Which Google bots matter for AI answers?

For AI answers in Google, two names matter: Googlebot and Google-Extended. Googlebot crawls for Google Search, and AI Overviews and AI Mode build on Search. Google’s guide to AI features says a page must be indexed and eligible to be shown in Search with a snippet to appear as a supporting link, and that there are no additional technical requirements. So there is no separate AI Overviews crawler to allow.

Google-Extended works differently. Google calls it a standalone product token for managing whether crawled content may be used to train future generations of Gemini models and to ground Gemini Apps answers at prompt time, and says it does not affect inclusion or ranking in Google Search. Blocking Google-Extended is therefore a decision about Gemini, not about AI Overviews. Google’s crawler page lists further crawlers for other products; for AI answers in Search and for Gemini, these two are the names to check.

To limit what Google shows from a page in Search and its AI features, Google points to the snippet controls nosnippet, data-nosnippet and max-snippet, and to noindex. The audit flags noindex and nosnippet on your homepage under “Not blocked from indexing”, and noindex on the inner pages it samples. How to read what AI Overviews then show for your questions is a separate job, covered in our guide to AI Overview tracking.

Apple follows the same pattern. Applebot crawls for search features in Spotlight, Siri and Safari, while Applebot-Extended only controls training use. Apple adds that if your robots.txt names Googlebot but not Applebot, Applebot follows your Googlebot rules. A rule you wrote for Google alone can therefore reach Apple’s search too.

Is Cloudflare blocking AI crawlers?

Not by default on new domains, according to Cloudflare’s documentation, but a single setting on your zone can, and your robots.txt will not show it. Cloudflare’s page on AI Crawler and Agent access policies, last updated November 2, 2026, sets an action for three behaviors: Search, Agent and Training. For new domains, all three default to “Allow (do not block)”.

The options include blocking across the whole zone and blocking only on pages with ads. Training also offers Disallow, which asks crawlers through robots.txt to stay away. In AI Crawl Control you allow or block single crawlers, and a block creates or updates a WAF custom rule on your zone that can answer with 403 Forbidden. Someone on your team may have switched one of these on long before anyone asked about AI answers.

That is the trap: robots.txt says yes, the firewall says no, and the crawler never sees your page. It cuts the other way too. Cloudflare reminds site owners that robots.txt directives are voluntary and crawlers can ignore them, and with Bot Preference Sync its Disallow setting also blocks AI training crawlers at its edge.

PerplexityBot
robots.txt
Allowed
CDN
403
Page
unread
Answer
uncited
Example, not measured data: robots.txt allows PerplexityBot, a CDN rule answers 403.

The same holds for any CDN, WAF or bot protection, not only Cloudflare. Perplexity’s crawler page says you may need to explicitly allow its bots in your firewall and recommends matching each bot’s user agent with its published IP ranges. Once access works, our guide to Perplexity SEO shows how to check which sources Perplexity cites.

The live visit in the audit is built for exactly this. It reloads your homepage with the user agents of OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot and GPTBot and compares each response with a normal visit: “Gets the page”, “Blocked”, “Bot challenge” or “Stripped page”. The requests come from FameScan’s servers, so they show whether your firewall turns away a request that presents itself as that crawler.

A robots.txt for AI crawlers, line by line

The following example is invented: a robots.txt for a fictional company that wants to be cited in AI answers and keeps its content out of model training. Copy the structure, not the choices. The training block is one possible decision among several.

# Search and user request: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# Training: blocked here
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# Everyone else (Googlebot, Bingbot …)
User-agent: *
Disallow: /admin/

Sitemap: https://example.com/map.xml
  • The first group lists the search and user-request crawlers one per line and allows everything. Several User-agent lines in a row share the rules beneath them.
  • The second group blocks training. Google-Extended and Applebot-Extended sit here even though they never visit, because these are the names Google and Apple read. Google-Extended also covers grounding in Gemini Apps, so a company that wants Gemini to draw on its pages may leave that line out.
  • The third group, the star, covers every crawler without a group of its own, including Googlebot. A crawler that finds its own name ignores the star group, so a Disallow you add there does not reach the crawlers named above.
  • The Sitemap line applies to everyone and helps search crawlers find your inner pages.

Two limits remain. User-triggered fetchers may not follow robots.txt the way index crawlers do: OpenAI says the rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Meta says Meta-ExternalFetcher may bypass them. And changes take time. OpenAI says it can take about 24 hours after a robots.txt update for its search systems to adjust.

If your site runs on Wix, you edit these rules in Wix’s Robots.txt Editor; our guide to Wix SEO tools shows where. For the other steps that make a page worth citing, see the GEO checklist.

Check access in FameScan, then watch the citations

Check access first, then let the measurement show whether citations follow. In FameScan, open AI readiness and click “Run audit”. The audit makes real requests to your site: robots.txt, the sitemap, visits with AI crawler user agents and up to five inner pages from your sitemap and navigation. It uses no Answer Credits.

The fix happens on Tallyform’s side: its developer relaxes the CDN rule. The marketing lead then runs the audit again and sees “Gets the page” for PerplexityBot.

Access is the precondition; the answers are the evidence. Over the following weeks, she filters the dashboards to Perplexity and watches three things on the same buyer questions:

  • Sources, Domains: whether Tallyform’s domain appears among the cited sources. Every stored answer opens with its engine, country, time and sources. How FameScan works explains how every answer is stored.
  • Citation rate: the selected period against the one before, with a 95% interval and a verdict of “Real change”, “Within noise” or “Too few answers”.
  • Alerts: “Source no longer cited” with the domain name, raised when a domain that was cited repeatedly drops out and the number of answers makes chance unlikely.

None of this promises a citation. Allowing a crawler makes your page reachable; whether an answer cites it depends on the question, the page and the competition. What you can show on Monday is the chain of evidence: the audit before and after, and the sources of the same Perplexity answers in the weeks that followed.

The audit comes with every paid plan and any running trial. Plans start with Lite at $12 a month, or $9 a month billed yearly, as of November 10, 2026; the pricing page has the rest. The Action Center then ranks your next moves from the measured answers, each with the evidence behind it.

Check your crawler access

Sign off your crawler policy

Open yourdomain.com/robots.txt in a browser today and fill in these three rows, each with a name. A file full of rules without a stated policy is exactly how a scraper rule ends up shutting out PerplexityBot.

GroupOur ruleSigned by
Search and user request“We let in …”whoever edits robots.txt
Training“We decided … because …”whoever decides how your content may be used
Firewall and CDN“Our CDN says the same.”whoever runs the CDN

Row three stays unsigned until someone has opened the CDN settings.

Frequently asked

Is ChatGPT a web crawler?

ChatGPT itself is an assistant, but OpenAI runs several crawlers around it. According to OpenAI’s crawler page, OAI-SearchBot surfaces websites in ChatGPT’s search features, ChatGPT-User visits a page when a user’s action needs it, and GPTBot crawls content that may be used to train OpenAI’s foundation models. Each has its own user agent name, so your robots.txt can treat them differently.

What are the top 10 web crawlers?

There is no official ranking. A useful ten for a site owner thinking about AI answers: Googlebot, Bingbot, Applebot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, CCBot and Bytespider. The search and user-request crawlers among them matter most for AI answers, because they decide whether a page can be fetched and cited with a link. Training crawlers are a separate decision.

Does blocking GPTBot hurt my visibility in ChatGPT search?

OpenAI documents GPTBot and OAI-SearchBot as separate crawlers and says each setting is independent of the others. Blocking GPTBot is a decision about training. Whether ChatGPT can show and link your pages in search answers depends on OAI-SearchBot access. Read OpenAI’s crawler page on the day you decide, because vendors update these pages.

Does Google-Extended remove my site from AI Overviews?

No, according to Google. Google-Extended manages whether crawled content may be used to train future Gemini models and to ground answers in Gemini Apps, and Google says it does not affect inclusion or ranking in Google Search. A page can appear as a supporting link in AI Overviews or AI Mode when it is indexed and eligible for a snippet in Search. To limit that, Google points to nosnippet, max-snippet and noindex.

Do AI crawlers follow robots.txt?

Anthropic says its bots honor robots.txt, and OpenAI, Perplexity, Google, Apple and Common Crawl explain their opt-outs through it. User-triggered fetchers are a different case: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. A CDN or firewall rule can also block a crawler that robots.txt allows, so check both layers.

How can I check whether AI crawlers can read my site?

Read your robots.txt for each crawler name, then request a page with that crawler’s user agent to see whether a firewall answers with an error or a challenge page. That is the core of any AI crawler checker. FameScan’s AI readiness audit does both for your homepage, checks up to five inner pages, sorts 22 AI crawlers into training, search and user request, and uses no Answer Credits.

Sources & further reading

Sources checked on 10 November 2026.

Keep reading

From understanding to measurement

What answers do your buyers get?

Start with your brand and your buyers’ questions. FameScan helps you understand mentions, sources and gaps.

Start brand discovery How FameScan worksDiscovery included · dashboard with subscription or active trial