Which AI crawlers to allow and which to block
AI bots do three different jobs: collect training data, fetch pages to ground an answer, and follow a link a user just asked about. Blocking them as one group costs visibility you probably want.
Contents
- Key takeaways
- Three different jobs, one word
- The user agents that are actually documented
- What robots.txt controls, and what it does not
- Blocking training is not blocking visibility
- What blocking costs, and what crawling costs you
- Controls beyond robots.txt
- llms.txt is a proposal, not a standard
- A default policy you can defend
- FAQ
AI bots do three different jobs. Some collect training data, some fetch pages to ground an answer being written now, and some follow a link a user asked about. Operator documentation names them separately, so robots.txt can too. Blocking them as one group removes visibility most sites want to keep.
Key takeaways
- Training crawlers and retrieval fetchers are different user agents with different consequences. GPTBot trains, OAI-SearchBot surfaces sites in ChatGPT search, and disallowing OAI-SearchBot removes a site from those answers.
- Google-Extended controls training and grounding for Gemini apps only. Google states it does not affect inclusion or ranking in Google Search, and it does not control AI Overviews.
- User-triggered fetchers are the exception to the model. OpenAI and Perplexity both document that robots.txt rules may not apply when a person asks for a specific page.
- robots.txt is a request, not an access control. RFC 9309 says the rules "are not a form of access authorization", and in August 2025 Cloudflare published evidence of undeclared crawling that ignored them.
- Adoption is still thin. In the 2025 Web Almanac, GPTBot appeared in 4.5% of desktop robots.txt files and ClaudeBot in 3.6%, both up sharply from 2024 but still a small minority of sites.
Three different jobs, one word
"AI crawler" collapses three behaviors that carry different business consequences.
Training collection downloads content to build or fine-tune a model. There is no attribution, no link and no traffic. The value exchange is one-directional, which is why publishers argue about it.
Retrieval and grounding fetches pages while an answer is being composed, and the assistant usually cites and links what it used. That is closer to a search index than to a dataset. It is the surface where being cited matters, following the selection logic described in how AI assistants choose the sources they cite.
User-triggered fetching happens when someone pastes a URL or asks about a specific page. Operators treat this as a browser acting for a person rather than as a crawl, and they document that robots.txt may not stop it.
A single Disallow: / for everything AI-shaped treats these as one thing. They are not.
The user agents that are actually documented
Every agent below appears in its operator's own documentation, which is where these rules should always be checked before deployment.
- [OpenAI](https://developers.openai.com/api/docs/bots) documents
GPTBotfor training,OAI-SearchBotfor surfacing sites in ChatGPT search, andChatGPT-Userfor user-initiated actions. OpenAI notes that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. - [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) documents
ClaudeBotfor collecting content that may contribute to training,Claude-Userfor user-initiated requests, andClaude-SearchBotfor improving search result quality. Each is blocked by its own robots.txt group. - [Google](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) documents
Googlebotfor Search andGoogle-Extendedas a standalone token covering Gemini training and grounding.Google-CloudVertexBotcrawls only for Vertex AI agents at a site owner's request. - [Perplexity](https://docs.perplexity.ai/guides/bots) documents
PerplexityBot, which surfaces and links sites in results and does not crawl for training. It also documentsPerplexity-User, which serves user-initiated visits and, per Perplexity's documentation, generally ignores robots.txt. - [Apple](https://support.apple.com/en-us/119829) documents
Applebotfor Spotlight, Siri and Safari, andApplebot-Extended, which is not a crawler at all. It is an opt-out token for training use, and pages that disallow it can still appear in Apple search results. - [Meta](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) documents
meta-externalagentfor training and indexing andmeta-externalfetcherfor user-requested link fetches, alongside the olderfacebookexternalhit. - [Common Crawl](https://commoncrawl.org/ccbot) operates
CCBot, whose archives feed many third-party datasets, and [Amazon](https://developer.amazon.com/en/amazonbot) operatesAmazonbot, which its documentation says may be used to train Amazon AI models.
What robots.txt controls, and what it does not
Robots.txt expresses a preference to well-behaved clients. It is not a permission system, and treating it as one is the most common mistake in this area.
RFC 9309, published in September 2022, standardizes the Robots Exclusion Protocol and is explicit about its limits. The rules "are not a form of access authorization", and the file "is not a substitute for valid content security measures". Listing a path also makes it publicly discoverable (IETF, September 2022).
Compliance is voluntary, and non-compliance is documented. In August 2025 Cloudflare published findings that Perplexity had used undeclared crawlers with rotating addresses to reach content on test domains whose robots.txt disallowed all bots. Cloudflare removed Perplexity from its verified bot list as a result. Whatever the eventual resolution, the operational lesson holds: if content must not be read by a machine, it needs authentication, not a directive.
Rules also apply per host. A robots.txt on the apex domain does nothing for a documentation subdomain or a separate media host, and Anthropic's documentation states this explicitly.
Blocking training is not blocking visibility
The most expensive error is a broad block that removes a site from assistant answers as a side effect.
Google's position is documented and unusually clear. Google-Extended governs training and grounding for Gemini apps and the Vertex AI API. Google states that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search". Separately, Google's documentation on AI features says that robots.txt directives for Googlebot control how sites are crawled for Search, and that nosnippet, data-nosnippet, max-snippet and noindex limit what appears. In other words, AI Overviews are governed by Search controls, not by Google-Extended.
The same split applies elsewhere. Disallowing GPTBot signals that content should not be used in training. Disallowing OAI-SearchBot removes the site from ChatGPT search answers. They are separate decisions and should be made separately.
What blocking costs, and what crawling costs you
The commercial trade-off is genuine, and the evidence points both ways.
Cloudflare's Radar analysis of 19–26 June 2025 measured crawl-to-refer ratios by platform. It reported roughly 71,000 HTML page requests from Anthropic for every referral sent back, while Mistral sent about ten referrals per crawl request (Cloudflare, July 2025).
For a publisher whose revenue depends on page views, that asymmetry is the whole argument. For a services business whose site exists to be found, being absent from an assistant's answer is usually worse than being read without a click. That is the reasoning behind treating content that compounds as an asset rather than a campaign.
Infrastructure changed the defaults too. On 1 July 2025 Cloudflare announced that new domains would block AI crawlers by default unless operators pay for access, alongside a pay-per-crawl mechanism. That does not make blocking correct; it makes it easy, which means many sites now block without deciding.
Controls beyond robots.txt
Three other layers exist, and they do different work.
Content signals. In September 2025 Cloudflare published the Content Signals Policy, a robots.txt extension expressing what may be done with content after it is fetched. It defines three signals: search, ai-input and ai-train. Cloudflare applied search=yes, ai-train=no to more than 3.8 million domains already using its managed robots.txt, leaving ai-input unset.
Snippet controls. nosnippet, data-nosnippet and max-snippet limit how much of a page can be reproduced without removing it from an index. On Google's own documentation these are the levers for AI features in Search.
Network enforcement. WAF rules, rate limits and bot verification by published IP range are what actually stop a client that ignores the file. Every operator listed here publishes address ranges for verification, which makes user-agent spoofing detectable.
llms.txt is a proposal, not a standard
`llms.txt` was proposed by Jeremy Howard on 3 September 2024 as a markdown file that helps language models use a site at inference time. The specification's own site calls it a proposal and describes it as open for community input.
In the 2025 Web Almanac, published on 15 January 2026, HTTP Archive found llms.txt on about 2% of mobile sites, with 324,184 valid files identified. Of those, 39.6% were generated by the All in One SEO plugin and 3.6% by Yoast, suggesting most were produced by defaults rather than chosen (HTTP Archive, January 2026).On 17 June 2025 Google's John Mueller wrote that "no AI system currently uses llms.txt". None of the operator documentation cited in this article mentions the file. Publishing one is cheap and harmless; treating it as a ranking or citation mechanism is not supported by anything an operator has published.
A default policy you can defend
For most commercial sites that want assistant visibility and are unwilling to donate a full training corpus, the defensible starting position is narrow rather than absolute.
Allow the retrieval and search agents: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot and Googlebot. Decide training separately by disallowing GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot and Amazonbot if the content is a commercial asset rather than a marketing surface. Leave user-triggered fetchers alone, since blocking them mainly breaks the case where someone deliberately shared your link.
Then verify rather than assume. Read server logs monthly by user agent and confirm requests against each operator's published address ranges. Re-read the documentation quarterly, because these tokens change more often than most technical SEO settings. The same discipline that governs a JavaScript budget applies here: a policy nobody measures is a policy nobody has.
FAQ
Does blocking GPTBot remove my site from ChatGPT?
No. OpenAI documents GPTBot as the training crawler and OAI-SearchBot as the agent that surfaces sites in ChatGPT search. Disallowing GPTBot addresses training use. Disallowing OAI-SearchBot is what removes a site from ChatGPT search answers, and OpenAI states that consequence directly.
Does Google-Extended control AI Overviews?
No. Google's documentation limits Google-Extended to training and grounding for Gemini apps and the Vertex AI API, and states it does not affect Search inclusion or ranking. AI features inside Search are governed by Googlebot access and by snippet controls such as nosnippet and data-nosnippet.
Do AI crawlers actually obey robots.txt?
Most declared agents do, and several publish IP ranges so you can verify them. Compliance is voluntary under RFC 9309, and Cloudflare documented an evasion case in August 2025. Content that genuinely must not be read by a machine belongs behind authentication.
Is llms.txt worth publishing?
It costs little and proves nothing. HTTP Archive found it on about 2% of mobile sites in 2025, mostly generated by SEO plugins. Google's John Mueller said in June 2025 that no AI system used it. Publish it if a plugin already does; do not build strategy on it.
What about user-triggered fetchers?
Treat them as browsers rather than crawlers. OpenAI documents that robots.txt rules may not apply to ChatGPT-User because a person initiated the request, and Perplexity documents that Perplexity-User generally ignores robots.txt. Blocking them mostly punishes the visitor who deliberately shared your page, and it removes no training exposure.
How often should this be reviewed?
Quarterly. New tokens appear, existing ones change scope, and operator documentation gets updated without announcement. A recurring calendar entry holding the operator documentation URLs is enough for the review itself, and the decision belongs with the other governance items in an AI budget approval.
Separate the three jobs first, then decide training in isolation from retrieval, then enforce at the network layer whatever the file only requests. Blanket blocking is a decision about revenue models made with a technical tool, and most companies making it have not priced it. Over the next year, watch whether paid crawl agreements reach smaller publishers and whether retrieval traffic converts at rates worth reporting. Until either happens, the honest position is the one visible in current agent deployments across the mid-market, where measurement still lags adoption.
