AI crawlers are automated bots that AI companies use to fetch web pages for three distinct jobs: collecting training data for language models, building the search indexes behind AI answers, and fetching a page on demand when a user's question needs it. For answer engine optimization (AEO), they are the access layer: a page an engine's crawler cannot reach cannot be retrieved, summarized or cited, whatever its quality.
Practical example
A SaaS company blocked "all AI bots" in 2024 by adding GPTBot, ClaudeBot and PerplexityBot to robots.txt. By 2026 competitors are cited in ChatGPT search and Perplexity, and it is not. The block on GPTBot and ClaudeBot only kept its content out of model training; the block on PerplexityBot removed it from Perplexity's index, and a later firewall rule silently stopped OAI-SearchBot as well. Allowing the search crawlers restores eligibility.
Why are AI crawlers important for AEO?
Every AI answer starts with a fetch. Google's AI Overviews and AI Mode draw on pages Googlebot has indexed; ChatGPT search draws on OAI-SearchBot's index; Perplexity on PerplexityBot's. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and Perplexity and Anthropic document the same for their search crawlers.
Because the bots are named separately, a site can opt out of training while staying visible in AI search, or lose AI visibility by accident through a firewall, a CDN bot-protection rule or a blanket robots.txt block. Checking which crawlers reach the site is therefore the first technical step in AEO.
How do AI crawlers work?
The operators now publish separate user agents for separate purposes, and each can be allowed or blocked on its own in robots.txt.
| Operator | Training data | Search index | User-triggered fetch | Notes |
|---|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User | Blocking GPTBot does not affect ChatGPT search; OpenAI says robots.txt rules may not apply to ChatGPT-User because a user initiated the request. |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User | Three documented bots, all honour robots.txt; blocking one does not block the others. |
| Perplexity | None (Perplexity says its bots are not used for training) | PerplexityBot | Perplexity-User | Perplexity states Perplexity-User generally ignores robots.txt because a user requested the fetch. |
| Google-Extended is a robots.txt token, not a crawler; it limits Gemini training and grounding | Googlebot (also powers AI Overviews and AI Mode) | Not applicable | A Search Console setting controls inclusion in Google's generative AI features; Google-Extended does not. | |
| Others | CCBot (Common Crawl, an open dataset many models train on) | Bingbot (also feeds Microsoft Copilot) | Varies | Names change; the operators' documentation is the source of truth. |
The pipeline has five stages:
- Discovery. Crawlers find pages through links, sitemaps and existing indexes. A page nobody links to and no sitemap lists is rarely fetched.
- Fetch. The bot requests the page and reads the HTML. Most AI crawlers do not execute JavaScript, unlike Googlebot, so content that only appears after scripts run can be invisible to them.
- Processing. Depending on the bot, the content enters a training corpus, a search index, or is read once for a live answer. Training crawls influence what a model knows; index crawls and live fetches decide what it can cite today.
- Retrieval. At answer time the engine's search layer selects pages from its index, or a user-triggered fetcher reads a page the engine wants to check.
- Citation. The answer may link to the retrieved pages. Being crawled is necessary but not sufficient: relevance, quality and the engine's own selection logic decide the rest.
Controls. robots.txt rules per user agent are the standard mechanism, and OpenAI, Anthropic and Perplexity say changes take about a day to apply. Firewalls and CDN bot protection can block crawlers regardless of robots.txt, so server logs are the only way to confirm which bots actually reach the site. For Google, robots.txt for Googlebot governs crawling, the nosnippet family limits what is shown, the Search Console generative AI setting governs inclusion in AI features, and the Google-Extended token governs Gemini training and grounding. In October 2026 Google and Microsoft announced support for a shared AI-training opt-out token in robots.txt, based on IETF work, to take effect in 2027.
Related concepts
- robots.txt: the file that tells crawlers which paths they may fetch, per user agent. It is a request, honoured by documented crawlers; user-triggered fetchers may not apply it.
- Google-Extended: a robots.txt product token, not a bot. It limits use of crawled content for Gemini training and grounding, and has no effect on Google Search, AI Overviews or AI Mode.
- Retrieval-augmented generation (grounding): the mechanism that turns a crawled page into a cited source. Crawling supplies the index; grounding selects from it.
- AI agents: software that browses sites to complete tasks for a user. They overlap with user-triggered fetchers and are governed by their own emerging conventions rather than classic crawler rules.
Frequently asked questions
Decide per purpose, not per company. Blocking a training crawler keeps future content out of that model's training and does not affect current AI visibility. Blocking a search crawler removes the site from that engine's answers. User-triggered fetchers may not honour robots.txt at all. Many sites allow search and fetch bots and block training bots; whichever choice is made, check that firewalls and CDN rules match it.
No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and page visits a user triggers use ChatGPT-User. A site that blocks GPTBot but allows OAI-SearchBot stays eligible for ChatGPT search answers; one that blocks OAI-SearchBot does not, though OpenAI says it may still appear as a navigational link.
Mostly not. Googlebot renders pages; the crawlers from OpenAI, Anthropic and Perplexity fetch the HTML and generally do not run scripts. Key content, including definitions, prices, comparison tables and FAQs, should be present in the server-rendered HTML rather than loaded afterwards, and server logs will show whether the bots return.