AI Crawler Bots Explained: ChatGPT, Gemini, Claude, Perplexity
Which AI bot actually visits your website, and does it belong to the platform you think it does?
Website owners increasingly want their pages to appear in ChatGPT, Gemini, Claude, Perplexity, Grok, Google AI Overviews and Microsoft Copilot. Yet identifying the actual crawler behind each AI service remains confusing.
Is OAI-SearchBot the same as GPTBot? Is Google-Extended a Gemini crawler? Does Claude operate three different agents? Does Copilot use bingbot?
These are not naming details. Confusing search crawlers with model-training bots can produce incorrect access rules and misleading GEO audits. A crawler appearing in server logs also does not prove that a site received an AI citation.
Direct answer: OpenAI documents OAI-SearchBot, GPTBot and ChatGPT-User. Anthropic documents Claude-SearchBot, ClaudeBot and Claude-User. Perplexity documents PerplexityBot and Perplexity-User. Google Search uses Googlebot; Google-Extended is a product-control token, not a separate crawler. Bing uses bingbot. A dedicated official Grok crawler identity was not established in this review.
This guide identifies the documented AI crawler bots for search discovery and citations, explains their declared purposes and separates verified identities from unsupported claims. It is an identity reference, not a speculative account of proprietary AI retrieval algorithms.
Which AI Crawler Bots Belong to Each Platform?
There is no single bot used by every generative AI platform. The correct agent depends on the provider, the product and the type of access involved.
For a technical GEO audit, identify the organization, exact bot token and documented purpose before drawing conclusions about search discovery or citation eligibility.
| Platform | Agent or token | Category | Documented relationship |
|---|---|---|---|
| ChatGPT / OpenAI | OAI-SearchBot |
Search crawler | Helps surface sites in ChatGPT search results |
| ChatGPT / OpenAI | GPTBot |
Training crawler | Collects content that may be used in model training |
| ChatGPT / OpenAI | ChatGPT-User |
User-requested fetcher | Accesses pages for certain user actions |
| Google Search / AI Overviews | Googlebot |
Search crawler | Google Search discovery and indexing |
| Gemini / Google | Google-Extended |
Product-control token | Controls specified training and grounding uses; not a separate HTTP bot |
| Gemini Notebook / Google | Google-GeminiNotebook |
Product-specific fetcher | Fetches user-specified notebook sources |
| Claude / Anthropic | Claude-SearchBot |
Search crawler | Search-related discovery |
| Claude / Anthropic | ClaudeBot |
Training crawler | Potential model-training data collection |
| Claude / Anthropic | Claude-User |
User-requested fetcher | User-directed website access |
| Perplexity | PerplexityBot |
Search crawler | Search-related content discovery |
| Perplexity | Perplexity-User |
User-requested fetcher | User-requested website retrieval |
| Microsoft Copilot / Bing | bingbot |
Search crawler | Bing search crawling and indexing for eligible public-web experiences |
| Grok / xAI | Dedicated crawler not verified | Unconfirmed | No official Grok-specific agent established in this review |
Search crawlers, training bots and user-requested fetchers
Search crawlers support the provider’s search discovery or indexing systems. Examples include OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and bingbot.
Training crawlers collect material that may contribute to model development. Examples include GPTBot and ClaudeBot. Training access does not necessarily determine search inclusion.
User-triggered fetchers may retrieve a specific URL in response to a user action. Examples include ChatGPT-User, Claude-User and Perplexity-User. These are not simply alternative names for search-index crawlers.
ChatGPT: OpenAI’s Official Web Crawlers
OpenAI officially distinguishes OAI-SearchBot, GPTBot and ChatGPT-User. They share an operator but serve separate documented purposes.
OAI-SearchBot: ChatGPT search discovery
Answer: OAI-SearchBot is the OpenAI search crawler used to surface websites in ChatGPT search features. It is the relevant search robots.txt token for managing automatic ChatGPT search crawling.
Allowing it can support search eligibility. It does not force the system to select, cite or recommend a particular URL.
GPTBot: model-development crawling
GPTBot crawls content that may be used to train OpenAI generative AI foundation models. It is not the dedicated ChatGPT Search crawler.
OpenAI states that publishers may allow OAI-SearchBot while disallowing GPTBot. These independent settings matter when a client wants search visibility without the same permission for training use.
ChatGPT-User: certain user-initiated website requests
ChatGPT-User identifies certain webpage requests made for user actions in ChatGPT or Custom GPTs. OpenAI states that this agent is not used for automatic web crawling to determine ChatGPT Search inclusion.
Because these requests can be user-initiated, standard robots.txt expectations may not apply in the same way as with automatic crawlers.
Technical takeaway: For ChatGPT, search discovery, potential training collection and user-directed access are three distinct functions. Check OpenAI’s official crawler reference
Google Gemini: Googlebot, Google-Extended and Product-Specific Fetchers
Gemini must not be assigned a made-up universal crawler merely because it is a Google AI product. Google’s shared crawling infrastructure, product-control tokens and user-triggered fetchers represent different technical relationships.
Googlebot: established Google Search infrastructure
Googlebot is Google’s established web-search crawler. It supports discovery and indexing for Google Search and the search eligibility relevant to Google AI Overviews and AI Mode.
This does not prove that every Gemini Apps interaction sends a Googlebot request. The Gemini product experience and Google Search features should be distinguished.
Google-Extended: a control token, not a crawler
Answer: Google-Extended is a robots.txt product token. Google explicitly says it has no separate HTTP User-Agent string. Existing Google crawler identities perform the network requests.
Google-Extended controls specified uses of crawled content for future Gemini model training and designated Gemini grounding experiences. Google states it does not affect Google Search inclusion and is not a Google Search ranking signal.
Consequently, blocking Google-Extended is not the same action as blocking Googlebot. It also does not automatically remove a URL from Google AI Overviews.
Google-GeminiNotebook and Google-Agent: specific fetcher contexts
Google documents Google-GeminiNotebook as a fetcher for URLs supplied by users to a Gemini Notebook experience. It should not be relabeled as a general Gemini search crawler.
Google also documents Google-Agent for certain user-requested agent activities. That identity does not replace Googlebot for ordinary Google Search indexing.
Technical takeaway: Distinguish a search crawler, a robots.txt product-control token and a product-specific user fetcher. Review Google’s common crawler reference and user-triggered fetcher reference.
Claude: Anthropic’s Three Documented Crawler Bots
Anthropic officially names Claude-SearchBot, ClaudeBot and Claude-User. These should be recorded as separate agents during a client website audit.
Claude-SearchBot: search-related discovery
Claude-SearchBot supports search-related discovery. Anthropic describes it as helping improve the relevance and accuracy of search responses.
Restricting it can affect whether website material is indexed for the relevant search functions. Permission to crawl still does not guarantee source citation.
ClaudeBot: potential model-training data collection
ClaudeBot collects public web data that may be used to train or improve Anthropic models. It is a distinct identity from Claude-SearchBot.
Claude-User: user-directed retrieval
Claude-User handles certain webpage retrieval requests associated with users. Anthropic explains that restrictions may affect its ability to fetch relevant website material for user-directed requests.
Technical takeaway: For Claude, the search-oriented bot, training-related crawler and user fetcher have different documented functions. Consult Anthropic’s crawler documentation.
Perplexity: PerplexityBot and Perplexity-User
Perplexity publicly documents two important identifiers: PerplexityBot for search-related crawling and Perplexity-User for user-requested retrieval.
PerplexityBot: search-related crawling
PerplexityBot helps discover material for Perplexity’s search functionality. It is the bot identity most relevant when reviewing automatic search discovery.
A server request from this agent should not be counted as an AI source citation. Citations must be observed separately in answer outputs or reporting evidence.
Perplexity-User: requested webpage access
Perplexity-User accesses content in certain user-requested contexts. Perplexity publishes documentation and bot IP information for identifying its agents.
Technical takeaway: Track search crawling separately from user-requested fetching. See Perplexity’s official crawler documentation
6. Microsoft Copilot: Bingbot and Public Web Discovery
Microsoft Copilot is a family of product experiences. Consumer search experiences, Bing-generated answers and Microsoft 365 Copilot cannot all be treated as one public-web crawler system.
Bingbot: Bing search discovery and indexing
bingbot is the documented Bing crawler. Bing indexing supports public-web search experiences that may contribute to eligible Copilot and Bing AI answers.
Bingbot should not be renamed CopilotBot without an authoritative source. It supports search discovery; it does not itself create a generated citation.
Is there a universal CopilotBot?
A separately documented, universal public-web CopilotBot should not be assumed. Other Copilot experiences may access organizational data or use authorized connectors rather than simply crawl public websites with bingbot.
AI citation reporting in Bing Webmaster Tools
Microsoft provides an AI Performance report in Bing Webmaster Tools for supported Copilot, Bing AI summary and partner experiences.
The report describes cited URLs, citation counts, grounding-query summaries and trends. Its numbers describe observed citations, not search rankings, clicks or proof that a crawler caused a particular citation.
Technical takeaway: Use bingbot to identify Bing public-web crawling, and AI Performance to evaluate supported citation outcomes. Keep those evidence categories separate.
Google AI Overviews: Does It Have a Separate Crawler?
Direct answer: Google does not document a separate AIOverviewBot as a prerequisite for AI Overviews. The relevant search-discovery infrastructure is Google Search and Googlebot.
Googlebot and search eligibility
Google explains that a webpage needs to be indexed and eligible to appear in Search with a snippet to be considered as a supporting link in AI Overviews or AI Mode.
This connects AI Overview eligibility to standard Search indexing and controls. No special AI search crawler file or standalone GEO markup is required.
Why Gemini Apps are different
Gemini Apps are distinct from AI Overviews inside Google Search. Google-Extended concerns designated Gemini training and grounding uses, not ordinary Google Search inclusion.
That distinction answers a common technical question: blocking Google-Extended does not, on its own, block a page from Google AI Overviews.
Official reference: Google’s AI features and your website.
Grok: What Is Officially Verified About Its Crawler?
Grok can provide responses informed by external information in supported contexts. That capability does not establish the public name of a dedicated xAI web crawler.
Is GrokBot a confirmed official crawler?
For this October 2026 review, a dedicated Grok-specific crawler identifier was not independently confirmed from official xAI documentation. Third-party mentions of GrokBot are not sufficient evidence to classify that token as official.
The absence of a confirmed token is not proof that Grok never accesses websites. Product retrieval capability and publicly documented crawler identity are different questions.
How to handle unknown crawler identities
Do not label server requests as authentic xAI traffic solely because a User-Agent string contains a Grok-related name. Require official provider information or other credible verification.
Technical takeaway: In a GEO crawler inventory, mark the Grok identity as not verified, not as a guessed bot name.
How to Identify an Official AI Crawler or Agent
Reliable AI search crawler identification requires more than recognizing a familiar token in a log file.
HTTP User-Agent versus robots.txt user-agent token
An HTTP User-Agent header is sent with a request and describes the requesting client. A robots.txt user-agent token selects crawler-specific access rules.
The names may look similar, but the technical fields are different. Google-Extended is a particularly important example because it is a product-control token with no distinct HTTP User-Agent.
Check published IP ranges and provider verification guidance
A User-Agent header can be spoofed. For a claimed Googlebot request, consult Google’s verification guidance.
For OpenAI and Perplexity, consult their official bot references and published IP data where available. IP matching and documented DNS procedures can strengthen identification.
Do not use an outdated third-party crawler directory as the sole authority for a current agent identity.
Fields for an AI crawler inventory
| Field | What to record |
|---|---|
| Operator | OpenAI, Google, Anthropic, Perplexity, Microsoft or another verified organization |
| Product context | Search, conversational assistant, enterprise experience or user-triggered tool |
| Exact identifier | Documented token and, when relevant, full HTTP User-Agent pattern |
| Declared role | Search discovery, training, user fetcher or product-use control |
| Primary source | Official documentation link and date reviewed |
| Identity evidence | Applicable IP or DNS verification, if available |
| Observed website access | Server logs, URL, HTTP status and applicable access settings |
| Citation evidence | Separate captured AI response or supported citation-reporting record |
A logged request should be marked unverified if identity verification was not completed. An absence of requests in limited logs also does not establish that an AI product could never access or cite the page.
What Crawler Identity Means for AI Discovery and Citations
Understanding actual AI bots is necessary for accurate technical documentation. It is not a substitute for measuring AI visibility.
Discovery and access
A search crawler or fetcher may request content when the system and access conditions permit. Successful requests establish access to a resource at a particular time.
Indexing and retrieval eligibility
A search provider may index eligible content or retrieve information through supported search and user-requested mechanisms. Each product has different requirements.
Source citation and mention
An AI source citation is a visible attribution to a source in an AI answer. A brand mention identifies a name without necessarily linking to evidence.
Neither is proved by a crawler log entry. Some citations may be based on an existing index or another permitted retrieval route, with no recent hit from a dedicated AI crawler recorded immediately beforehand.
What to report in a client GEO audit
| Assessment | Evidence needed | What it does not prove |
|---|---|---|
| Bot identity | Official documentation and authenticated request details | That a page was cited |
| Access | Server request, status and resource returned | That the page was indexed |
| Eligibility | Available search index or provider-specific evidence | That the page will be selected |
| Citation | Captured response, source URL or supported citation report | That a specific bot or change caused the result |
| Business impact | Attributable analytics and conversions where available | That citations necessarily increased revenue |
For organizations building a GEO program, the first job is accurate classification. The second is separate measurement of citations, referral traffic and conversions.
Professionals seeking practical implementation can explore Endtrace’s SEO-integrated GEO course for SaaS and technology websites, which covers technical foundations, AI visibility evaluation and project-based measurement.
Frequently Asked Questions About AI Crawler Bots
What is ChatGPT Search’s official crawler?
OAI-SearchBot is OpenAI’s documented crawler for ChatGPT search discovery. GPTBot has a separate model-development purpose.
Can I allow OAI-SearchBot but block GPTBot?
Yes. OpenAI documents independent controls for search discovery and model-training crawling. Allowing search access still does not guarantee a citation.
What is the difference between OAI-SearchBot and ChatGPT-User?
OAI-SearchBot supports automatic search discovery. ChatGPT-User handles certain user-initiated requests and is not used to determine Search inclusion.
Does Claude use separate search and training bots?
Yes. Anthropic documents Claude-SearchBot for search-related functions and ClaudeBot for potential model-training data collection. Claude-User handles user-directed retrieval.
Does PerplexityBot train models or support search?
Perplexity documents PerplexityBot as a search-related crawler. Its separate Perplexity-User identifier is for user-requested fetching.
Is Google-Extended a crawler or a control token?
It is a product-control token in robots.txt, not a separate HTTP crawler. Google uses existing crawler identities for network requests.
Does blocking Google-Extended remove a page from AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search. AI Overviews eligibility depends on applicable Google Search requirements.
Does Gemini have one official universal web search bot?
Do not assume one. Google documents shared crawlers, product controls and specific user-triggered fetchers. Each product context must be identified separately.
Does Google AI Overviews use Googlebot?
Google’s AI search features rely on the established Google Search crawling and indexing infrastructure, including Googlebot. There is no separately required AIOverviewBot.
Does Microsoft Copilot use bingbot?
Bingbot supports public Bing search discovery and indexing relevant to Bing-based Copilot experiences. Other Copilot products may use different data-access arrangements.
Is CopilotBot an official universal public-web crawler?
A universal official CopilotBot should not be assumed. Use documented Bing crawler identities for the public Bing search context.
Has xAI officially confirmed a GrokBot user agent?
A dedicated Grok-specific crawler identity was not verified from primary xAI documentation during this review. Do not treat third-party bot names as authoritative.
Where can I find official crawler IP information?
Start with the provider’s bot documentation and published IP feeds where available. OpenAI and Perplexity publish bot reference information, while Google provides verification guidance.
Why might a Googlebot User-Agent in logs be fake?
The HTTP User-Agent header can be spoofed. Verify apparent Googlebot requests using Google’s recommended IP or DNS validation methods.
Can AI cite a page without a recent crawler request?
Yes. Responses may use content from an existing index or another authorized retrieval source. A fresh bot visit immediately before a citation is not always required.
Does allowing an AI search crawler guarantee citation?
No. Crawler access, search eligibility and citation are separate outcomes. Actual citations require response-level or supported reporting evidence.
What belongs in a client AI crawler audit?
Record the provider, product, exact agent, declared purpose, official source, identity verification, access status and separate citation observations.
Final Perspective: Know the Bot Before Assessing AI Visibility
An accurate AI crawler reference does not need unsupported claims about secret algorithms. It needs correct identities, official sources, clear product boundaries and careful distinctions.
OAI-SearchBot is not GPTBot. Google-Extended is not Googlebot. Claude-SearchBot is not ClaudeBot. PerplexityBot is not Perplexity-User. Bingbot is not automatically a separate CopilotBot.
These distinctions help website owners avoid incorrect crawler policies and misleading technical reports.
Most importantly, a confirmed bot visit is not an AI citation. Search discovery, model-training access, user-requested fetching and visible source attribution are different technical outcomes.
When someone claims a website is optimized for every AI bot, ask a simple question: Which officially documented agents are being evaluated?
That is where a reliable technical SEO and GEO discussion begins.
