AI search crawlers are the bots AI companies use to fetch web content for indexing, retrieval and answering, each with its own user agent and controls, such as OAI-SearchBot, GPTBot and PerplexityBot. This is a cross-vendor reference page, verified at the review date below. Its organising fact: every major vendor now separates search indexing, model training and user-triggered fetching into distinct agents, and treating them as one “AI bot” is how sites end up invisible by accident or trained on by accident.
In one sentence
The crawlers are the front door of AI visibility: which ones can reach a site decides which engines can cite it, before content quality gets a vote.
The current agents, by vendor
OpenAI. OAI-SearchBot indexes for ChatGPT search: the citation eligibility control. GPTBot collects content that may be used for training. ChatGPT-User fetches pages on a user’s direct request. The three are independent; robots.txt changes take up to about a day to affect search eligibility, and OpenAI publishes IP ranges for verification.
Anthropic. Claude-SearchBot indexes for Claude’s search: the citability gate. ClaudeBot is the training crawler. Claude-User handles user-triggered fetches. All honour robots.txt including crawl-delay; the older Claude-Web and anthropic-ai tokens are deprecated and match nothing current.
Perplexity. PerplexityBot builds the search index and, per Perplexity, its content is not used for foundation model training since Perplexity does not build them. Perplexity-User performs user-initiated fetches, which Perplexity treats as agent activity rather than crawling for robots purposes; its compliance record has been publicly contested, and stricter control sits at the WAF layer.
Google. Googlebot does all the fetching; there is no separate AI crawler. The Google-Extended robots token governs whether crawled content trains and grounds Gemini, and it does not affect Search inclusion, AI Overviews or AI Mode, which run on standard crawling and snippet controls.
The rules that hold across vendors
Robots.txt is advisory, honoured by declared agents and irrelevant to anything else, so real enforcement lives at the server, CDN and WAF layer, which is also where sites most often block AI crawlers without meaning to: a firewall rule sits above robots.txt and fails silently. Verification is by published IP ranges plus user-agent, since agents are trivially spoofed. And the search, training and fetch decisions are separable everywhere, so the common publisher stance, citable but not trained on, is a configuration, not a compromise. Server logs, not assumptions, are the ground truth for which agents actually reach a site.
Why crawler literacy matters
Access failures are the cheapest visibility problems to fix and the most common to have, and they cap everything downstream: no crawl, no index, no citation probability, whatever the content deserves. Crawler names and behaviours also change; this page carries a verification date for exactly that reason.
Related concepts
References
- OpenAI Help Center, Publishers and Developers FAQ: help.openai.com
- Perplexity, Perplexity Crawlers documentation: docs.perplexity.ai
- Google Search Central, Optimising your website for generative AI features on Google Search: developers.google.com
Author: Harpal Singh · Last verified: 7 August 2026