AI crawlers list
54 bots from 27 companies are crawling the web to train AI models, power search, and fetch content for users. Know who visits your site and what they do with your content.
Want to control which bots access your site?
Use robots.txt rules, generate the rules, or run an AlfaUMi analysis to see which crawlers your robots.txt lets in.
| Bot name | Company | Influence | Monthly users | Description | Source |
|---|---|---|---|---|---|
GPTBot | OpenAI | 8/ 10High | 900M+ | Crawls content that may train OpenAI's foundation models. Blocking it keeps content out of training, but does NOT remove the site from ChatGPT search: that is OAI-SearchBot. | operator docs for GPTBot ↗ |
ChatGPT-User | OpenAI | 10/ 10Critical | 900M+ | Fetches a page when a ChatGPT or Custom GPT user asks something. User-initiated, so OpenAI says robots.txt rules may not apply. | operator docs for ChatGPT-User ↗ |
OAI-SearchBot | OpenAI | 10/ 10Critical | 900M+ | Surfaces sites in ChatGPT's search features. OpenAI: sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. | operator docs for OAI-SearchBot ↗ |
OAI-AdsBot | OpenAI | 4/ 10Low | Advertisers | Validates the safety of pages submitted as ads on ChatGPT. Only relevant if you advertise there. | operator docs for OAI-AdsBot ↗ |
Google-Extended | 10/ 10Critical | 750M+ | A robots.txt token, not a crawler: controls Gemini training and grounding in Gemini Apps and Vertex AI. Google says it does not affect inclusion in Google Search. Leaving AI Overviews is a separate Search Console control. | operator docs for Google-Extended ↗ | |
GoogleOther | 8/ 10High | 750M+ | General Google crawler for AI products (NotebookLM, Vertex AI). Used across various Google AI services. | operator docs for GoogleOther ↗ | |
Gemini-Deep-Research | 8/ 10High | 750M+ | Gemini Deep Research crawler for in-depth page analysis. Fetches content for advanced queries. | independent list lists Gemini-Deep-Research ↗ | |
Google-GeminiNotebook | 7/ 10High | ~50M | Fetcher for Gemini Notebook, Google's document analysis tool. Renamed from Google-NotebookLM on 2026-07-16; the old token still works for now. | operator docs for Google-GeminiNotebook ↗ | |
Google-NotebookLM | 5/ 10Medium | ~50M | Former name of Google-GeminiNotebook, still honoured. Keep rules for both until Google retires it. | operator docs for Google-NotebookLM ↗ | |
Google-Agent | 7/ 10High | New (2026) | Used by agents hosted on Google infrastructure to navigate the web and act on a user's request. Documented 2026-03-20 with its own IP ranges; as a user-triggered fetcher it generally ignores robots.txt. | operator docs for Google-Agent ↗ | |
GoogleAgent-Mariner | 6/ 10Medium | New (2025) | Experimental Google agent for autonomous web browsing. Project Mariner. | independent list lists GoogleAgent-Mariner ↗ | |
Google-CloudVertexBot | 6/ 10Medium | Enterprise | Google Cloud Vertex AI crawler for enterprise customers. B2B, smaller consumer reach. | operator docs for Google-CloudVertexBot ↗ | |
ClaudeBot | Anthropic | 7/ 10High | 30M+ | Collects web content that may contribute to training Anthropic's models. Blocking it excludes future material from training; search and user fetches are Claude-SearchBot and Claude-User. | operator docs for ClaudeBot ↗ |
Claude-User | Anthropic | 9/ 10Critical | 30M+ | Fetches a page when a Claude user asks. Anthropic says blocking it may reduce your visibility for user-directed web search. | operator docs for Claude-User ↗ |
Claude-Web | Anthropic | 3/ 10Low | Legacy | Legacy token: Anthropic's current crawler documentation no longer lists it. Write rules for ClaudeBot, Claude-SearchBot and Claude-User instead. | independent list lists Claude-Web ↗ |
Claude-SearchBot | Anthropic | 9/ 10Critical | 30M+ | Navigates the web to improve Claude's search results. Anthropic says blocking it may reduce your visibility and accuracy in user search results. | operator docs for Claude-SearchBot ↗ |
anthropic-ai | Anthropic | 3/ 10Low | Legacy | Legacy token: Anthropic's current crawler documentation no longer lists it. | independent list lists anthropic-ai ↗ |
PerplexityBot | Perplexity | 9/ 10Critical | 45M+ | Surfaces and links sites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models. | operator docs for PerplexityBot ↗ |
Perplexity-User | Perplexity | 8/ 10High | 45M+ | Fetches a page when a Perplexity user asks something. Perplexity says this fetcher generally ignores robots.txt, because a user initiated the request. | operator docs for Perplexity-User ↗ |
Grokbot | xAI | 8/ 10High | 78M+ | xAI (Elon Musk) crawler for Grok. Integrated with X (Twitter). Rapid growth via premium X. | unverified |
xAI-Grok | xAI | 8/ 10High | 78M+ | Alternative Grok crawler identifier. Indexes pages for Grok knowledge base. | unverified |
Meta-WebIndexer | Meta | 8/ 10High | 1B+ | Navigates the web to improve Meta AI search result quality. This is Meta's discoverability token: block it and you stay out of Meta AI answers. | independent list lists Meta-WebIndexer ↗ |
Meta-ExternalAgent | Meta | 8/ 10High | 1B+ | Crawls for use cases such as training foundation AI models, per Meta's crawler page. Not the bot that puts you inside a Meta AI answer. | independent list lists Meta-ExternalAgent ↗ |
Meta-ExternalFetcher | Meta | 8/ 10High | 1B+ | Fetches individual links at a user's request. Meta says it may bypass robots.txt, because the fetch is user-initiated. | independent list lists Meta-ExternalFetcher ↗ |
Meta-ExternalAds | Meta | 5/ 10Medium | 1B+ | Crawls for use cases such as improving advertising and other business-related products. Named on Meta's crawler page, but absent from the community list, and that page cannot be machine-checked from here. | unverified |
meta-externalagent | Meta | 8/ 10High | 1B+ | Lowercase form seen in logs (meta-externalagent/1.1). Same control as Meta-ExternalAgent, not a second bot. | independent list lists meta-externalagent ↗ |
FacebookBot | Meta | 5/ 10Medium | 1B+ | NOT on Meta's current crawler page, which documents FacebookExternalHit, Meta-ExternalAgent, Meta-ExternalFetcher, Meta-WebIndexer and Meta-ExternalAds. Kept because the community list still carries it and old rules still name it. | independent list lists FacebookBot ↗ |
Applebot | Apple | 8/ 10High | 2B+ | Apple crawler for Siri, Spotlight and Apple News. Apple does not use nosnippet-tagged pages as context for AI answers, and Applebot does not follow crawl-delay. | operator docs for Applebot ↗ |
Applebot-Extended | Apple | 7/ 10High | 2B+ | Does not crawl. A robots.txt token that opts content out of training Apple's foundation models; pages that disallow it can still appear in search results. | operator docs for Applebot-Extended ↗ |
bingbot | Microsoft | 9/ 10Critical | 100M+ | Main Bing crawler. Also used by Copilot and other Microsoft AI products. | operator docs for bingbot ↗ |
BingPreview | Microsoft | 7/ 10High | 100M+ | Generates link previews in Bing/Teams/Outlook. Preview thumbnails. | unverified |
Amazonbot | Amazon | 7/ 10High | 500M+ | Amazon crawler for Alexa and Amazon Search. Answers voice queries. | operator docs for Amazonbot ↗ |
AmazonBuyForMe | Amazon | 6/ 10Medium | New (2026) | New Amazon shopping agent. Autonomously browses pages for purchases. | independent list lists AmazonBuyForMe ↗ |
Amzn-SearchBot | Amazon | 6/ 10Medium | 500M+ | Amazon crawler for product and information search. | operator docs for Amzn-SearchBot ↗ |
DuckAssistBot | DuckDuckGo | 7/ 10High | 100M+ | DuckDuckGo AI Assistant crawler. Privacy-first, doesn't track users. | operator docs for DuckAssistBot ↗ |
YouBot | You.com | 6/ 10Medium | 10M+ | You.com crawler - AI search engine. Perplexity competitor. | independent list lists YouBot ↗ |
PhindBot | Phind | 6/ 10Medium | 5M+ | Phind crawler - AI search for developers. Specializes in code. | independent list lists PhindBot ↗ |
Andibot | Andi | 5/ 10Medium | 2M+ | Andi Search crawler - conversational AI search engine. | independent list lists Andibot ↗ |
Bravebot | Brave | 6/ 10Medium | 70M+ | Brave Search and Leo AI crawler. Privacy-focused browser with AI. | independent list lists Bravebot ↗ |
iAskBot | iAsk.ai | 5/ 10Medium | 3M+ | iAsk.ai crawler - free AI search engine. Popular in education. | independent list lists iAskBot ↗ |
KomoSearch | Komo | 4/ 10Low | 1M+ | Komo AI Search crawler. Smaller player in AI search market. | unverified |
DeepSeekBot | DeepSeek | 8/ 10High | 97M+ | Chinese DeepSeek crawler. #4 globally, cheap model, 35% users from China. | independent list lists DeepSeekBot ↗ |
MistralAI-User | Mistral | 6/ 10Medium | 5M+ | Mistral AI crawler (France). European OpenAI competitor, open-source. | independent list lists MistralAI-User ↗ |
cohere-ai | Cohere | 5/ 10Medium | Enterprise | Cohere crawler - Canadian AI company. Mainly B2B/enterprise. | independent list lists cohere-ai ↗ |
cohere-training-data-crawler | Cohere | 5/ 10Medium | Enterprise | Cohere crawler for collecting training data. | independent list lists cohere-training-data-crawler ↗ |
Diffbot | Diffbot | 6/ 10Medium | Enterprise | Diffbot crawler - structures web data. B2B, knowledge graphs. | independent list lists Diffbot ↗ |
CCBot | Common Crawl | 7/ 10High | Research | Common Crawl crawler - nonprofit collecting data for AI research. Used by many AI companies. | operator docs for CCBot ↗ |
DataForSeoBot | DataForSEO | 5/ 10Medium | B2B | DataForSEO crawler - SEO data for marketers and tools. | operator docs for DataForSeoBot ↗ |
Bytespider | ByteDance | 8/ 10High | 1B+ | ByteDance (TikTok, Douyin) crawler. Indexes pages for TikTok search and AI. | independent list lists Bytespider ↗ |
TikTokSpider | ByteDance | 7/ 10High | 1B+ | Alternative TikTok crawler. Fetches content for in-app display. | independent list lists TikTokSpider ↗ |
PetalBot | Huawei | 6/ 10Medium | 500M+ | Huawei Petal Search crawler. Popular in China and on Huawei devices. | independent list lists PetalBot ↗ |
SemrushBot | Semrush | 5/ 10Medium | B2B | Semrush crawler - SEO tool. Analyzes sites for marketers. | operator docs for SemrushBot ↗ |
AhrefsBot | Ahrefs | 5/ 10Medium | B2B | Ahrefs crawler - SEO tool. Builds link index. | operator docs for AhrefsBot ↗ |
YandexBot | Yandex | 6/ 10Medium | 100M+ | Yandex crawler - Russian search engine. Popular in Russia and CIS countries. | operator docs for YandexBot ↗ |
GPTBot8/ 10HighOpenAI900M+ usersCrawls content that may train OpenAI's foundation models. Blocking it keeps content out of training, but does NOT remove the site from ChatGPT search: that is OAI-SearchBot.
ChatGPT-User10/ 10CriticalOpenAI900M+ usersFetches a page when a ChatGPT or Custom GPT user asks something. User-initiated, so OpenAI says robots.txt rules may not apply.
OAI-SearchBot10/ 10CriticalOpenAI900M+ usersSurfaces sites in ChatGPT's search features. OpenAI: sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.
OAI-AdsBot4/ 10LowOpenAIAdvertisersValidates the safety of pages submitted as ads on ChatGPT. Only relevant if you advertise there.
Google-Extended10/ 10CriticalGoogle750M+ usersA robots.txt token, not a crawler: controls Gemini training and grounding in Gemini Apps and Vertex AI. Google says it does not affect inclusion in Google Search. Leaving AI Overviews is a separate Search Console control.
GoogleOther8/ 10HighGoogle750M+ usersGeneral Google crawler for AI products (NotebookLM, Vertex AI). Used across various Google AI services.
Gemini-Deep-Research8/ 10HighGoogle750M+ usersGemini Deep Research crawler for in-depth page analysis. Fetches content for advanced queries.
Google-GeminiNotebook7/ 10HighGoogle~50M usersFetcher for Gemini Notebook, Google's document analysis tool. Renamed from Google-NotebookLM on 2026-07-16; the old token still works for now.
Google-NotebookLM5/ 10MediumGoogle~50M usersFormer name of Google-GeminiNotebook, still honoured. Keep rules for both until Google retires it.
Google-Agent7/ 10HighGoogleNew (2026)Used by agents hosted on Google infrastructure to navigate the web and act on a user's request. Documented 2026-03-20 with its own IP ranges; as a user-triggered fetcher it generally ignores robots.txt.
GoogleAgent-Mariner6/ 10MediumGoogleNew (2025)Experimental Google agent for autonomous web browsing. Project Mariner.
Google-CloudVertexBot6/ 10MediumGoogleEnterpriseGoogle Cloud Vertex AI crawler for enterprise customers. B2B, smaller consumer reach.
ClaudeBot7/ 10HighAnthropic30M+ usersCollects web content that may contribute to training Anthropic's models. Blocking it excludes future material from training; search and user fetches are Claude-SearchBot and Claude-User.
Claude-User9/ 10CriticalAnthropic30M+ usersFetches a page when a Claude user asks. Anthropic says blocking it may reduce your visibility for user-directed web search.
Claude-Web3/ 10LowAnthropicLegacyLegacy token: Anthropic's current crawler documentation no longer lists it. Write rules for ClaudeBot, Claude-SearchBot and Claude-User instead.
Claude-SearchBot9/ 10CriticalAnthropic30M+ usersNavigates the web to improve Claude's search results. Anthropic says blocking it may reduce your visibility and accuracy in user search results.
anthropic-ai3/ 10LowAnthropicLegacyLegacy token: Anthropic's current crawler documentation no longer lists it.
PerplexityBot9/ 10CriticalPerplexity45M+ usersSurfaces and links sites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models.
Perplexity-User8/ 10HighPerplexity45M+ usersFetches a page when a Perplexity user asks something. Perplexity says this fetcher generally ignores robots.txt, because a user initiated the request.
Grokbot8/ 10HighxAI78M+ usersxAI (Elon Musk) crawler for Grok. Integrated with X (Twitter). Rapid growth via premium X.
unverified
xAI-Grok8/ 10HighxAI78M+ usersAlternative Grok crawler identifier. Indexes pages for Grok knowledge base.
unverified
Meta-WebIndexer8/ 10HighMeta1B+ usersNavigates the web to improve Meta AI search result quality. This is Meta's discoverability token: block it and you stay out of Meta AI answers.
Meta-ExternalAgent8/ 10HighMeta1B+ usersCrawls for use cases such as training foundation AI models, per Meta's crawler page. Not the bot that puts you inside a Meta AI answer.
Meta-ExternalFetcher8/ 10HighMeta1B+ usersFetches individual links at a user's request. Meta says it may bypass robots.txt, because the fetch is user-initiated.
Meta-ExternalAds5/ 10MediumMeta1B+ usersCrawls for use cases such as improving advertising and other business-related products. Named on Meta's crawler page, but absent from the community list, and that page cannot be machine-checked from here.
unverified
meta-externalagent8/ 10HighMeta1B+ usersLowercase form seen in logs (meta-externalagent/1.1). Same control as Meta-ExternalAgent, not a second bot.
FacebookBot5/ 10MediumMeta1B+ usersNOT on Meta's current crawler page, which documents FacebookExternalHit, Meta-ExternalAgent, Meta-ExternalFetcher, Meta-WebIndexer and Meta-ExternalAds. Kept because the community list still carries it and old rules still name it.
Applebot8/ 10HighApple2B+ usersApple crawler for Siri, Spotlight and Apple News. Apple does not use nosnippet-tagged pages as context for AI answers, and Applebot does not follow crawl-delay.
Applebot-Extended7/ 10HighApple2B+ usersDoes not crawl. A robots.txt token that opts content out of training Apple's foundation models; pages that disallow it can still appear in search results.
bingbot9/ 10CriticalMicrosoft100M+ usersMain Bing crawler. Also used by Copilot and other Microsoft AI products.
BingPreview7/ 10HighMicrosoft100M+ usersGenerates link previews in Bing/Teams/Outlook. Preview thumbnails.
unverified
Amazonbot7/ 10HighAmazon500M+ usersAmazon crawler for Alexa and Amazon Search. Answers voice queries.
AmazonBuyForMe6/ 10MediumAmazonNew (2026)New Amazon shopping agent. Autonomously browses pages for purchases.
Amzn-SearchBot6/ 10MediumAmazon500M+ usersAmazon crawler for product and information search.
DuckAssistBot7/ 10HighDuckDuckGo100M+ usersDuckDuckGo AI Assistant crawler. Privacy-first, doesn't track users.
YouBot6/ 10MediumYou.com10M+ usersYou.com crawler - AI search engine. Perplexity competitor.
PhindBot6/ 10MediumPhind5M+ usersPhind crawler - AI search for developers. Specializes in code.
Andibot5/ 10MediumAndi2M+ usersAndi Search crawler - conversational AI search engine.
Bravebot6/ 10MediumBrave70M+ usersBrave Search and Leo AI crawler. Privacy-focused browser with AI.
iAskBot5/ 10MediumiAsk.ai3M+ usersiAsk.ai crawler - free AI search engine. Popular in education.
KomoSearch4/ 10LowKomo1M+ usersKomo AI Search crawler. Smaller player in AI search market.
unverified
DeepSeekBot8/ 10HighDeepSeek97M+ usersChinese DeepSeek crawler. #4 globally, cheap model, 35% users from China.
MistralAI-User6/ 10MediumMistral5M+ usersMistral AI crawler (France). European OpenAI competitor, open-source.
cohere-ai5/ 10MediumCohereEnterpriseCohere crawler - Canadian AI company. Mainly B2B/enterprise.
cohere-training-data-crawler5/ 10MediumCohereEnterpriseCohere crawler for collecting training data.
Diffbot6/ 10MediumDiffbotEnterpriseDiffbot crawler - structures web data. B2B, knowledge graphs.
CCBot7/ 10HighCommon CrawlResearchCommon Crawl crawler - nonprofit collecting data for AI research. Used by many AI companies.
DataForSeoBot5/ 10MediumDataForSEOB2BDataForSEO crawler - SEO data for marketers and tools.
Bytespider8/ 10HighByteDance1B+ usersByteDance (TikTok, Douyin) crawler. Indexes pages for TikTok search and AI.
TikTokSpider7/ 10HighByteDance1B+ usersAlternative TikTok crawler. Fetches content for in-app display.
PetalBot6/ 10MediumHuawei500M+ usersHuawei Petal Search crawler. Popular in China and on Huawei devices.
SemrushBot5/ 10MediumSemrushB2BSemrush crawler - SEO tool. Analyzes sites for marketers.
AhrefsBot5/ 10MediumAhrefsB2BAhrefs crawler - SEO tool. Builds link index.
YandexBot6/ 10MediumYandex100M+ usersYandex crawler - Russian search engine. Popular in Russia and CIS countries.
What blocking actually does
“Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.” — OpenAI says so in its own documentation ↗
“When a site restricts ClaudeBot access, it signals that the site’s future materials should be excluded from our AI model training datasets.” — Anthropic says so in its own documentation ↗
“Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.” — OpenAI says so in its own documentation ↗
“Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” — Google says so in its own documentation ↗
Changelog
RSSWhat blocking a bot actually costs, read from the operators
GPTBot and ClaudeBot decide whether your content trains a model. OAI-SearchBot and Claude-SearchBot decide whether you appear in ChatGPT and Claude search. This table had it backwards for GPTBot, telling readers that blocking it meant vanishing from ChatGPT. Claude-Web and anthropic-ai are now marked legacy, because Anthropic no longer documents them, and three rows arrived: Google-Agent, Google-GeminiNotebook (the NotebookLM rename of 2026-07-16) and OAI-AdsBot.
Every row now says where it comes from
Each bot carries its provenance: a link to the operator page that names it, or to an independently maintained list, or the plain word "unverified" when neither could be found. Checking rather than trusting recall moved six documentation links that had quietly relocated, OpenAI and Anthropic among them.
The freshness date is measured, not typed
The footer stated February 2026 for seven months. It now comes from the change history of the bot data itself, so it ages visibly instead of standing still, and the page says nothing at all rather than substituting a deploy date it cannot vouch for.
Standards and independent lists
- RFC 9309 — Robots Exclusion Protocol ↗ — the standard behind every robots.txt rule on this page, including what a crawler is and is not obliged to honour.
- ai.robots.txt ↗ — a community-maintained block list of AI crawlers, kept independently of us — broader than this table and worth comparing against it.
Is your site ready for AI crawlers?
Run a free AlfaUMi analysis to check your robots.txt, structured data, and AI visibility score.
Analyze your websiteBot list last changed: . Data compiled from public sources. Influence scores reflect estimated impact on content visibility. Of 54 bots, 26 link to an operator page that names them and 23 more appear on an independent list; 5 we could not corroborate from either — listed, and labelled as such.