AI Crawlers List
54 bots from 27 companies are crawling the web to train AI models, power search, and fetch content for users. Know who visits your site and what they do with your content.
Want to control which bots access your site?
Use robots.txt rules , generate the rules, or run an AlfaUMi analysis to see which crawlers currently access your website.
| Bot Name | Company | Influence | Monthly Users | Description | Source |
|---|---|---|---|---|---|
GPTBot | OpenAI | 8/ 10High | 900M+ | Crawls content that may train OpenAI's foundation models. Blocking it keeps content out of training, but does NOT remove the site from ChatGPT search: that is OAI-SearchBot. | operator docs for GPTBot ↗ |
ChatGPT-User | OpenAI | 10/ 10Critical | 900M+ | Fetches a page when a ChatGPT or Custom GPT user asks something. User-initiated, so OpenAI says robots.txt rules may not apply. | operator docs for ChatGPT-User ↗ |
OAI-SearchBot | OpenAI | 10/ 10Critical | 900M+ | Surfaces sites in ChatGPT's search features. OpenAI: sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. | operator docs for OAI-SearchBot ↗ |
OAI-AdsBot | OpenAI | 4/ 10Low | Advertisers | Validates the safety of pages submitted as ads on ChatGPT. Only relevant if you advertise there. | operator docs for OAI-AdsBot ↗ |
Google-Extended | 10/ 10Critical | 750M+ | A robots.txt token, not a crawler: controls Gemini training and grounding in Gemini Apps and Vertex AI. Google says it does not affect inclusion in Google Search. Leaving AI Overviews is a separate Search Console control. | operator docs for Google-Extended ↗ | |
GoogleOther | 8/ 10High | 750M+ | General Google crawler for AI products (NotebookLM, Vertex AI). Used across various Google AI services. | operator docs for GoogleOther ↗ | |
Gemini-Deep-Research | 8/ 10High | 750M+ | Gemini Deep Research crawler for in-depth page analysis. Fetches content for advanced queries. | independent list lists Gemini-Deep-Research ↗ | |
Google-GeminiNotebook | 7/ 10High | ~50M | Fetcher for Gemini Notebook, Google's document analysis tool. Renamed from Google-NotebookLM on 2026-07-16; the old token still works for now. | operator docs for Google-GeminiNotebook ↗ | |
Google-NotebookLM | 5/ 10Medium | ~50M | Former name of Google-GeminiNotebook, still honoured. Keep rules for both until Google retires it. | operator docs for Google-NotebookLM ↗ | |
Google-Agent | 7/ 10High | New (2026) | Used by agents hosted on Google infrastructure to navigate the web and act on a user's request. Documented 2026-03-20 with its own IP ranges; as a user-triggered fetcher it generally ignores robots.txt. | operator docs for Google-Agent ↗ | |
GoogleAgent-Mariner | 6/ 10Medium | New (2025) | Experimental Google agent for autonomous web browsing. Project Mariner. | independent list lists GoogleAgent-Mariner ↗ | |
Google-CloudVertexBot | 6/ 10Medium | Enterprise | Google Cloud Vertex AI crawler for enterprise customers. B2B, smaller consumer reach. | operator docs for Google-CloudVertexBot ↗ | |
ClaudeBot | Anthropic | 7/ 10High | 30M+ | Collects web content that may contribute to training Anthropic's models. Blocking it excludes future material from training; search and user fetches are Claude-SearchBot and Claude-User. | operator docs for ClaudeBot ↗ |
Claude-User | Anthropic | 9/ 10Critical | 30M+ | Fetches a page when a Claude user asks. Anthropic says blocking it may reduce your visibility for user-directed web search. | operator docs for Claude-User ↗ |
Claude-Web | Anthropic | 3/ 10Low | Legacy | Legacy token: Anthropic's current crawler documentation no longer lists it. Write rules for ClaudeBot, Claude-SearchBot and Claude-User instead. | independent list lists Claude-Web ↗ |
Claude-SearchBot | Anthropic | 9/ 10Critical | 30M+ | Navigates the web to improve Claude's search results. Anthropic says blocking it may reduce your visibility and accuracy in user search results. | operator docs for Claude-SearchBot ↗ |
anthropic-ai | Anthropic | 3/ 10Low | Legacy | Legacy token: Anthropic's current crawler documentation no longer lists it. | independent list lists anthropic-ai ↗ |
PerplexityBot | Perplexity | 9/ 10Critical | 45M+ | Surfaces and links sites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models. | operator docs for PerplexityBot ↗ |
Perplexity-User | Perplexity | 8/ 10High | 45M+ | Fetches a page when a Perplexity user asks something. Perplexity says this fetcher generally ignores robots.txt, because a user initiated the request. | operator docs for Perplexity-User ↗ |
Grokbot | xAI | 8/ 10High | 78M+ | xAI (Elon Musk) crawler for Grok. Integrated with X (Twitter). Rapid growth via premium X. | unverified |
xAI-Grok | xAI | 8/ 10High | 78M+ | Alternative Grok crawler identifier. Indexes pages for Grok knowledge base. | unverified |
Meta-WebIndexer | Meta | 8/ 10High | 1B+ | Navigates the web to improve Meta AI search result quality. This is Meta's discoverability token: block it and you stay out of Meta AI answers. | independent list lists Meta-WebIndexer ↗ |
Meta-ExternalAgent | Meta | 8/ 10High | 1B+ | Crawls for use cases such as training foundation AI models, per Meta's crawler page. Not the bot that puts you inside a Meta AI answer. | independent list lists Meta-ExternalAgent ↗ |
Meta-ExternalFetcher | Meta | 8/ 10High | 1B+ | Fetches individual links at a user's request. Meta says it may bypass robots.txt, because the fetch is user-initiated. | independent list lists Meta-ExternalFetcher ↗ |
Meta-ExternalAds | Meta | 5/ 10Medium | 1B+ | Crawls for use cases such as improving advertising and other business-related products. Named on Meta's crawler page, but absent from the community list, and that page cannot be machine-checked from here. | unverified |
meta-externalagent | Meta | 8/ 10High | 1B+ | Lowercase form seen in logs (meta-externalagent/1.1). Same control as Meta-ExternalAgent, not a second bot. | independent list lists meta-externalagent ↗ |
FacebookBot | Meta | 5/ 10Medium | 1B+ | NOT on Meta's current crawler page, which documents FacebookExternalHit, Meta-ExternalAgent, Meta-ExternalFetcher, Meta-WebIndexer and Meta-ExternalAds. Kept because the community list still carries it and old rules still name it. | independent list lists FacebookBot ↗ |
Applebot | Apple | 8/ 10High | 2B+ | Apple crawler for Siri, Spotlight and Apple News. Apple does not use nosnippet-tagged pages as context for AI answers, and Applebot does not follow crawl-delay. | operator docs for Applebot ↗ |
Applebot-Extended | Apple | 7/ 10High | 2B+ | Does not crawl. A robots.txt token that opts content out of training Apple's foundation models; pages that disallow it can still appear in search results. | operator docs for Applebot-Extended ↗ |
bingbot | Microsoft | 9/ 10Critical | 100M+ | Main Bing crawler. Also used by Copilot and other Microsoft AI products. | operator docs for bingbot ↗ |
BingPreview | Microsoft | 7/ 10High | 100M+ | Generates link previews in Bing/Teams/Outlook. Preview thumbnails. | unverified |
Amazonbot | Amazon | 7/ 10High | 500M+ | Amazon crawler for Alexa and Amazon Search. Answers voice queries. | operator docs for Amazonbot ↗ |
AmazonBuyForMe | Amazon | 6/ 10Medium | New (2026) | New Amazon shopping agent. Autonomously browses pages for purchases. | independent list lists AmazonBuyForMe ↗ |
Amzn-SearchBot | Amazon | 6/ 10Medium | 500M+ | Amazon crawler for product and information search. | operator docs for Amzn-SearchBot ↗ |
DuckAssistBot | DuckDuckGo | 7/ 10High | 100M+ | DuckDuckGo AI Assistant crawler. Privacy-first, doesn't track users. | operator docs for DuckAssistBot ↗ |
YouBot | You.com | 6/ 10Medium | 10M+ | You.com crawler - AI search engine. Perplexity competitor. | independent list lists YouBot ↗ |
PhindBot | Phind | 6/ 10Medium | 5M+ | Phind crawler - AI search for developers. Specializes in code. | independent list lists PhindBot ↗ |
Andibot | Andi | 5/ 10Medium | 2M+ | Andi Search crawler - conversational AI search engine. | independent list lists Andibot ↗ |
Bravebot | Brave | 6/ 10Medium | 70M+ | Brave Search and Leo AI crawler. Privacy-focused browser with AI. | independent list lists Bravebot ↗ |
iAskBot | iAsk.ai | 5/ 10Medium | 3M+ | iAsk.ai crawler - free AI search engine. Popular in education. | independent list lists iAskBot ↗ |
KomoSearch | Komo | 4/ 10Low | 1M+ | Komo AI Search crawler. Smaller player in AI search market. | unverified |
DeepSeekBot | DeepSeek | 8/ 10High | 97M+ | Chinese DeepSeek crawler. #4 globally, cheap model, 35% users from China. | independent list lists DeepSeekBot ↗ |
MistralAI-User | Mistral | 6/ 10Medium | 5M+ | Mistral AI crawler (France). European OpenAI competitor, open-source. | independent list lists MistralAI-User ↗ |
cohere-ai | Cohere | 5/ 10Medium | Enterprise | Cohere crawler - Canadian AI company. Mainly B2B/enterprise. | independent list lists cohere-ai ↗ |
cohere-training-data-crawler | Cohere | 5/ 10Medium | Enterprise | Cohere crawler for collecting training data. | independent list lists cohere-training-data-crawler ↗ |
Diffbot | Diffbot | 6/ 10Medium | Enterprise | Diffbot crawler - structures web data. B2B, knowledge graphs. | independent list lists Diffbot ↗ |
CCBot | Common Crawl | 7/ 10High | Research | Common Crawl crawler - nonprofit collecting data for AI research. Used by many AI companies. | operator docs for CCBot ↗ |
DataForSeoBot | DataForSEO | 5/ 10Medium | B2B | DataForSEO crawler - SEO data for marketers and tools. | operator docs for DataForSeoBot ↗ |
Bytespider | ByteDance | 8/ 10High | 1B+ | ByteDance (TikTok, Douyin) crawler. Indexes pages for TikTok search and AI. | independent list lists Bytespider ↗ |
TikTokSpider | ByteDance | 7/ 10High | 1B+ | Alternative TikTok crawler. Fetches content for in-app display. | independent list lists TikTokSpider ↗ |
PetalBot | Huawei | 6/ 10Medium | 500M+ | Huawei Petal Search crawler. Popular in China and on Huawei devices. | independent list lists PetalBot ↗ |
SemrushBot | Semrush | 5/ 10Medium | B2B | Semrush crawler - SEO tool. Analyzes sites for marketers. | operator docs for SemrushBot ↗ |
AhrefsBot | Ahrefs | 5/ 10Medium | B2B | Ahrefs crawler - SEO tool. Builds link index. | operator docs for AhrefsBot ↗ |
YandexBot | Yandex | 6/ 10Medium | 100M+ | Yandex crawler - Russian search engine. Popular in Russia and CIS countries. | operator docs for YandexBot ↗ |
GPTBot8/ 10HighCrawls content that may train OpenAI's foundation models. Blocking it keeps content out of training, but does NOT remove the site from ChatGPT search: that is OAI-SearchBot.
ChatGPT-User10/ 10CriticalFetches a page when a ChatGPT or Custom GPT user asks something. User-initiated, so OpenAI says robots.txt rules may not apply.
OAI-SearchBot10/ 10CriticalSurfaces sites in ChatGPT's search features. OpenAI: sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.
OAI-AdsBot4/ 10LowValidates the safety of pages submitted as ads on ChatGPT. Only relevant if you advertise there.
Google-Extended10/ 10CriticalA robots.txt token, not a crawler: controls Gemini training and grounding in Gemini Apps and Vertex AI. Google says it does not affect inclusion in Google Search. Leaving AI Overviews is a separate Search Console control.
GoogleOther8/ 10HighGeneral Google crawler for AI products (NotebookLM, Vertex AI). Used across various Google AI services.
Gemini-Deep-Research8/ 10HighGemini Deep Research crawler for in-depth page analysis. Fetches content for advanced queries.
Google-GeminiNotebook7/ 10HighFetcher for Gemini Notebook, Google's document analysis tool. Renamed from Google-NotebookLM on 2026-07-16; the old token still works for now.
Google-NotebookLM5/ 10MediumFormer name of Google-GeminiNotebook, still honoured. Keep rules for both until Google retires it.
Google-Agent7/ 10HighUsed by agents hosted on Google infrastructure to navigate the web and act on a user's request. Documented 2026-03-20 with its own IP ranges; as a user-triggered fetcher it generally ignores robots.txt.
GoogleAgent-Mariner6/ 10MediumExperimental Google agent for autonomous web browsing. Project Mariner.
Google-CloudVertexBot6/ 10MediumGoogle Cloud Vertex AI crawler for enterprise customers. B2B, smaller consumer reach.
ClaudeBot7/ 10HighCollects web content that may contribute to training Anthropic's models. Blocking it excludes future material from training; search and user fetches are Claude-SearchBot and Claude-User.
Claude-User9/ 10CriticalFetches a page when a Claude user asks. Anthropic says blocking it may reduce your visibility for user-directed web search.
Claude-Web3/ 10LowLegacy token: Anthropic's current crawler documentation no longer lists it. Write rules for ClaudeBot, Claude-SearchBot and Claude-User instead.
Claude-SearchBot9/ 10CriticalNavigates the web to improve Claude's search results. Anthropic says blocking it may reduce your visibility and accuracy in user search results.
anthropic-ai3/ 10LowLegacy token: Anthropic's current crawler documentation no longer lists it.
PerplexityBot9/ 10CriticalSurfaces and links sites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models.
Perplexity-User8/ 10HighFetches a page when a Perplexity user asks something. Perplexity says this fetcher generally ignores robots.txt, because a user initiated the request.
Grokbot8/ 10HighxAI (Elon Musk) crawler for Grok. Integrated with X (Twitter). Rapid growth via premium X.
unverified
xAI-Grok8/ 10HighAlternative Grok crawler identifier. Indexes pages for Grok knowledge base.
unverified
Meta-WebIndexer8/ 10HighNavigates the web to improve Meta AI search result quality. This is Meta's discoverability token: block it and you stay out of Meta AI answers.
Meta-ExternalAgent8/ 10HighCrawls for use cases such as training foundation AI models, per Meta's crawler page. Not the bot that puts you inside a Meta AI answer.
Meta-ExternalFetcher8/ 10HighFetches individual links at a user's request. Meta says it may bypass robots.txt, because the fetch is user-initiated.
Meta-ExternalAds5/ 10MediumCrawls for use cases such as improving advertising and other business-related products. Named on Meta's crawler page, but absent from the community list, and that page cannot be machine-checked from here.
unverified
meta-externalagent8/ 10HighLowercase form seen in logs (meta-externalagent/1.1). Same control as Meta-ExternalAgent, not a second bot.
FacebookBot5/ 10MediumNOT on Meta's current crawler page, which documents FacebookExternalHit, Meta-ExternalAgent, Meta-ExternalFetcher, Meta-WebIndexer and Meta-ExternalAds. Kept because the community list still carries it and old rules still name it.
Applebot8/ 10HighApple crawler for Siri, Spotlight and Apple News. Apple does not use nosnippet-tagged pages as context for AI answers, and Applebot does not follow crawl-delay.
Applebot-Extended7/ 10HighDoes not crawl. A robots.txt token that opts content out of training Apple's foundation models; pages that disallow it can still appear in search results.
bingbot9/ 10CriticalMain Bing crawler. Also used by Copilot and other Microsoft AI products.
BingPreview7/ 10HighGenerates link previews in Bing/Teams/Outlook. Preview thumbnails.
unverified
Amazonbot7/ 10HighAmazon crawler for Alexa and Amazon Search. Answers voice queries.
AmazonBuyForMe6/ 10MediumNew Amazon shopping agent. Autonomously browses pages for purchases.
Amzn-SearchBot6/ 10MediumAmazon crawler for product and information search.
DuckAssistBot7/ 10HighDuckDuckGo AI Assistant crawler. Privacy-first, doesn't track users.
YouBot6/ 10MediumYou.com crawler - AI search engine. Perplexity competitor.
PhindBot6/ 10MediumPhind crawler - AI search for developers. Specializes in code.
Andibot5/ 10MediumAndi Search crawler - conversational AI search engine.
Bravebot6/ 10MediumBrave Search and Leo AI crawler. Privacy-focused browser with AI.
iAskBot5/ 10MediumiAsk.ai crawler - free AI search engine. Popular in education.
KomoSearch4/ 10LowKomo AI Search crawler. Smaller player in AI search market.
unverified
DeepSeekBot8/ 10HighChinese DeepSeek crawler. #4 globally, cheap model, 35% users from China.
MistralAI-User6/ 10MediumMistral AI crawler (France). European OpenAI competitor, open-source.
cohere-ai5/ 10MediumCohere crawler - Canadian AI company. Mainly B2B/enterprise.
cohere-training-data-crawler5/ 10MediumCohere crawler for collecting training data.
Diffbot6/ 10MediumDiffbot crawler - structures web data. B2B, knowledge graphs.
CCBot7/ 10HighCommon Crawl crawler - nonprofit collecting data for AI research. Used by many AI companies.
DataForSeoBot5/ 10MediumDataForSEO crawler - SEO data for marketers and tools.
Bytespider8/ 10HighByteDance (TikTok, Douyin) crawler. Indexes pages for TikTok search and AI.
TikTokSpider7/ 10HighAlternative TikTok crawler. Fetches content for in-app display.
PetalBot6/ 10MediumHuawei Petal Search crawler. Popular in China and on Huawei devices.
SemrushBot5/ 10MediumSemrush crawler - SEO tool. Analyzes sites for marketers.
AhrefsBot5/ 10MediumAhrefs crawler - SEO tool. Builds link index.
YandexBot6/ 10MediumYandex crawler - Russian search engine. Popular in Russia and CIS countries.
What blocking actually does
“Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.” — OpenAI says so in its own documentation ↗
“When a site restricts ClaudeBot access, it signals that the site’s future materials should be excluded from our AI model training datasets.” — Anthropic says so in its own documentation ↗
“Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.” — OpenAI says so in its own documentation ↗
“Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” — Google says so in its own documentation ↗
Changelog
RSSWhat blocking a bot actually costs, read from the operators
GPTBot and ClaudeBot decide whether your content trains a model. OAI-SearchBot and Claude-SearchBot decide whether you appear in ChatGPT and Claude search. This table had it backwards for GPTBot, telling readers that blocking it meant vanishing from ChatGPT. Claude-Web and anthropic-ai are now marked legacy, because Anthropic no longer documents them, and three rows arrived: Google-Agent, Google-GeminiNotebook (the NotebookLM rename of 2026-07-16) and OAI-AdsBot.
Every row now says where it comes from
Each bot carries its provenance: a link to the operator page that names it, or to an independently maintained list, or the plain word "unverified" when neither could be found. Checking rather than trusting recall moved six documentation links that had quietly relocated, OpenAI and Anthropic among them.
The freshness date is measured, not typed
The footer stated February 2026 for seven months. It now comes from the change history of the bot data itself, so it ages visibly instead of standing still, and the page says nothing at all rather than substituting a deploy date it cannot vouch for.
Standards and independent lists
- RFC 9309 — Robots Exclusion Protocol ↗ — the standard behind every robots.txt rule on this page, including what a crawler is and is not obliged to honour.
- ai.robots.txt ↗ — a community-maintained block list of AI crawlers, kept independently of us — broader than this table and worth comparing against it.
Is your site ready for AI crawlers?
Run a free AlfaUMi analysis to check your robots.txt, structured data, and AI visibility score.
Analyze your websiteBot list last changed: . Data compiled from public sources. Influence scores reflect estimated impact on content visibility. Of 54 bots, 26 link to an operator page that names them and 23 more appear on an independent list; 5 we could not corroborate from either — listed, and labelled as such.