AI Crawlers List

Main OpenAI crawler for GPT model training. Indexes pages for ChatGPT knowledge base. Blocking = absent from ChatGPT results.

Fetches pages on ChatGPT user request (Browse with Bing). Triggered by human clicking a link.

Crawler for ChatGPT Search (Google competitor). Indexes pages for ChatGPT search results.

Google crawler for Gemini and Bard training. Separate from Googlebot. Blocking doesn't affect Search indexing.

General Google crawler for AI products (NotebookLM, Vertex AI). Used across various Google AI services.

Gemini Deep Research crawler for in-depth page analysis. Fetches content for advanced queries.

NotebookLM crawler - Google's document analysis and AI podcast creation tool.

Experimental Google agent for autonomous web browsing. Project Mariner.

Google Cloud Vertex AI crawler for enterprise customers. B2B, smaller consumer reach.

Main Anthropic crawler for source citation in Claude responses. Fetches pages for context.

Fetches pages on Claude user request (Browse feature). User-initiated.

Anthropic crawler for page indexing. Builds Claude knowledge base.

Crawler for Claude Search - Anthropic's new search feature.

General Anthropic crawler identifier. Used for model training.

Main Perplexity AI crawler. Indexes pages for AI search results. Fastest growing AI search.

Fetches pages on Perplexity user request. User-initiated, often ignores robots.txt.

xAI (Elon Musk) crawler for Grok. Integrated with X (Twitter). Rapid growth via premium X.

Alternative Grok crawler identifier. Indexes pages for Grok knowledge base.

Meta crawler for Facebook/Instagram previews and Meta AI. Used for Llama training.

Meta AI crawler for fetching pages for WhatsApp/Messenger/Instagram users.

Fetches content for Meta AI. Active in WhatsApp (63% of Meta AI traffic).

Lowercase variant of Meta-ExternalAgent. Some systems use lowercase.

Apple crawler for Siri, Spotlight and Apple News. Used across all Apple devices.

Extended Apple crawler for Apple Intelligence training. New since iOS 18.

Main Bing crawler. Also used by Copilot and other Microsoft AI products.

Generates link previews in Bing/Teams/Outlook. Preview thumbnails.

Amazon crawler for Alexa and Amazon Search. Answers voice queries.

New Amazon shopping agent. Autonomously browses pages for purchases.

Amazon crawler for product and information search.

DuckDuckGo AI Assistant crawler. Privacy-first, doesn't track users.

You.com crawler - AI search engine. Perplexity competitor.

Phind crawler - AI search for developers. Specializes in code.

Andi Search crawler - conversational AI search engine.

Brave Search and Leo AI crawler. Privacy-focused browser with AI.

iAsk.ai crawler - free AI search engine. Popular in education.

Komo AI Search crawler. Smaller player in AI search market.

Chinese DeepSeek crawler. #4 globally, cheap model, 35% users from China.

Mistral AI crawler (France). European OpenAI competitor, open-source.

Cohere crawler - Canadian AI company. Mainly B2B/enterprise.

Cohere crawler for collecting training data.

Diffbot crawler - structures web data. B2B, knowledge graphs.

Common Crawl crawler - nonprofit collecting data for AI research. Used by many AI companies.

DataForSEO crawler - SEO data for marketers and tools.

ByteDance (TikTok, Douyin) crawler. Indexes pages for TikTok search and AI.

Alternative TikTok crawler. Fetches content for in-app display.

Huawei Petal Search crawler. Popular in China and on Huawei devices.

Semrush crawler - SEO tool. Analyzes sites for marketers.

Ahrefs crawler - SEO tool. Builds link index.

Yandex crawler - Russian search engine. Popular in Russia and CIS countries.