Skip to content

Do AI bots see your site? How to recognise the real ones and check what they get

AI bots visit websites for three reasons: they collect text to train models, they build an index for AI search, or they fetch a page someone has just asked an assistant about. A visit does not mean the bot saw your content, though. According to the only large study, most AI bots do not run JavaScript; some are stopped by a CDN firewall; and the bot name in your logs can be faked. Below: how to recognise a real bot, what it may miss, and how to check it yourself.

By Piotr ZientaraPublished Updated

Three kinds of AI bots

  • Training bots collect the text models learn from: for example GPTBot (OpenAI), ClaudeBot (Anthropic), Meta-ExternalAgent (Meta). Blocking them does not remove a site from AI search answers.
  • AI search bots build the index an assistant takes its sources from: OAI-SearchBot (ChatGPT search), Claude-SearchBot, PerplexityBot. To appear in AI Overviews and AI Mode, a page must be in Google’s ordinary index, which Googlebot builds. Gemini uses the same index when it grounds an answer in search results.
  • On-demand bots come when a user gives an assistant a link or asks a question your page is needed for: ChatGPT-User, Claude-User, Perplexity-User. OpenAI and Perplexity say their bots of this kind may not follow robots.txt.

Every bot, described by its operator and marked for whether it follows robots.txt, is on the AI crawlers list.

The traffic is large. Vercel, a hosting company, counted on its network in the month before December 2024: 569 million fetches by GPTBot, 370 million by Claude’s bots, 314 million by AppleBot and 24.4 million by PerplexityBot. Googlebot over the same period: 4.5 billion.

A bot’s name is a claim, not proof

Every request carries the name of the program asking (its user agent). The requester writes it, so anyone can call themselves GPTBot or Googlebot. Google warns about this in so many words in its documentation.

An example: Duane Forrester described in Search Engine Journal 14 days of logs from his new site. 33 visits introduced themselves as AI assistants; only 6 came from their operators’ addresses. Of 799 requests signed as Googlebot, 107 came from Google’s addresses. The author himself warns that this is one small site and no general conclusion follows. The lesson is general all the same: the name in your logs says who someone claims to be; the IP address says who they are.

How to check that a bot is real

Operators describe two ways:

  • A list of IP addresses. The operator publishes a file of the address ranges its bots come from. If the address in your log falls within a range, the bot is real.
  • A reverse DNS lookup. You check which name the address belongs to, then whether that name leads back to the same address. A real Googlebot has a name under googlebot.com; Applebot, under applebot.apple.com.
As documented on 19 November 2026.
OperatorList of IP addressesReverse DNS
Googlefive files: common crawlers, special crawlers, on-demand fetchers and agents kept apartyes, step by step
Appleapplebot.jsonyes
OpenAIa file for each bot, e.g. gptbot.json, searchbot.json, chatgpt-user.jsonnot described
Perplexityperplexitybot.json and perplexity-user.jsonnot described
Anthropicone list for all its bots (bots.json)not described
Metanone; it only says the IP address is more reliable than the namenot described

Perplexity advises checking the name and the address together, and always taking the current version of the file, because the ranges change. Anthropic warns that blocking its bots by IP address may not work, because the bot then cannot read robots.txt. Turning bots away is robots.txt’s job.

Where to see bot visits

  • The server log. Each line holds the IP address, the time, the page requested, the response code and the user agent. That is enough to check: take the name from the log and compare the address with the operator’s list. If the site sits behind a CDN or a proxy, your server sees the intermediary’s address. The bot’s address is then in a header the CDN adds, or in the CDN’s own logs.
  • Cloudflare. AI Crawl Control (formerly AI Audit) is on all plans. It shows which AI services fetch your site, how often and which pages. The number of visits those services sent back to the site is shown on paid plans only.
  • Vercel. The AI Bots ruleset is on all plans and off by default. It lets you block training bots and assistant bots separately.

How to tell, in your logs, a bot that reads your page because someone asked about it: see our post How to measure traffic and customers from AI.

What AI bots do not see: JavaScript

Vercel, together with the firm MERJ, analysed bot traffic on its network (a vendor study, December 2024). None of the major AI bots ran JavaScript: not OAI-SearchBot, ChatGPT-User or GPTBot (OpenAI), and not ClaudeBot (Anthropic), PerplexityBot, Meta-ExternalAgent or Bytespider (ByteDance). OpenAI’s and Anthropic’s bots fetched JavaScript files (11.5% and 23.8% of their fetches) but did not execute them. Only Gemini, which uses Googlebot’s infrastructure, and AppleBot read pages with their JavaScript.

It is the only large study on this we found. OpenAI, Anthropic and Perplexity do not say in their bot documentation whether they run JavaScript. Apple says Applebot may read a page in a browser, JavaScript included.

What follows: content that only a script adds in the browser may not exist for these bots. That covers, for instance, a product description, a price, or an article in an app that does not render pages on the server. Content in the HTML the server returns is safe.

Google: the ordinary search engine’s rules

Google has no separate requirements for AI Overviews and AI Mode. A page must be in the index and eligible to appear in results with a snippet. Google says there are no other technical requirements. Google Search runs JavaScript in a current version of the Chromium browser. Google adds two caveats, though. Content that is not in the rendered HTML will not be indexed. And Search does not click or scroll, so content that loads only after a click or a scroll may be missed.

A block you don’t know about: the CDN and the firewall

Google, describing AI features in Search, reminds site owners that crawlers must be allowed not only by robots.txt but also by the CDN and the hosting. A bot can be invited in robots.txt and turned away by the firewall.

Cloudflare has changed its approach here. On 1 July 2025 it announced that every new domain chooses at sign-up whether to let AI bots in. According to its documentation of 2 November 2026, new domains let search bots, agents and training bots in by default. The setting is easy to change and easy to forget, so check yours: Security > Settings > AI Crawler and Agent access policies.

Our bot access test fetches your site as each bot and shows where robots.txt and the server disagree. It asks from our servers, so a firewall that checks IP addresses may treat the real bot differently.

How to check what a bot gets

  1. Look at the page without JavaScript. Open the page source in your browser, or switch JavaScript off, and look for the sentences that matter most: your offer, your prices, your answers to common questions. If they are not there, bots that do not run scripts will not see them either.
  2. Check what Google sees. In Search Console, the URL Inspection tool and its “View crawled page” show the HTML and a screenshot of what Google saw. That is Google’s view, not OpenAI’s or Anthropic’s.
  3. Run the free analysis. The AlfaUMi analyser reads a page without running its scripts: the HTML and structured data only. What it cannot see, bots that do not run JavaScript most likely cannot see either.
  4. Check access. The bot access test shows whether your server and CDN answer each bot at all.

What this data does not tell you

  • The JavaScript study is almost two years old, was done by a hosting company and covered sites on its network. Bots may have changed since, and their operators do not say.
  • The fake-bot example is one small site over 14 days. We found no independent measurement at scale; the figures going around come from companies that sell bot protection.
  • Google does not say outright that AI Overviews read the rendered version of a page. It says the page must meet Search’s requirements.
  • A bot’s visit is not a citation. A bot can fetch a page and use it in no answer at all. Whether AI mentions a brand and links to its site is measured separately: how we measure AI visibility.

Read more: AI Crawlers

Check your own site

The free analysis shows in a minute how search engines and AI assistants see your site.

Analyze free

Frequently asked questions

Does ChatGPT see content loaded by JavaScript?

According to the Vercel and MERJ study of December 2024, OpenAI’s bots (GPTBot, OAI-SearchBot and ChatGPT-User) fetched pages without running JavaScript. OpenAI does not describe this in its documentation. The safest course is to have your key content in the HTML the server returns.

How do I check that a GPTBot visit is real?

Compare the IP address in your log with the ranges in openai.com/gptbot.json. The name GPTBot in a log proves nothing on its own, because anyone can write it.

Does Cloudflare block AI bots?

It depends on the domain’s settings. According to Cloudflare’s documentation of November 2026, new domains let AI bots in by default, but every owner can change that. Check your domain’s settings under Security > Settings.

Sources