Lumi GEO

Original research

AI crawler blocking statistics (2026): which AI bots popular websites shut out

Updated

The 2026 Lumi GEO AI Crawler Access Index found that 23.9% of the world's most popular websites block at least one AI crawler in robots.txt, based on 658 sites of the Tranco top 1,000 checked on November 10, 2026. GPTBot, OpenAI's training crawler, is blocked by 17.2% of them. OAI-SearchBot, the crawler behind ChatGPT search, is blocked by 9.7%.

Research by Mattéo Springmann, who makes Lumi GEO, the Chrome extension mentioned at the end. Every number of the index comes from one measurement made on November 10, 2026. The method, its limits and the source of every outside figure are further down. This page is updated each time the study is run again.

Key AI crawler statistics for 2026

  • 23.9% of popular websites block at least one AI crawler in robots.txt: 157 of 658 sites (Lumi GEO AI Crawler Access Index 2026).
  • 19.5% of popular websites block an AI crawler by name, with a robots.txt rule written for that crawler (Lumi GEO AI Crawler Access Index 2026).
  • 17.2% of popular websites block GPTBot, OpenAI's training crawler (Lumi GEO AI Crawler Access Index 2026).
  • 9.7% of popular websites block OAI-SearchBot, which keeps them out of ChatGPT search answers (Lumi GEO AI Crawler Access Index 2026).
  • 7.4% of popular websites block GPTBot but still allow OAI-SearchBot: out of training, still in ChatGPT search (Lumi GEO AI Crawler Access Index 2026).
  • Bytespider is the most blocked AI crawler: 19.8% of popular websites block it (Lumi GEO AI Crawler Access Index 2026).
  • 6.5% of popular websites shut out every crawler that has no rule of its own, AI or not (Lumi GEO AI Crawler Access Index 2026).
  • 18.2% of popular websites publish an llms.txt file: 114 of 628 sites (Lumi GEO AI Crawler Access Index 2026).

How many websites block AI crawlers in 2026?

157 of the 658 popular websites measured block at least one of the 14 AI crawlers of the index, which is 23.9%. "Block" means that robots.txt forbids the home page, and so the whole site, to that crawler.

Not every block is aimed at AI. 128 sites (19.5%) block an AI crawler with a rule that names it. The others block AI crawlers as a side effect: 43 sites (6.5%) use User-agent: * with Disallow: /, which shuts out every crawler that has no group of its own.

  • 23.4% of popular websites block at least one AI training crawler (154 sites).
  • 14.7% of popular websites block at least one AI search crawler (97 sites).
  • 9.4% of popular websites block all three AI search crawlers of the index: OAI-SearchBot, PerplexityBot and Claude-SearchBot (62 sites).
  • 27.2% of popular websites mention at least one AI crawler by name in robots.txt, to block it or to allow it (179 sites).

Which AI crawlers are blocked the most?

Bytespider is the most blocked AI crawler in the 2026 Lumi GEO AI Crawler Access Index, at 19.8% of popular websites. The table lists the 14 crawlers, from the most blocked to the least. "Blocked by name" counts the sites whose robots.txt has a group written for that crawler.

Crawler (robots.txt token)CompanyWhat it is forSites blocking itShareBlocked by name
BytespiderByteDanceTraining13019.8%92
CCBotCommon CrawlTraining12719.3%90
ClaudeBotAnthropicTraining11417.3%84
GPTBotOpenAITraining11317.2%85
meta-externalagentMetaTraining10516%70
Google-ExtendedGoogleTraining9714.7%69
Applebot-ExtendedAppleTraining9714.7%65
PerplexityBotPerplexityAI search9314.1%65
Perplexity-UserPerplexityFetch asked by a user8112.3%42
Claude-UserAnthropicFetch asked by a user7511.4%42
Claude-SearchBotAnthropicAI search7411.2%40
ChatGPT-UserOpenAIFetch asked by a user7311.1%43
MistralAI-UserMistral AIFetch asked by a user7311.1%33
OAI-SearchBotOpenAIAI search649.7%34

How many websites block GPTBot?

113 of 658 popular websites block GPTBot in robots.txt, which is 17.2%. 85 of them name GPTBot in a rule; the others block it through a rule for all crawlers.

GPTBot collects content that may be used to train OpenAI's models. It is not the crawler that decides whether a site appears in ChatGPT search: that one is OAI-SearchBot.

How many websites block ChatGPT search (OAI-SearchBot)?

64 of 658 popular websites block OAI-SearchBot, which is 9.7%. OpenAI's crawler documentation says that sites which opt out of OAI-SearchBot are not shown in ChatGPT search answers, although they can still appear as navigational links.

34 of these sites name OAI-SearchBot in a rule. The others never mention it: a rule for all crawlers blocks it.

Do websites block AI training but allow AI search?

Some do. 49 popular websites (7.4%) block GPTBot and still allow OAI-SearchBot. OpenAI documents this choice: the two settings are independent, so a site can stay out of training and stay in ChatGPT search.

More sites block both: 64 popular websites (9.7%) block GPTBot and OAI-SearchBot together. Whether each one of them means to leave ChatGPT search cannot be read from a robots.txt file.

Across the 14 crawlers, training crawlers are blocked more often than search crawlers: GPTBot at 17.2% against OAI-SearchBot at 9.7%, and ClaudeBot at 17.3% against Claude-SearchBot at 11.2%.

How many websites have an llms.txt file?

114 of 628 popular websites publish an llms.txt file, which is 18.2%. A file is counted when /llms.txt answers with real text that starts with a Markdown title, the one element the llmstxt.org proposal requires.

llms.txt is a community proposal, not an official standard. Having one is not a guarantee of being cited by an AI.

What other research found

Each figure below belongs to the organisation that measured it.

  • HasData, AI Crawler Block Index (last updated October 26, 2026): 10.3% of 9,746 sites of the Tranco top 10,000 block at least one AI crawler in robots.txt, and 7.9% block GPTBot. Among 1,148 news publishers, 56.4% and 50.5% do.
  • HasData, same study: of 592 tested sites whose robots.txt disallows GPTBot, 234 (39.5%) still answered a GPTBot request with a normal page.
  • Ahrefs, Monika Butterweck (September 18, 2026): 2.27% of 24,550 German, Austrian and Swiss hosts block GPTBot in robots.txt, and 1.22% name it.
  • Originality.ai, reported by Danny Goodwin in Search Engine Land (September 2023): 242 of the top 1,000 websites blocked GPTBot, with 67 robots.txt files not inspected.
  • Vercel and MERJ (December 17, 2024): none of the major AI crawlers rendered JavaScript in their measurements.

These studies use different lists of sites, dates and counting rules, so their percentages cannot be compared with this index line by line. Read together, they point the same way: the more popular the sample, the larger the share of sites that block GPTBot.

How the index is measured

The sample is the first 1,000 domains of the Tranco list, a public ranking of popular domains built for research, downloaded on November 10, 2026.

  1. For each domain, a script requests /robots.txt and /llms.txt over HTTPS, on the bare domain and then on www. if the first gives no answer. It identifies itself as LumiGEO-Study/1.0, waits up to 12 seconds, and tries silent domains a second time.
  2. A site enters the base when its robots.txt request gives a usable answer: a real robots.txt (514 sites) or no robots.txt at all (144 sites), which means every crawler is allowed under RFC 9309. That makes 658 sites.
  3. A crawler counts as blocked when the rules that apply to it forbid the home page (/). The rules are chosen as RFC 9309 says: the group that names the crawler, otherwise the User-agent: * group. A rule on one folder only does not count.
  4. The robots.txt reader and the list of 14 crawlers are the ones of the Lumi GEO extension.

274 domains gave no answer. 197 of them do not lead to any server: the name is infrastructure, with no website on it. 65 refused the request (HTTP 401, 403 or 429) and 3 answered with a server error. These are left out of every percentage, because their answer says nothing about their rules for AI crawlers.

Limits of these numbers

  • robots.txt is a declaration, not a lock. A site can block a crawler at its firewall or CDN without a word in robots.txt, or forbid a crawler in robots.txt and still serve it. This index only reads what sites declare.
  • The top 1,000 is not the whole web. It holds large brands, platforms and technical domains, and several domains belong to the same company.
  • It is one measurement, on one day, from one network in Spain. Sites that refused the script may answer a real AI crawler differently.

How to cite these statistics

Cite the study by its name, with the year: "Lumi GEO AI Crawler Access Index 2026", and link to this page. Example: "17.2% of popular websites block GPTBot in robots.txt (Lumi GEO AI Crawler Access Index 2026, 658 sites)."

The full data is public: download the result for each of the 1,000 domains (CSV). Questions about the method: contact@lumigeo.app.

Checking one site

The same check can be done by hand on any site: can ChatGPT see your website? explains it in three free steps. Lumi GEO, the Chrome extension I make, does it in one click and reads robots.txt with the same rules as this index.

Frequently asked questions

What percentage of websites block AI crawlers?

23.9% of popular websites block at least one AI crawler in robots.txt, according to the Lumi GEO AI Crawler Access Index 2026 (157 of 658 sites of the Tranco top 1,000, November 2026). 19.5% do it with a rule that names an AI crawler.

Which AI crawler is blocked the most?

Bytespider, blocked by 19.8% of popular websites in the Lumi GEO AI Crawler Access Index 2026. GPTBot is blocked by 17.2% and ClaudeBot by 17.3%.

Does blocking GPTBot remove a site from ChatGPT search?

No. OpenAI's documentation says GPTBot and OAI-SearchBot are controlled separately, and only OAI-SearchBot decides whether a site is shown in ChatGPT search answers. In the Lumi GEO AI Crawler Access Index 2026, 7.4% of popular websites block GPTBot while allowing OAI-SearchBot.

How many websites have an llms.txt file?

18.2% of popular websites: 114 of 628 sites of the Tranco top 1,000 in the Lumi GEO AI Crawler Access Index 2026. llms.txt is a community proposal, not an official standard.

Sources