Lumi GEO

Guide

Can ChatGPT see your website? A 5-minute check for site owners

Updated

For your pages to appear in ChatGPT's search answers, three things must be true: your robots.txt lets OAI-SearchBot in, your host or CDN does not turn OpenAI's crawler away, and your text is in the HTML your server sends rather than added later by JavaScript. Each check takes a minute or two with free tools.

Written by Mattéo Springmann, who makes Lumi GEO, the Chrome extension mentioned at the end. Every check below works without it. Facts checked on November 7, 2026 against OpenAI's and Cloudflare's own documentation (sources at the bottom).

Which OpenAI robot decides whether you show up in ChatGPT search?

OAI-SearchBot. OpenAI's crawler documentation says that sites which opt out of OAI-SearchBot are not shown in ChatGPT search answers, although they can still appear as navigational links. OpenAI runs three other robots with different jobs, and each one is controlled separately:

Robot (user agent)What it doesWhat blocking it in robots.txt changes
OAI-SearchBotFinds pages for ChatGPT's search answersYour pages stop appearing in search answers
GPTBotCollects content that may be used to train OpenAI's modelsYou opt out of training, not out of search
ChatGPT-UserOpens a page when someone asks ChatGPT about itPossibly nothing: OpenAI says robots.txt rules may not apply, because a person triggered the visit
OAI-AdsBotChecks landing pages submitted as ChatGPT adsOnly visits pages submitted as ads

So blocking training and staying visible in search is a supported choice: allow OAI-SearchBot and disallow GPTBot.

Check 1: does your robots.txt let OAI-SearchBot in?

You pass this check if no rule in your robots.txt stops OAI-SearchBot from reading the pages you care about. Open https://your-domain/robots.txt in your browser and read it in this order:

  1. Look for a group that starts with User-agent: OAI-SearchBot. If there is one, only the rules of that group apply to OAI-SearchBot.
  2. If there is none, OAI-SearchBot follows the User-agent: * group. That is the rule of RFC 9309, the robots.txt standard.
  3. In the group that applies, Disallow: / blocks the whole site, and Disallow: /blog/ blocks only that folder.

The classic trap is a robots.txt that blocks everything under User-agent: * and only names Googlebot as an exception. Google can crawl that site; OAI-SearchBot cannot.

Two more things can block you without a single Disallow line:

  • robots.txt answers with a server error. If the file returns an error in the 500–599 range, RFC 9309 tells crawlers to assume that everything is disallowed. A missing file (404) is fine; a broken one is not.
  • The change is too recent. OpenAI says it can take about 24 hours after you update robots.txt for its search systems to adjust.

To stay in search while opting out of training, these two groups are enough:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Check 2: does your server or CDN turn OpenAI's crawler away?

robots.txt only says what you allow; a firewall, a CDN or a security plugin can still refuse the visit. OpenAI's help center says a site is eligible for ChatGPT search when it allows OAI-SearchBot and when its host or CDN accepts traffic from OpenAI's published IP addresses, listed at https://openai.com/searchbot.json. Three ways to check, quickest first:

1. Look at your CDN's bot settings. On Cloudflare, as of November 7, 2026, go to Security › Settings, filter by Bot traffic, then open AI Crawler and Agent access policies. The Search behaviour should be set to Allow (do not block). Cloudflare also notes that the Training setting applies to crawlers used for both search and training, so a strict training block can reach further than you expect.

Some articles say Cloudflare blocks AI crawlers by default. Cloudflare's documentation, updated on November 2, 2026, says that new domains start with Search, Agents and Training all set to Allow. A domain set up earlier keeps the settings chosen at the time, so check yours rather than assume. On other hosts, look for firewall rules, "bot protection" options or security plugins that mention AI crawlers, user agents or blocked countries and IP ranges.

2. Fetch a page as OAI-SearchBot. In a terminal, this command asks your server for the home page with the user agent OpenAI documents (the version number may change):

curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://your-domain/

200 means the server answered normally. 403, 429 or 503, or a "Just a moment" challenge page if you fetch the full page, means something filters on the user agent. When I ran it on lumigeo.app on November 7, 2026, the answer was 200 and the page text was in the response. One limit: the request leaves from your own connection, not from OpenAI's addresses, so a block based on IP ranges will not show here.

3. Read your server logs. They are the only real proof. Search them for OAI-SearchBot and look at the status codes it received. Anyone can fake a user agent, so compare the visiting IP addresses with OpenAI's published list before trusting a line. No visit at all does not always mean a block: a new or little-linked site may simply not have been discovered yet.

Check 3: is your text in the HTML, or added by JavaScript?

ChatGPT's crawlers read the HTML your server sends; they do not run your scripts. Vercel and MERJ measured this in December 2024: none of the major AI crawlers rendered JavaScript, and OpenAI's crawlers downloaded JavaScript files without executing them. Google's Gemini is the exception, because it relies on Googlebot, which does render pages. That is why a page can look perfect in Google and still be half empty for ChatGPT. Crawlers change, so treat this as the state of things at the time of that study.

The test takes under a minute:

  1. Open the page in Chrome and press Ctrl+U (Cmd+Option+U on a Mac). This shows the page source, the raw HTML. Do not use "Inspect": it shows the page after scripts have run.
  2. Press Ctrl+F (Cmd+F) and search for a sentence from the middle of the page, not the title.
  3. Found: crawlers can read that text. Not found: it is added by JavaScript, and ChatGPT's crawlers will not see it.

Repeat it on one page of each kind (home, product or service page, article), since templates differ. Content often added by scripts: reviews widgets, prices loaded from an API, tabs and accordions filled on click, "load more" lists. The usual fix is to render those pages on the server or at build time (most frameworks offer server-side rendering or static generation), or to prerender the pages that matter.

Is your page in the search index ChatGPT uses?

Passing the three checks makes a page eligible; ChatGPT still has to find it. OpenAI's help center says that ChatGPT search sometimes partners with other search providers and sends them rewritten versions of the user's question, and the providers it points to for privacy questions are Microsoft and Shopify. OpenAI does not say how much its answers depend on them, so treat this as a reasonable extra check rather than a rule: make sure your important pages are indexed by Bing, for example with Bing Webmaster Tools, which is free and can import a site from Google Search Console.

What a passing check does not tell you

  • Whether ChatGPT will show you. OpenAI ranks results with several factors and writes it plainly: "Placement is not guaranteed." The checks remove technical blocks; they do not earn a citation.
  • Whether asking ChatGPT is a valid test. When you ask ChatGPT to read a URL, the visit may come from ChatGPT-User, which OpenAI says is not used to decide what appears in search. A "yes, I can read it" in a chat proves little.
  • Anything about training. Allowing or blocking GPTBot has no effect on search answers, and the reverse is also true.

Quick diagnosis

What you seeLikely causeWhat to do
Disallow: / in the OAI-SearchBot group, or under * with no OAI-SearchBot grouprobots.txt blocks ChatGPT searchAdd an OAI-SearchBot group with Allow: /, then wait about 24 hours
robots.txt returns a 5xx errorCrawlers must treat the whole site as blockedFix the server so robots.txt returns 200, or 404 if you have none
The OAI-SearchBot fetch gets 403, 429, 503 or a challengeA firewall, CDN or plugin filters the user agentAllow OAI-SearchBot or the Search behaviour in that tool
The fetch returns 200 but your logs show refusalsA block based on IP addressesAllow the ranges in openai.com/searchbot.json
A sentence is missing from the page sourceThe text is added by JavaScriptRender the page on the server or at build time
Everything passes and ChatGPT never cites youNot an access problemWork on pages that answer real questions and on mentions from other sites

Doing checks 1 and 3 in one click (the tool I make)

If you check pages often, your own or a client's, I built Lumi GEO for that: a free Chrome extension that reads the robots.txt rules for 14 AI robots, including OAI-SearchBot, GPTBot and ChatGPT-User, and greys out on the page every block that only exists after JavaScript runs. It is free for 5 different pages a day, needs no account, and the analysis stays in your browser. It has the same limit as the terminal test: it works from your browser, so it cannot see blocks based on OpenAI's IP addresses, and it cannot tell you whether ChatGPT will cite a page. See how it works or get it on the Chrome Web Store.

Frequently asked questions

Can ChatGPT see my website?

Yes, if three conditions hold: your robots.txt allows OAI-SearchBot, your host or CDN accepts OpenAI's crawler, and your text is in the HTML rather than added by JavaScript. Whether ChatGPT then shows your pages depends on its ranking, which OpenAI says is not guaranteed.

Why can't ChatGPT read my website?

The usual causes are a robots.txt rule that blocks OAI-SearchBot, a firewall or CDN that rejects OpenAI's crawler, or text that only appears after JavaScript runs, which OpenAI's crawlers did not execute in Vercel's December 2024 measurements.

Does blocking GPTBot remove my site from ChatGPT search?

No. GPTBot collects content for model training, while ChatGPT search relies on OAI-SearchBot. OpenAI treats the two separately, so you can block GPTBot and still allow OAI-SearchBot.

How long does a robots.txt change take to work?

For ChatGPT search, OpenAI says it can take about 24 hours after you update robots.txt for its systems to adjust.

Can I just ask ChatGPT whether it can read my page?

It is not a reliable test. ChatGPT may open the page with ChatGPT-User, an on-demand visitor that OpenAI says is not used to decide what appears in search. The three checks above tell you more.

Sources (checked on November 7, 2026)