GPTBot vs OAI-SearchBot: Blocking the Wrong One Silently Removes You From ChatGPT

OpenAI runs four separate crawlers, not one, and two of them get confused constantly. GPTBot decides whether your content trains future models. OAI-SearchBot decides whether ChatGPT may cite you at all. We fetched robots.txt from 267 real websites to see how often sites block the second one by accident: 6.4% of the general-sample sites we could read do — removing themselves from ChatGPT search answers while, in most cases, leaving the training crawlers they were probably worried about still fully allowed.

The short version

• GPTBot and OAI-SearchBot are not the same switch. OpenAI's own documentation states each setting is independent: GPTBot collects content for training, OAI-SearchBot builds the index that decides whether your pages can appear in ChatGPT search answers.
• OpenAI's wording is unambiguous about the consequence: sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers." Disallowing GPTBot has no documented effect on that.
• We audited 267 domains; 174 served a robots.txt. In the general sample (n=110), 19.1% block GPTBot and 6.4% block OAI-SearchBot — including 7 sites that block the crawler whose documented job is to make them visible.
• Your robots.txt may not be yours. 15 of the 174 readable files carry a byte-identical block stamped # BEGIN Cloudflare Managed content, drawn from both populations — and it is the correct configuration. The mistakes are not coming from the largest preset.
• The raw audit output is public and the sample is reproducible: aggregate results, per-domain verdicts for the vendor and reference populations, seed 20260915.

OpenAI runs four crawlers, not one

Almost every article on this topic describes "two OpenAI bots" or "three." The official documentation lists four, and the differences between them are the entire subject:

User agent	What OpenAI says it does	Does blocking it affect citations?
OAI-SearchBot	"Used to surface websites in search results in ChatGPT's search features." Sites opted out "will not be shown in ChatGPT search answers, though can still appear as navigational links."	Yes — this is the one
GPTBot	"Used to crawl content that may be used in training our generative AI foundation models." Disallowing it "indicates a site's content should not be used in training."	No
ChatGPT-User	Fetches pages when a user asks ChatGPT to visit one. "Not used for crawling the web in an automatic fashion."	No — and robots.txt may not even apply
OAI-AdsBot	Validates landing pages submitted as ads on ChatGPT. "Only visits pages submitted as ads," and its data is not used to train models.	No

Three details from that documentation do most of the work here.

First, the settings are independent. OpenAI's own sentence: "Each setting is independent of the others — for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training." That is the whole article in one line. You can opt out of training and stay in discovery. The two decisions were designed to be separable.

Second, ChatGPT-User is a different kind of visitor. Because those fetches happen because a person asked, OpenAI notes that "robots.txt rules may not apply." So a rule that blocks ChatGPT-User is not a reliable control anyway — it is a control on something that is already user-initiated.

Third, if you allow both crawlers, OpenAI may only crawl once. The documentation says that when a site allows both bots, "we may use the results from just one crawl for both use cases to avoid duplicative crawling." This is why server logs can be misleading: seeing GPTBot in your logs does not tell you whether OAI-SearchBot ever came, and vice versa.

What blocking each crawler actually costs you

Put the four crawlers into two groups and the trade-off becomes a decision rather than a guess.

The training group — GPTBot, plus other vendors' training crawlers. Blocking these opts your content out of future model training. It costs you nothing in the citation path. If your reason for blocking is "I don't want my content training someone else's model," this group is the correct target, and it is a legitimate choice.

The retrieval group — OAI-SearchBot, ChatGPT-User, OAI-AdsBot, and other engines' equivalents. Blocking these removes you from the path where a live question gets answered. For OAI-SearchBot specifically, OpenAI documents the outcome directly rather than alluding to it.

There is a nuance most guides get wrong in the other direction too. Being crawlable and indexed is not the same as being cited. OAI-SearchBot makes your page available to the retrieval layer; it does not oblige ChatGPT to quote it. Crawler access is the entry ticket. What earns the citation is what is on the page — which is a separate problem from robots.txt, and we measured that separately in our study of what readers and AI engines actually reward.

We checked 267 sites. Here is what they do

We fetched /robots.txt from two defined populations and evaluated, for each of eleven AI crawlers, whether the file blocks the root path.

• Population A — AI-visibility vendors (60 domains requested, 57 served a robots.txt). Companies that sell AI visibility, GEO, or AI-crawler tooling. This group is not a finding; it is our positive control. We picked the population most likely to have the crawler distinction right, so that the parser had to prove it could detect a correct configuration before we trusted what it reported about anyone else. Its results appear in the table only as that check — we make no claim about the difference between 0/57 and the general sample, because with samples this size that difference is not statistically resolvable.
• Population A2 — hand-picked reference sites (7 domains, all served a robots.txt). ahrefs.com, semrush.com, hubspot.com, searchengineland.com, neilpatel.com, openai.com — and our own chatgptgrow.com. They are not in the table because all seven block nothing. They exist so the arithmetic reconciles and so the parser had a second known-good group. Naming them is deliberate: a reference group whose membership you cannot see is not a reference group you can check, and one of these is ours, which the table would not have told you.
• Population B — general sites (200 sampled). A seeded random sample of the Tranco top 50,000 domains. 110 of the 200 served a robots.txt.

The arithmetic. 60 + 7 + 200 = 267 domains requested; 57 + 7 + 110 = 174 served a robots.txt. Every percentage below is a share of the population that served a file, not of the population requested.

The reference group is also where this sample is weakest, and it is worth saying so plainly: it is a hand-picked list, not a random sample, so nothing in it generalises. It is a control, not a finding.

Method. GET https://<domain>/robots.txt, parsed by standard group semantics (consecutive User-agent lines share the rules that follow). A crawler counts as blocked when its effective group contains Disallow: / or Disallow: /* with no Allow: /; a crawler-specific group takes precedence over *. This evaluates the root path only — it is not a full robots.txt evaluator. The aggregate output for all three populations and all eleven crawlers is published at /research/robots-audit-summary.json, and the per-domain verdicts for the vendor and reference populations at /research/robots-audit-domains.csv.

Why the general sample is not published per-domain. We said we would not name the individual general-sample sites that block a search crawler, and a per-domain file would name them. The sample is reproducible from the seed and the Tranco list version above, so this is a commitment about what we publish, not an anonymity claim: anyone who wants to re-derive the 110 domains can.

Crawler	AI-visibility vendors (n=57)	General sites (n=110)
OAI-SearchBot	0 (0%)	7 (6.4%)
ChatGPT-User	0 (0%)	9 (8.2%)
OAI-AdsBot	0 (0%)	4 (3.6%)
PerplexityBot	0 (0%)	8 (7.3%)
Any search crawler	0 (0%)	10 (9.1%)
GPTBot	3 (5.3%)	21 (19.1%)
ClaudeBot	3 (5.3%)	17 (15.5%)
CCBot	7 (12.3%)	23 (20.9%)
Bytespider	7 (12.3%)	17 (15.5%)
Google-Extended	4 (7.0%)	17 (15.5%)
Applebot-Extended	4 (7.0%)	16 (14.5%)
meta-externalagent	4 (7.0%)	17 (15.5%)

Two readings matter, and only one of them is a comparison we can make.

General sites split the two groups, and some get it backwards. 19.1% block GPTBot and 6.4% block OAI-SearchBot. Training-crawler blocking is genuinely higher in the general sample — that gap is wide enough to survive the sample size (GPTBot: 19.1% of 110, 95% CI 11.7–26.4%, against 5.3% of 57, CI 0–11.1%). And 6.4% block OAI-SearchBot — the crawler whose documented function is to make them visible. Those sites are not protecting anything; they have removed themselves from ChatGPT search answers while, in most cases, leaving the training crawlers they were probably worried about still fully allowed. We are not naming the individual general-site domains that block the search crawler; the aggregate and the method are the publishable part.

The vendor row is a parser check, not a finding. Not one of the 57 blocks a retrieval crawler, and 48 block nothing at all. That is the result you would expect from the group most likely to know the distinction — which is exactly why we used it to validate the parser rather than to make a point. It also means we should be explicit about what we are not claiming: the 0/57 against 6.4% of 110 is not a difference this sample can resolve. Zero out of 57 is consistent with a true rate as high as 5%, which overlaps the general sample's interval. Where the sample cannot support a comparison, we are not making one.

What this adds to research that already exists

This is not the first study to read robots.txt at scale, and the honest thing is to say where our numbers sit next to the others.

The closest prior work is Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web (arXiv 2510.10315), which reports that 60.0% of reputable news sites disallow at least one AI crawler, up from 23% in September 2023 to nearly 60% by May 2025. There is also Scrapers Selectively Respect robots.txt Directives (arXiv 2505.21733) on how consistently crawlers honour the files.

60% against our 19.1% is a wide gap, and it is mostly definitional rather than contradictory:

	Their study	This audit
Population	Curated list of reputable news sites	Seeded random sample of the Tranco top 50,000
Metric	Disallows at least one AI crawler, of any kind	Blocks each crawler at the root path
Crawler set	Broad (they count an average of 15.5 forbidden agents)	Fixed set of 11

Counting "at least one of fifteen" will always exceed counting "GPTBot specifically, at the root," and news organisations have both a stronger rights posture and staff who act on it. If you want the news-industry number, use theirs. What this audit adds is different: a fixed crawler set applied identically to a population chosen without regard to AI policy, plus the training-versus-retrieval split, plus the raw per-domain verdicts so the counting can be checked.

Who wrote your robots.txt

The most useful thing we found is hidden in the raw files. 15 of the 174 domains that served a robots.txt — 8.6% — carry the same AI-crawler block, byte for byte, wrapped in # BEGIN Cloudflare Managed content / # END Cloudflare Managed Content. It appears in both populations: 6 AI-visibility vendors and 9 general sites, from a Portuguese university (ctt.pt) to a Telugu film site to tapology.com. Seven of those files are identical down to the last byte, comments and all.

Nobody typed that. Here is what the block says, and it matters that we quote it rather than describe it:

# BEGIN Cloudflare Managed content

User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /

User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CloudflareBrowserRenderingCrawler
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: meta-externalagent
Disallow: /

# END Cloudflare Managed Content

That is the correct configuration. Read the list: nine training crawlers disallowed, Content-Signal: search=yes,ai-train=no, and every retrieval crawler — OAI-SearchBot, ChatGPT-User, OAI-AdsBot, PerplexityBot — left alone. It is the same separation the rest of this article argues for, written by a machine and applied to 8.6% of everything we could read.

So the preset is not the villain of this story, and our earlier reading of this data was wrong in that direction. We expected a platform preset to be a plausible source of accidental blocking. The largest preset in our sample formalised the distinction instead, and is now propagating the correct answer across a small but measurable slice of the web. That is a better result than a warning would have been, and it moves the blame somewhere less comfortable: the 6.4% who block OAI-SearchBot are not following a bad default. They decided to. Those are individual choices, made by people, in every case we can see.

What survives from the original point is a different and more practical worry. Your robots.txt may be a document you have never actually read, and the rules in it can change on someone else's release schedule. The Cloudflare block is correct today; you would have no way to know that from your own dashboard, and no notification if it changed. So the question is still "who wrote my robots.txt" — not because the writer is likely to be wrong, but because you cannot audit a decision you did not make and were not told about.

The robots.txt that does what you probably want

If your goal is "do not train on my content, but do let ChatGPT cite me," the configuration is three separate decisions, not one:

# 1. Search / discovery crawlers — keep these allowed.
#    OAI-SearchBot is the one that decides ChatGPT search answers.
User-agent: OAI-SearchBot
Allow: /

# 2. Training crawlers — block these if you do not want your
#    content in future model training. This does NOT affect citations.
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

# 3. Everything else — your normal rules.
User-agent: *
Allow: /

What you want	What to block	What to leave open
Stay citable, opt out of training	GPTBot, CCBot, Google-Extended, Bytespider, Applebot-Extended, meta-externalagent	OAI-SearchBot, ChatGPT-User, OAI-AdsBot, PerplexityBot
Opt out of everything	the training list plus OAI-SearchBot	nothing — accept that ChatGPT cannot cite you
Do nothing explicit	nothing	everything (robots.txt defaults to allow)

Two operational notes from OpenAI's documentation that change how you verify this. First, allow ~24 hours: "it can take ~24 hours from a site's robots.txt update for our systems to adjust." A change you make now will not show up in a test you run in ten minutes. Second, use the published IP ranges, not the user-agent string alone, when you are auditing your logs — OpenAI publishes separate ranges for each crawler at openai.com/searchbot.json, openai.com/gptbot.json, openai.com/adsbot.json, and openai.com/chatgpt-user.json. It also notes that robots.txt requests may carry an extra robots.txt marker in the user-agent, precisely so you can tell a rules check apart from a content fetch.

How to check your own site in two minutes

1. Open https://yourdomain.com/robots.txt. No file is fine — the default is allow. What matters is what it says.
2. Search the file for OAI-SearchBot. If it appears under a Disallow: /, you are opted out of ChatGPT search answers.
3. Then search for GPTBot. If GPTBot is disallowed and OAI-SearchBot is not, you have the configuration most informed sites use. If both are disallowed, you have opted out of training and citations — which may be intentional, but is worth being intentional about.
4. Check who wrote it. If the rules arrived with a hosting, CDN, or security preset, confirm the preset's current behaviour rather than assuming this year's list matches last year's intent.
5. Re-check the * group. A blanket Disallow: / under User-agent: * blocks every crawler in this article at once, and is the most common way to lose citations without writing a single AI-related rule.

Frequently asked questions

Does blocking GPTBot stop ChatGPT from citing my website? No. OpenAI's documentation describes GPTBot as the crawler that collects content for training foundation models, and states that disallowing it "indicates a site's content should not be used in training." The crawler that affects whether your site can appear in ChatGPT search answers is OAI-SearchBot, a separate setting that OpenAI documents as independent.

Which crawler should I allow to appear in ChatGPT search results? OAI-SearchBot. OpenAI's documentation says sites opted out of it "will not be shown in ChatGPT search answers, though can still appear as navigational links," and recommends allowing it plus requests from the published IP ranges.

Is blocking AI crawlers bad for SEO? It depends entirely on which crawler. Blocking training crawlers (GPTBot, CCBot, Google-Extended, Bytespider) removes you from future model training with no documented effect on citations. Blocking search crawlers (OAI-SearchBot, PerplexityBot) removes you from the answer layer of those products. In our audit, 0 of 57 AI-visibility vendors blocked a search crawler, and 48 blocked nothing at all — that group is our parser check rather than a finding, since a sample that size cannot resolve a difference against the general-site rate.

What does ChatGPT-User do, and does robots.txt control it? ChatGPT-User is the agent used when a user asks ChatGPT to visit a page. OpenAI says it is "not used for crawling the web in an automatic fashion," and notes that because those actions are initiated by a user, "robots.txt rules may not apply." It is also explicitly not the setting that determines Search appearance.

Why do so many sites have identical AI crawler rules? Because a platform wrote them. We found three unrelated domains serving byte-identical AI-crawler blocks, including the distinctive user-agent name CloudflareBrowserRenderingCrawler. CDN, hosting, and security presets commonly ship an "AI bots" rule set, which means your robots.txt may reflect a vendor's default rather than a decision you made.

How long after editing robots.txt does ChatGPT notice? OpenAI documents roughly 24 hours: "it can take ~24 hours from a site's robots.txt update for our systems to adjust." Testing immediately after a change will produce a misleading result.

How was this audit conducted? We fetched /robots.txt from 60 AI-visibility vendor domains (57 served a file) plus 7 hand-picked reference sites (naming them in the method section above), and from a seeded random sample of 200 domains from the Tranco top 50,000 (110 of which served a robots.txt). The audit ran on 15 September 2026. We evaluated the root path for eleven crawlers using standard group semantics, with crawler-specific groups taking precedence over *. Individual general-site domains that block a search crawler are not named. Aggregate output for all three populations and all eleven crawlers: /research/robots-audit-summary.json. Per-domain verdicts for the vendor and reference populations: /research/robots-audit-domains.csv.

Crawler access is the entry ticket, not the win

Everything above buys you one thing: the possibility of being retrieved. It does not buy the citation. Once a page is crawlable, the next question is whether it contains anything worth quoting — and that is where most content fails, not at robots.txt.

If you want to know where you currently stand, our free AI citeability checker tests whether a page is structured so an AI can parse and lift it cleanly. Treat a passing score as the floor. Chatgpt Grow is built for the part the checker cannot see: turning a customer question into original research and a published article, one a day, on autopilot.

---

Method note. The crawler semantics quoted above are from OpenAI's official crawler documentation (developers.openai.com/api/docs/bots), retrieved September 2026. The audit ran on 15 September 2026 and requested robots.txt from 267 domains, 174 of which served one. The general-site population is a seeded random sample (seed 20260915) of the Tranco top 50,000. Evaluation covers the root path only and is not a complete robots.txt implementation. Because site configurations change, treat the percentages as a snapshot of September 2026.