Block AI bots without vanishing from ChatGPT

You can shut AI out of your site in two lines of robots.txt, and plenty of people have. What almost nobody is told is what those two lines cost. "AI" isn't a single door: behind the acronym sit several distinct crawlers with distinct jobs, and the same instruction that keeps your content out of a training set can also take you out of ChatGPT's answers, where your customers are now asking who to hire. Below, bot by bot, which ones to block and which ones you want crawling you — and how to check, in a couple of minutes, what your own file is doing right now.

Most coverage of this topic tells you the same thing: paste User-agent: GPTBot with Disallow: / and you're done. It's advice that conflates two things that aren't the same, and for a business that wants to be found it can get expensive. It's worth understanding the map before you touch the file.

The costly mistake: "blocking AI" is several decisions, not one

Every serious AI company doesn't run one bot, it runs a family of them, and it separates them on purpose. There are, broadly, three roles:

• Training. Collects pages that may end up inside the next model. Blocking it says "don't use me to train."
• Search. Indexes your site so it can surface and cite you when someone asks a question inside the product. Blocking it says "don't cite me in your answers."
• User request. Fetches a specific page because, right then, a person asked for it. It doesn't crawl automatically.

The trap is that many people want the first thing —not appearing in the training data— and, by pasting a blanket Disallow: /, switch on the second without realizing it. They opt out of training and, in passing, erase themselves from the one channel they wanted to show up in. It's the equivalent of bricking up your storefront window so nobody copies your display.

The bot map, provider by provider

These are the user-agents that matter as of mid-2026, per each company's official documentation. Names are case-sensitive and each one needs its own rule: blocking ClaudeBot does not block Claude-SearchBot.

OpenAI (ChatGPT)

• GPTBot — training. OpenAI describes it as the crawler that gathers content "that may be used in training our generative AI foundation models." Block it if you don't want to feed the model.
• OAI-SearchBot — search. The one "used to surface websites in search results in ChatGPT's search features." If you want to be citable inside ChatGPT, this one has to get through.
• ChatGPT-User — user request. Acts when someone in ChatGPT asks for a specific page; it doesn't crawl automatically.

Anthropic (Claude)

• ClaudeBot — training. Collects web content that "could potentially contribute to" training Anthropic's models.
• Claude-SearchBot — search. Navigates the web "to improve search result quality" for Claude's users. Blocking it reduces your visibility there.
• Claude-User — user request. Accesses sites when a person asks Claude a specific question.

Perplexity

• PerplexityBot — search, not training. Its sole purpose, per Perplexity, is to "surface and link websites in search results." It isn't used to train models. Blocking it pulls you out of Perplexity with zero "don't train me" upside, because it wasn't training you anyway.
• Perplexity-User — user request. Perplexity itself warns it "generally ignores robots.txt," because a user requested that page.

Google (Gemini and Vertex AI)

Here's the most widespread misunderstanding. Google-Extended isn't a crawler with its own identity: it's a control token. Google states two points in writing that are worth memorizing:

• It does not affect inclusion in Google Search and is not a ranking signal. You can block it without losing organic positions.
• What it controls is whether your content helps improve Gemini and the Vertex AI generative APIs. Grounding with Vertex AI, for instance, doesn't use pages that have disallowed Google-Extended.

The important consequence: your presence in Google Search —and, by extension, in AI Overviews, which are built on the regular index— depends on Googlebot, not on Google-Extended. Blocking the latter is a decision about training Google's AI, not about your visibility. Two separate levers, and many people merge them into one.

So, block or allow? Three honest postures

There's no single answer, there are three, depending on the kind of site you run and what matters most to you.

1. I want visibility: allow almost everything

This is the default posture for most businesses, blogs, SaaS and stores that live off being found. You let every search bot through and, at most, block the training ones if feeding them bothers you. The reasoning is simple: the traffic and mentions that come from AI answers are the fastest-growing discovery channel, and a public site gains almost nothing by protecting from training the very content it wants people to read.

2. I want to protect my data without disappearing: the scalpel

The middle posture, and the one that makes the most sense for publishers and original-content creators. You block training, you allow search. Your content stays out of the datasets, but you're still citable when AI answers. Here's how it looks in robots.txt:

# Training: out. AI search: in.

# OpenAI
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

# Anthropic
User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

# Perplexity (doesn't train: let it in to appear)
User-agent: PerplexityBot
Allow: /

# Google: Gemini/Vertex out, no impact on your Search ranking
User-agent: Google-Extended
Disallow: /

One caveat so you don't fool yourself: if your file has no global Disallow: / rule for User-agent: *, those Allow: / lines are redundant —they were already allowed by default. We keep them explicit on purpose: they document your intent and cover you if someone later adds a blanket block. Clarity beats minimalism in a file you touch once a year.

3. I want to close everything: the slam

Legitimate for paywalled content, intranets, confidential documentation, or anything that simply shouldn't be in an AI answer. Here, yes, Disallow: / for every bot. Just don't complain later about not showing up: you're choosing, consciously, not to be there. The mistake isn't closing the door; it's closing it by accident.

What robots.txt does not guarantee

Before you trust your strategy to a text file, three limits worth being clear about.

It's a signal, not a barrier. robots.txt works because the big companies choose to respect it. "Ethical" training crawlers comply; the user-agents a person triggers (ChatGPT-User, Perplexity-User) often don't, because they read it as a human asking for that specific page. And a malicious scraper ignores it outright. If you need a real block, the right place is the firewall or WAF, filtering by user-agent and verifying against the official IP ranges each company publishes.

The default is shifting. On July 1, 2025, Cloudflare —which carries roughly 20% of web traffic— began blocking AI crawlers by default on new domains, and launched a "pay per crawl" marketplace. If your site sits behind Cloudflare or another proxy, you may already be blocking AI bots without having decided it in your robots.txt. Worth checking the panel before assuming anything.

Allowing isn't appearing. Letting OAI-SearchBot or PerplexityBot through is necessary, not sufficient. Being crawlable doesn't mean you'll be cited: for that, clear content, corroboration from other sources, and everything we cover in the rest of the cluster still carry the weight. robots.txt only decides whether they let you into the room; what happens inside is another story.

Where to start

Do two things this week. First, open your current robots.txt and check whether you're already blocking a search bot you didn't mean to —it's surprisingly common to inherit a Disallow from a template or a plugin. Second, pick your posture from the three above and leave it written down, commented, so the next person who touches the file understands why it's set that way.

And before optimizing anything, measure your starting point: if AI doesn't mention you today, the problem is rarely robots.txt. It's usually that you aren't citable enough yet, and that's fixed with content and reputation, not with a text file.

Frequently asked questions