Guide · Inspeccia

What blocking in robots.txt actually does (and what it doesn't)

Published August 3, 2026 · Approx. 9-minute read

Almost everyone uses robots.txt for something robots.txt does not do. The scene repeats itself: someone wants a page gone from Google, writes a Disallow, relaxes, and weeks later the URL is still there. It is not a bug and it is not Google being slow. It is that blocking a crawl and removing from the index are two different operations with two different tools, and confusing them produces the two most expensive mistakes in technical SEO.

The short answer. Disallow controls crawling: it tells the robot which addresses it may not enter. It does not control the index: it deletes nothing, and a blocked URL can still show up in results — without a description.

If what you want is for a page not to appear in Google, the tool is noindex (or a password, or deleting it). And for noindex to work you have to let the robot in to read it: if you block the page, it never receives the order.

Crawling is not indexing

These are two separate stages of the same process, and the whole confusion comes from treating them as one. Crawling is a robot downloading the page. Indexing is the search engine deciding to store it and show it as a result. Robots.txt lives entirely in the first stage.

Google's documentation puts it without ambiguity: a robots.txt file tells search engine crawlers which URLs the crawler can access on your site. And then it clarifies the part almost nobody reads: it is not a mechanism for keeping a web page out of Google. To do that, it says, block indexing with noindex or password-protect the page.

Everything else follows from there. Google can know a URL exists without ever having visited it — a link from another site is enough — and can decide to show it. Since it could not get in, it has nothing to build a snippet from. That is why Google warns that if your web page is blocked with a robots.txt file, its URL can still appear in search results, but the search result will not have a description. That bare result, no text underneath, is the unmistakable signature of this confusion.

Mistake #1: you block it to hide it, and it stays visible

This is the thank-you page after a form, the internal dashboard, the staging environment, the PDF with the price list. Someone decides that should not show up in Google, drops it into robots.txt, and considers the matter closed.

The result is worse than doing nothing. The URL can stay listed, and now Google cannot even read it to understand what it is, so it shows the naked address. And there is a side effect that goes unnoticed: the URL itself sometimes says more than you want. Addresses like /clients/acme-contract-renewal leak information in the name even though the content was never crawled.

Google says it as a direct warning: don't use a robots.txt file as a means to hide your web pages from Google Search results. If something genuinely must be out of public reach, the answer is not a text file that anyone can read by typing /robots.txt into the address bar — worth remembering that robots.txt is public — it is real authentication.

Mistake #2: you set noindex and block it at the same time

This one is subtler, which is why it survives for years on otherwise well-run sites. Someone does the right thing and adds noindex to the pages they want out of the index. Then, to reinforce it, they also add a Disallow in robots.txt. It sounds like belt and braces. It is, in fact, cancelling the order.

noindex does not live in robots.txt: it lives inside the page, as a tag in the <head> or as an X-Robots-Tag header in the server response. To obey it, the robot has to come in and read it. Google documents it literally: if the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results.

The correct sequence, then, has an order you cannot rearrange:

Remove the block from robots.txt for that path.
Leave the noindex on the page, by tag or by header.
Wait for Google to crawl again and apply the order. When Googlebot crawls that page and extracts the tag or header, Google drops the page entirely from results, regardless of whether other sites link to it.
Only then, if you have a crawl-budget reason, might you consider blocking it again. You almost never need to.

Which tool to use, depending on what you want

The fast way to stop getting this wrong is to stop asking "how do I block this?" and start with "what exactly do I want to happen?".

I want it not to appear in Google. noindex, via meta tag or X-Robots-Tag header. The header is the route for non-HTML: PDFs, images, video.
I want nobody unauthorized to see it. Password or authentication. Neither robots.txt nor noindex protects anything: they are instructions for well-behaved robots.
I want it to stop existing. Delete the page and return 404 or 410.
I want robots not to waste time on useless areas. Here, yes, robots.txt. This is its real job: internal search filters generating infinite URLs, calendars, print versions, parameters that duplicate content.
I want one specific crawler kept out. Robots.txt with its user agent. That is the AI-bot conversation, and it has its own nuances.

How Google reads your file (the rules that surprise people)

Four details from the specification that explain most of the "but I had it set correctly" cases.

Only one group applies, not all of them

Google's crawlers determine the correct group of rules by finding the group with the most specific user agent that matches the crawler's user agent. And they apply that one, only. If you have a block for * and another for Googlebot, Googlebot obeys its own and completely ignores the generic one, including rules that existed only in the generic block. This is the number one cause of rules that "don't apply".

The longest rule wins, not the first

When matching rules to URLs, crawlers use the most specific rule based on the length of the rule path. Where rules conflict, Google uses the least restrictive rule. That is why this file allows the blog even though it blocks everything under that folder:

User-agent: *
Disallow: /resources/
Allow: /resources/blog/

The logic is not top-to-bottom like a firewall. It is precision-based: the Allow is longer, so it wins over the Disallow for blog URLs.

One file per origin

The rules apply only to the host, protocol and port number where the robots.txt file is hosted. Your blog or docs subdomain needs its own; the main domain's file does not cover it. And it has to sit at the root of that origin.

There is a size cap and a cache

Google enforces a file size limit of 500 kibibytes: content after the maximum file size is ignored. Rare, but lethal when it happens, because it fails silently. And the file is generally cached for up to 24 hours, though it may be cached longer when refreshing the cached version isn't possible: a change does not take effect instantly.

The two-minute check. Open yourdomain.com/robots.txt in a browser and read the whole thing. Look for three things: a stray Disallow: / nobody remembers adding, paths you actually wanted indexed, and pages carrying noindex that are also listed as blocked. That last overlap is the one we find most often.

The separate case: AI bots

Everything above applies equally to AI crawlers, because they are crawlers and they read the same file under the same rules. What changes is the decision, and there it pays not to generalize.

AI bots do not all do the same thing. Some collect material to train models; others go out and search live when someone asks a question, and those are the ones that decide whether you can be cited in an answer. Blocking the first kind is a defensible stance about your content. Blocking the second erases you from AI search, which is almost never what the person wanted when they wrote the rule. It is a scalpel, not a hammer, and we cover it in full — bot by bot, with each provider's documentation — in blocking AI bots in robots.txt.

One more thing is worth adding, because it connects to the above. In Google's AI features there is no special markup to switch on: the official documentation says you don't need to create new machine readable files, AI text files, or markup to appear in these features, and that to be eligible as a supporting link a page must be indexed and eligible to be shown with a snippet. Translated: if you block yourself by mistake, no AI setting compensates for it. Robots.txt hygiene is the optimization.

Frequently asked questions

Does blocking a page in robots.txt remove it from Google?

No. It is the most widespread misunderstanding and also the most expensive. A robots.txt file tells crawlers which URLs they can access on your site, and nothing more: it is not a mechanism for keeping a web page out of Google. Google's own documentation says it flatly, and adds that if your web page is blocked with a robots.txt file, its URL can still appear in search results, but the search result will not have a description. If what you want is for it to disappear, the tool is a different one: noindex, password protection, or removing the page.

Why does my blocked page still appear in Google?

Because blocking the crawl does not stop the URL from being indexed. Google can learn that the address exists some other way — typically a link from another site — and decide to show it even though it was never able to read the content. Since it could not get in, it has no material for a snippet, so it shows the URL bare: title or address, no description. Seeing it there is not a robots.txt failure; it is exactly what robots.txt does.

Can I use noindex and block the page in robots.txt at the same time?

You can, but it backfires, and this is the second classic mistake. The noindex rule lives inside the page, in a meta tag or a response header. To obey it, the crawler has to come in and read it. If you also block the page in robots.txt, you are shutting the door on the very robot that was coming to receive the removal order. Google documents it exactly: if the page is blocked by a robots.txt file or the crawler cannot access the page, the crawler will never see the noindex rule, and the page can still appear in search results. To get a page out of the index: allow crawling first, let Google read the noindex, and only once it is gone consider blocking it.

So what is robots.txt actually for?

For managing crawl traffic, which is a real problem on large sites. It keeps robots from getting lost in areas that add nothing — internal search filters that generate infinite URLs, carts, calendars, print versions — so they spend their time on what you do want crawled. It also decides which specific crawlers may come in, which is where the AI-bot conversation lives. What it is not, to say it once more, is a privacy switch or a delete button.

How does Google decide which rules in my robots.txt apply?

With two chained criteria. First it finds the group whose user agent most specifically matches the crawler that arrived, and applies that group and only that one: it does not stack rules from several groups. Then, inside that group, for each URL it uses the most specific rule based on the length of the rule path, and where rules of equal length conflict, it uses the least restrictive one. In practice this means a long, precise Allow can carve an exception inside a shorter, broader Disallow, which is how most useful configurations are built.

Does one robots.txt cover my whole domain?

Not necessarily, and silent errors creep in here. The rules apply only to the host, protocol and port number where the robots.txt file is hosted. The robots.txt at https://www.example.com does not govern https://blog.example.com, nor the non-www version if it resolves as a separate host. Every origin needs its own, served at the root. Another detail with consequences: Google enforces a 500 kibibyte file size limit, and content after the maximum file size is ignored.

How long does Google take to see a robots.txt change?

Usually up to a day. Google documents that it generally caches the contents of the robots.txt file for up to 24 hours, but may cache it longer in situations where refreshing the cached version is not possible. That has a practical consequence worth remembering: if you make a mistake and block half your site, fixing the file does not reverse the effect instantly, and if the mistake was severe the recovery afterwards will take considerably longer than that day.

Does the same apply to AI bots?

The mechanics are identical — they are crawlers and they read the same file — but the underlying decision is different and deserves its own analysis, because AI bots do not all do the same thing. Some gather material to train models and others search live so they can cite you in an answer; blocking the second kind leaves you invisible in AI search, which is almost never what people wanted. We cover it in the dedicated guide to blocking AI bots in robots.txt.

Sources cited

Google Search Central — "Introduction to robots.txt" (robots.txt tells crawlers which URLs they can access; it is not a mechanism for keeping a page out of Google; a blocked URL can still appear without a description; don't use it to hide pages). Official documentation.
Google Search Central — "How Google interprets the robots.txt specification" (scope by host, protocol and port; 500 KiB limit; most specific user-agent group; longest rule and, on conflict, the least restrictive; cache of up to 24 hours). Specification.
Google Search Central — "Block Search indexing with noindex" (if the page is blocked by robots.txt the crawler will never see the noindex rule; implementation via meta tag or X-Robots-Tag header). Official documentation.
Google Search Central — "AI features and your website" (no special files or markup needed; to appear as a supporting link a page must be indexed and eligible for a snippet). Official documentation.

Are you blocking yourself without knowing?

Every analysis checks your robots.txt and your indexing directives alongside what AI actually answers about your business. It is the fast way to find out the problem was never the strategy, but a line of text added three years ago.

Analyze my site free →