llms.txt: what it is and whether it really works

You've seen it everywhere lately: agencies, threads and newsletters all insisting you need an llms.txt file on your site — "the robots.txt for the age of AI" — so ChatGPT and Google take you seriously. The urgency sounds convincing. The problem is that Google disputes it in writing, and can see it in its own server logs: no AI search engine is requesting that file. It's worth understanding what llms.txt actually is, who really reads it, and why so much of what's been written about it points you in the wrong direction.

What it is, without the marketing wrapper

llms.txt is a concrete proposal: a plain-text file in Markdown, placed at the root of your domain (yoursite.com/llms.txt), that hands a language model a clean index of your site — a summary with links to the pages that matter — so it can understand it without swallowing the full HTML, with its menu, banners and noise. Jeremy Howard, of Answer.AI, proposed it in September 2024.

The idea comes from a real limitation. As the spec itself puts it, "large language models increasingly rely on website information, but face a critical limitation: context windows are too small to handle most websites in their entirety." llms.txt is an attempt to solve that by serving the content pre-chewed. There's also a variant, llms-full.txt, which instead of an index dumps the full content of those pages into a single file, ready to inject whole into a model's context.

So far, a sensible engineering idea. The unjustified leap came next: turning it into "the new SEO standard for AI" that your business supposedly needs right now.

The promise you're being sold

Search "llms.txt" and the pattern repeats page after page: it's "the new frontier of SEO," "key to modern SEO," the file that "lets AI cite your content" and sends you traffic from ChatGPT. The comparison to robots.txt shows up almost every time, and it's the one that confuses most, because it implies a standard adopted and obeyed by the big players. Spoiler: it isn't.

The promise has the structure of every good piece of SEO snake oil: a simple, cheap technical gesture that supposedly unlocks visibility in a hot new channel. It's the same mold as the meta keywords tag in the 2000s. And, as it happens, that's exactly the comparison Google chose to describe it.

What the data says (and Google, in writing)

Here it pays to anchor the conversation in the sources, not in what circulates on LinkedIn. And the sources, in this case, are unusually clear.

Google's John Mueller said it flatly: "FWIW no AI system currently uses llms.txt." And he added the detail that ends the argument: you can tell when you look at your own server logs, because the bots don't even check for the file. The chatbots you care about — the ones that send you traffic — do crawl your pages, for training and for grounding, but none of them bother to look for your llms.txt. Mueller compared it, word for word, to the meta keywords tag: a field that existed, that nobody ended up using, and that lingered for a while as superstition.

It isn't one person's stray opinion. At the Search Central Deep Dive event in Asia-Pacific, in July 2025, Gary Illyes was just as blunt: Google doesn't support llms.txt and isn't planning to; what gets you into AI Overviews is plain old SEO. And Google's official documentation on optimizing for its AI features shut the door for good: it groups llms.txt alongside "content chunking," "AI rewriting" and "special schema" as tactics you don't need for AI Overviews or AI Mode.

Three Google voices, in three different formats, saying the same thing. Not much room left for creative interpretation.

So why do Stripe, Vercel and Anthropic have one?

Here's the nuance almost nobody mentions, and it's the one that separates the useful idea from the myth. It's true that plenty of serious companies publish an llms.txt: Stripe, Vercel, Cloudflare, Anthropic itself. Stop reading there and it looks like they validate the hype. But notice what they have in common: they're all developer products, with dense documentation, and their audience builds software with AI assistants.

That's the real reader of llms.txt today. It's not the AI search engine answering "what's the best coffee shop in Montevideo?"; it's the coding assistant and the browser agent. Tools like Cursor, Windsurf, Claude Code or GitHub Copilot, when you point them at a documentation site, look for /llms.txt and /llms-full.txt to understand the API without swallowing the whole site. For a company whose user lives inside a code editor asking a model to integrate its API, that file is functional, today, right now.

Google itself drew that line in an almost comic way. In May 2026, its Lighthouse tool added an experimental audit checking whether your site has an llms.txt — but framed it as useful for "browser-based agents," not for ranking. In other words: the Search team tells you to ignore it for search visibility, and the Chrome team tells you it can help agents. They don't contradict each other; they're talking about two different uses that the marketing blurred into one.

The honest call for your site

With all of that on the table, the decision is fairly simple and hinges on a single question: who is your audience?

If you have a developer product, an API or technical documentation that people consume with coding assistants, add your llms.txt without hesitation. It has a real reader, today. Plenty of SEO generators and plugins build it from your sitemap in a couple of clicks.

If you run an ordinary business — a store, a studio, a local service, a brand — and what you're chasing is for AI to recommend you, the honest answer is: it won't move that needle. That's no reason to delete it if your CMS already generates one; it's cheap and harmless. But don't pour time, agency money or — worse — peace of mind into believing you've "optimized for AI" because you uploaded a text file. That false sense of a job done is the real cost of the myth.

What to do instead of (or alongside) an llms.txt

If what you want is to show up in AI answers, the real levers aren't secret; they're the same ones Google keeps pointing to, and they're less glamorous than uploading a new file.

• Make sure AI can read your content in the raw HTML. The crawlers behind ChatGPT, Claude and Perplexity don't execute JavaScript: if your main content only appears after the JS runs, they don't see it. We cover this in the technical AISO guide.
• Answer the question clearly, and up top. Concrete headings, self-contained answers, specific data. A model cites what it can extract cleanly.
• Get corroborated. Other sources in your category mentioning you weighs more than any self-proclamation. Those are the entity signals that make you citable.
• Use schema to be understood, not as a magic switch. Schema helps Google understand what each part of your page is; it doesn't switch anything on by itself. We go deep in schema markup for AI search.

None of those is a file you upload once and forget. That's why they don't sell as well. But they're the ones that, page after page, decide whether AI names you or names your competitor.

Frequently asked questions