Technical AISO: how to structure your site for answer engines

There's a question almost nobody asks before polishing a single word of their content for AI: can the AI read the page? It sounds trivial. It isn't. For a huge share of the sites built over the last decade, the honest answer is "not entirely," and no amount of text optimization fixes a problem that happens earlier, in the technical layer.

The reason is a detail classic SEO had treated as solved. Googlebot learned years ago to execute JavaScript and render pages the way a browser would. AI crawlers, by contrast, still don't. And that difference, which looks like plumbing, decides whether your site exists or is invisible to ChatGPT, Claude and Perplexity.

What an AI bot actually sees

Vercel published an analysis worth keeping in mind: across more than 500 million GPTBot requests, it found no evidence the crawler executed JavaScript. According to its report, GPTBot downloads the initial HTML your server returns and moves on. It doesn't wait for the browser to paint the page, doesn't make a second attempt, doesn't run your scripts. The same goes for ClaudeBot and PerplexityBot: they read the raw HTML, extract what they find, and leave.

The consequence is brutal in its simplicity. If your main content —the price, the description, the article, the listing— only appears after JavaScript runs in the browser, the AI bot doesn't see it. It sees the empty skeleton the server sends: a <div id="root"></div> and little else. To you, in your browser, the page looks perfect. To the crawler, it's blank.

Vercel's report also sizes up why this stopped being a marginal problem: GPTBot traffic grew roughly 305% year over year, and AI bots now account for around 4.2% of all HTML page requests. It's not an experimental channel; it's a real and growing slice of who visits your site.

There's a home test to know where you stand. Open your page, disable JavaScript in the browser and reload. What's left on screen is, more or less, what an AI crawler sees. If your content disappears, there's your first problem, and it's more urgent than any keyword.

The hierarchy of technical levers that actually matter

Not all technical AISO tasks weigh the same. It's worth ordering them by real impact, high to low, because the industry tends to sell them all as equally critical and they aren't.

1. Server-side rendering

It's first and most important, for everything above. If your site is a single-page application that builds content in the browser, you need server-side rendering or prerendering: the server delivers the HTML already assembled, with the text inside. Frameworks like Next.js, Nuxt and similar solve it. A server-rendered site —like, in fact, this one— gives the crawler everything in the first response. This single decision separates visible sites from invisible ones.

2. Crawlability: don't shoot yourself in the foot with robots.txt

The second point is not to block, by accident or fear, the bots that can cite you. It's worth distinguishing two families: training crawlers, which take content to train future models, and live-retrieval crawlers, which look for sources to answer a specific query and link to you. Blocking the latter in robots.txt cuts off the very citation traffic you want. If your intent is to control training, use User-Agent-specific rules (GPTBot, ClaudeBot, Google-Extended, PerplexityBot and friends) instead of a blanket block that leaves you out of the answers.

3. Semantic structure and citable formatting

Only here does what most guides put first come in. Content AI cites tends to have a recognizable structure. Kevin Indig's analysis of 1.2 million ChatGPT responses found that 44.2% of citations came from the first 30% of the page, and that question-format headings correlated with roughly twice the citations. Translated: use real semantic HTML (h1, h2, p, lists), put the concrete answers up top and not buried, and structure sections as questions when it makes sense. It's not aesthetics; it's giving the model clean, self-contained blocks to extract.

4. Structured data

Schema markup helps the machine understand what kind of thing each part of your page is: an article, an FAQ, an organization, a product. It's no guaranteed ticket to citations, but it reduces ambiguity and is cheap to implement. We cover it in detail in the cluster's schema guide.

5. llms.txt: last on the list, not first

And here's a correction to the hype. The llms.txt file —a late-2024 proposal to point models toward your best content— is sometimes sold as the new must-have. The evidence says otherwise. An SE Ranking study of 300,000 domains found no measurable impact on citations, and the major crawlers barely request it. Adoption sits around 10% of domains. It's not bad to do it —it's optional, free and risk-free— but put it last: rendering, crawlability and structure first, because those are what actually move the needle.

The technical mistakes we see over and over

Auditing sites, the same stumbles repeat. The most common is stores and dashboards that load all their content via JavaScript: catalogs, product pages, reviews, real-time data. For the user it works; for the AI crawler it's an empty page. It's the difference between having a thousand products and having none, depending on who's looking.

The second is accidental blocking. Teams that, worried AI will "steal" their content, block all bots in robots.txt and then wonder why they never appear in answers. They cut the very wire they wanted to keep.

The third, subtler, is important content hidden behind interactions: accordions that load their text only on click, tabs that fetch content via a later call, "see more" that fires a new request. If the text isn't in the initial HTML, assume AI doesn't have it.

And the fourth is confusing Google speed with AI readability. A site can have impeccable Core Web Vitals and still hand the AI crawler empty HTML, because those metrics measure the experience of a browser that does run JavaScript. They're two different problems.

What we think makes sense to do

An opinion, to be taken as such. Start with the no-JavaScript test: if your content survives, you're already in better shape than most. If not, that's priority number one, above any copy or keyword work.

Then, review your robots.txt with the right question: are you blocking any retrieval bot that could cite you? Distinguish training from retrieval and decide on purpose, not by default.

Third, take care of structure: semantic HTML, answers up top, clear headings, structured data where it helps. It's foundational work that serves Google and AI alike.

And leave llms.txt for when the above is solved. Doing it does no harm, but prioritizing it over rendering is tidying the library while the front door is bricked up.

Questions we get a lot