llms.txt: What the Evidence Actually Shows

Correction, 17 September 2026. The original version of this post, published 27 April 2026, argued that answer engines had started reading llms.txt and that domains publishing one saw a 6 to 9 point lift in first-mention rate. That claim came from an early internal comparison we ran on a small, uncontrolled sample, and it does not survive the evidence published since. Google now states in writing that Search ignores the file; a 137,000-domain study found that almost no bot fetches it. We have rewritten the post rather than quietly deleting it, because we told people to prioritise this and some of them did.

The short version: llms.txt is a cheap, harmless file with a narrow real use. It is not a citation lever, and no measurement we or anyone else can point to says otherwise.

What llms.txt is

llms.txt is a plain Markdown file at the root of a domain, https://example.com/llms.txt, holding a curated summary of the site's most important pages. Jeremy Howard proposed it in late 2024 as a complement to robots.txt and sitemap.xml for the LLM era. The structure is deliberately simple: an H1 with the brand name, a one-line blockquote description, then Markdown sections of links with one-line summaries. The spec lives at llmstxt.org and is still a draft.

The idea is reasonable. Retrieval is token-budget constrained, so a 2 to 4 KB file that tells a model what a site covers should save it from guessing. The problem is not the idea. The problem is that the systems it was designed for do not read it.

What the evidence says

Google Search ignores it, on the record. Google's official guide to generative AI features, published 15 May 2026, lists llms.txt under things site owners can skip: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search." Gary Illyes confirmed at Search Central Live that Google is not pursuing it, and John Mueller compared it to the keywords meta tag, a signal search engines stopped honouring two decades ago.

Almost nothing fetches it. Ahrefs analysed 137,210 domains in June 2026. Of the domains publishing a valid llms.txt, 97 percent received zero traffic to that file in May 2026. Of the requests that did arrive, only 19.5 percent came from named AI tools, and the retrieval bots that actually assemble answers, PerplexityBot and OAI-SearchBot, accounted for 1.1 percent. No AI bot went looking for the file on domains that lacked one.

The most-cited domains do not use it. SE Ranking's scan of 300,000 domains put adoption at 10.13 percent. Among the fifty domains AI answers cite most often, exactly one publishes an llms.txt. If the file were a meaningful input, that number would look different.

No major provider has committed to it. OpenAI, Google, Anthropic, Meta and Mistral have all declined to say their production retrieval systems read it. GPTBot, ClaudeBot, PerplexityBot and Google-Extended overwhelmingly crawl HTML.

Where it still earns its place

Two narrow cases, both real:

Coding agents and documentation. This is where the file demonstrably gets read. OpenAI, Anthropic and a long list of developer-tool vendors publish llms.txt for their docs, and the bots that fetch it most in the Ahrefs data are coding agents. If you run a documentation site and want an agent to navigate it accurately, ship one.

Agent readiness, per Chrome. Lighthouse 13.3, released 7 May 2026, promoted an "Agentic Browsing" audit category into the default config, and it checks for llms.txt. So Google Search ignores the file while Google Chrome audits it. The two are answering different questions: search visibility versus browser-agent compatibility. The Chrome check is the honest reason to keep the file, and it has nothing to do with whether ChatGPT cites you.

How we score it, and why

Menra measures llms.txt and shows the result, but it moves our GEO readiness composite by at most five points out of a hundred. That is a deliberate cap, written down in a decision note, and it replaced a twenty-point weight we could not defend once we read the evidence above. We kept the measurement because the file is free to publish and costs nothing to check. We refuse to score it higher because we would be selling a lever that does not move.

If a tool tells you llms.txt is a major ranking factor for AI search, ask which measurement says so. There is a public 137,000-domain study, a public Google statement, and a public adoption scan, and all three point the same way.

What to do instead

The things that survive scrutiny are unglamorous:

Be crawlable by the bots that matter. Check that GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended and Bingbot are not blocked in robots.txt or at your CDN's bot-protection layer. A WAF rule quietly returning 403 to AI crawlers costs more citations than any file you can add.

Write passages an engine can lift. The controlled study by Aggarwal and colleagues (KDD 2024) found that adding quotations, statistics and cited sources lifted generative visibility 30 to 40 percent, while keyword-style tuning did essentially nothing. Open each section with a self-contained sentence that names the subject, then back it with a number and a source.

Make your entity unambiguous. State on your About page what the company is, when it was founded, by whom, and where it operates, and mirror that in Organization schema with sameAs links to profiles that actually exist. Engines disambiguate brands before they cite them.

Earn genuine third-party mentions. Reviews on the platforms your category's answers cite, honest participation in the forums where it is discussed, inclusion in independent comparisons. Google's guide is explicit that manufactured mentions are not worth chasing, and Bing's February 2026 guidelines added "artificially engineered language" to its abuse definitions. Fabricated consensus gets filtered and the brand carries the risk.

Measure the outcome, not the checklist. Pick the prompts your buyers actually type, run them on a fixed schedule, and watch whether your share of mentions moves. A file you shipped is not a result. A citation is.

Should you ship one anyway?

Yes, if it takes you an hour and you understand what you are buying: a tidy machine-readable index, a passing Lighthouse agent-readiness check, and nothing else. Write the blockquote description carefully, list the ten to fifteen pages that matter, keep summaries to one line, and revisit it when your positioning changes.

Just do not ship it instead of the work above. Ours is at menra.ai/llms.txt, and it is the least important thing on that list.

– The Menra Team