What is the difference between llms.txt and llms-full.txt?

llms.txt is a curated markdown index of your most important pages with short descriptions. llms-full.txt concatenates the actual full content of those pages into a single markdown file, so an AI system can ingest everything in one fetch.

Do any AI engines officially read llms-full.txt?

No major engine — OpenAI, Google, Anthropic, or Perplexity — has documented automatic crawling of llms-full.txt. Its practical use today is manual: developers paste it into a chat context, and AI coding assistants fetch it for documentation lookups.

How big is too big for llms-full.txt?

Files from large documentation sites routinely exceed several hundred thousand tokens, which overflows most model context windows. If your export passes roughly 100,000 tokens, split it per product area or rely on the curated llms.txt index instead.

Is llms-full.txt Worth Generating?

Last updated: July 5, 2026 · Menra Team

llms-full.txt is worth generating if you run a documentation or reference site whose users paste content into AI assistants — and mostly not worth it otherwise. Unlike the curated llms.txt index, no major answer engine is confirmed to fetch llms-full.txt automatically, so its value today is human- and agent-mediated, not crawler-mediated.

What llms-full.txt actually is

The llms.txt proposal, published by Jeremy Howard of Answer.AI in September 2024, defines two files. llms.txt is a short markdown index: title, summary, and links with one-line descriptions. llms-full.txt is the maximalist sibling — the complete text of your documentation flattened into one markdown file. Anthropic publishes one for its own developer docs, and documentation platforms like Mintlify began auto-generating both files for hosted docs in late 2024.

Where it genuinely helps

The strongest use case is developer tools. Engineers regularly paste an entire llms-full.txt into Claude or ChatGPT to get accurate answers about an API, and AI coding assistants such as Cursor can be pointed at the file as a context source. For a SaaS with technical docs, that means fewer hallucinated method names and wrong code samples circulating about your product — a real, if unmeasured, accuracy win.

For marketing sites, blogs, and ecommerce catalogs, the case is weak. Answer engines build their picture of your brand from normal HTML crawling and search indexes, not from a bulk markdown dump. Time spent on answer-first page structure returns more than a file no retrieval pipeline reads.

The size problem

Full-content exports get large fast. Documentation sites with a few hundred pages produce files of several hundred thousand tokens — beyond the context window of most production models, which typically handle 128K to 1M tokens. A file nothing can ingest whole serves nobody. Two mitigations work: split exports per section (llms-full-api.txt, llms-full-guides.txt) or keep only llms.txt and let agents fetch individual pages on demand.

How to automate it

Do not maintain it by hand; it will rot. Docs platforms (Mintlify, GitBook) generate it natively, Firebase and several static-site generators ship plugins, and a custom build step that walks your content directory and concatenates markdown takes under an hour to write. Regenerate on every deploy so llms-full.txt never contradicts the live site.

Verdict

Generate it if automation makes it free and your audience includes developers or AI agents. Skip the manual effort if you publish marketing content — put that hour into content structure and crawler access instead, and track whether engines actually cite you.

Frequently asked questions

What is the difference between llms.txt and llms-full.txt?: llms.txt is a curated markdown index of your most important pages with short descriptions. llms-full.txt concatenates the actual full content of those pages into a single markdown file, so an AI system can ingest everything in one fetch.
Do any AI engines officially read llms-full.txt?: No major engine — OpenAI, Google, Anthropic, or Perplexity — has documented automatic crawling of llms-full.txt. Its practical use today is manual: developers paste it into a chat context, and AI coding assistants fetch it for documentation lookups.
How big is too big for llms-full.txt?: Files from large documentation sites routinely exceed several hundred thousand tokens, which overflows most model context windows. If your export passes roughly 100,000 tokens, split it per product area or rely on the curated llms.txt index instead.

Keep exploring

See how AI engines talk about your brand — track mentions across ChatGPT, Perplexity, Claude, Gemini and 5 more. Start with Menra