What Is llms.txt and How Do You Write One?
llms.txt is a plain-text file that introduces a site to AI models. Here is what it does and how to put one together properly.
rabbitclip teamPublished: 4 min read
Short answer
Llms.txt is a plain-text file sitting at a site's root that lists its most important pages with short descriptions. Its job is to make it easier for large language models to work out what matters on a site while crawling it.
Proposed in 2024 by Jeremy Howard of Answer.AI, the format is not an official W3C or IETF standard, it spread as a community proposal instead. Even so, companies including Anthropic, Stripe and Cloudflare use it on their own sites, and the canonical definition lives at llmstxt.org.
What problem does llms.txt actually solve?
A language model does not browse a site through a menu the way a person does, it takes the raw HTML, navigation clutter and all. llms.txt gives it a simplified content map it can go to directly.
That is similar in spirit to how robots.txt manages crawl permissions or sitemap.xml lists pages, but its job is different: it does not say what can be crawled, it says what matters and what it is about.
How is the file structured?
The specification defines two files. /llms.txt is a short markdown navigation summary: a title, a one-sentence description, then links grouped under section headings. /llms-full.txt is the fuller version, gathering an entire documentation set into a single file.
A small corporate site usually only needs the first. A software product with heavy documentation might keep both.
How do you actually write one?
The file is plain markdown, placed at the root. The site or brand name sits at the top as an H1, followed immediately by a one-sentence summary. Section headings starting with '## ' then group the most important pages, each with a short, accurate description.
- An H1 with the site or brand name
- A one-sentence summary paragraph
- Section headings (## Services, ## Blog, and so on)
- A short, accurate description for each link
- Only genuinely important pages, not a full site map
What mistakes make the file useless?
The most common one is copying an automated sitemap wholesale and listing hundreds of links, which defeats the whole point of prioritisation. Another is writing descriptions in marketing language; a model is looking for a short, accurate definition, not a slogan.
Writing the file once and forgetting it also causes drift. As the site changes, especially when a new service or product is added, the file needs updating too.
Is a site broken without one?
No. This is not an official requirement, it is a proposal that is still gaining support. Google, OpenAI and Perplexity's own search features do not currently look for llms.txt specifically; models still crawl standard page content regardless.
It remains a low-cost step though. A few hours of work gives a documentation-heavy or many-paged site a clear map to hand a model.
Common mistakes
A typical one we see on a software product's documentation site is generating llms.txt automatically and publishing it unreviewed. The result is a near-exact copy of sitemap.xml that loses the entire point of prioritisation.
Another is copying a template without adapting the language; if the brand's own content is written in one language and the file stays in a different one regardless, it creates an inconsistency that helps no one.
A useful test: every link in the file should pass the question 'does this page genuinely explain this topic to a visitor'. If it does not, it should come off the list.
How to set up an llms.txt, step by step
Building the file from scratch does not take more than an afternoon, provided the steps happen in the right order.
- List the site's ten to twenty most visited and most asked-about pages.
- Write a one-sentence, marketing-free description for each.
- Group the pages under sensible section headings (## Services, ## Blog, ## FAQ, and so on).
- Upload the file to the root as /llms.txt and check it opens directly in a browser.
- Add a calendar reminder to review the file whenever a new service or page is added.
llms.txt is a short summary a site hands to a model rather than a person. Writing one takes an afternoon, but it also forces some clarity about the site's own structure, which is why it is a small but real part of GEO work. For a team following these steps, the file stops being a one-off project and becomes a small maintenance habit that grows alongside the site. rabbitclip can look at a site's current structure and work out what that file should actually say.
FAQ
Is llms.txt an official standard?
No. It spread as a community proposal; the canonical definition sits at llmstxt.org, without W3C or IETF approval.
Can AI not crawl a site without llms.txt?
It can. The file does not gate crawling, it only makes prioritisation easier.
Does llms.txt do the same job as sitemap.xml?
No. sitemap.xml lists every page; llms.txt presents only the most important ones with a short description.
How often should the file be updated?
Whenever the site's structure changes meaningfully, such as adding a new service, product or section.
