What is llms.txt? The proposal, what it does, and whether it works

It is often called a robots.txt for AI, which it is not: robots.txt sets crawl rules, while llms.txt is a reading list that nothing is obliged to read.

What the llms.txt proposal says

Howard published the proposal on llmstxt.org in September 2024 and revised it as version 2 in August 2026. Agents read websites constantly, but "web pages are built for people", wrapped in navigation, ads and JavaScript, and context windows "are still too small for most websites in their entirety". A short Markdown file at a predictable address gives an agent "concise, expert-level information gathered in a single, accessible location."

The file is meant to be read on demand while an agent helps a user, not crawled for training; the proposal expected llms.txt "would mainly be useful for inference rather than training". A file can also sit at a subpath and cover the pages under it, so /docs/llms.txt describes a documentation section. By the proposal's own account, llms.txt files are "used most heavily for software documentation, where coding agents follow them to find API references and tutorials."

The llms.txt format

In order, a file contains:

1. An H1 with the name of the site or project, the only required part.
2. A blockquote with a short summary.
3. Optional paragraphs or lists, without headings, for anything an agent should know up front.
4. H2 sections, each a list of links written as [name](url): notes.
5. Optionally, a section titled "Optional" for secondary links an agent can skip when it is short of context.

A small business's file might look like this:

# Example Garden Co.

> Example Garden Co. designs and builds small gardens, from a one-day consultation to a finished planting plan.

We work within 30 miles of our studio and do not sell plants online.

## Services

- [Garden design](https://example.com/design): Packages, prices and what each includes
- [Planting plans](https://example.com/planting): How a plan is made and delivered

## Guides

- [Shrubs for clay soil](https://example.com/blog/clay-soil-shrubs): Our most-read guide

## Optional

- [About us](https://example.com/about): The team and how we work

Some documentation platforms also publish an llms-full.txt, a companion file with the full text of every page in one document, meant as context for AI tools. The proposal itself does not define it.

llms.txt vs robots.txt and sitemaps

File	Read by	What it does	Enforces anything
robots.txt	Crawlers, before they fetch	Says which URLs a crawler may fetch	Yes, for crawlers that honor it
sitemap.xml	Search engines	Lists the indexable URLs	No
llms.txt	AI agents, on demand	Summarizes the site and points to its key pages	No

The proposal draws the same lines: sitemaps "list all pages for search engines", while llms.txt "offers a curated overview for LLMs", and where robots.txt governs access, llms.txt is "used on demand". In an analysis of 584,107 llms.txt files reported by Search Engine Journal in September 2026, Common Crawl found that 6% carried rules about AI use, such as rate limits and copyright notices, which the format cannot enforce, and its crawler found 136,578 robots.txt files sitting at the llms.txt address. Common Crawl's summary of the format: it "grants nothing and blocks nothing, and no crawler is obliged to read it." Crawl rules belong in robots.txt.

Does llms.txt work?

For search rankings and AI citations, the evidence so far shows no measurable effect.

Google ignores it for Search. Google's generative AI guide lists llms.txt among things to ignore: you do not need "machine readable files, AI text files, markup, or Markdown to appear in Google Search", and keeping one "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." Google's John Mueller has compared the file to the keywords meta tag, a site owner's own claim about what the site is about, and later called it "a temporary crutch, perhaps to save some tokens" for AI coding tools reading developer docs, as Ahrefs reports.

Few systems request it. Ahrefs found that 28% of the 137,000 domains in its web analytics sample publish an llms.txt, a sample it notes is unusually tech-savvy, yet of about 38,000 domains with a valid file, 97% received no requests for it in May 2026. Semrush added one to Search Engine Land in March 2025 and found no link to AI performance; from mid-August to late October 2025 the file got no visits from Google-Extended, GPTBot, PerplexityBot or ClaudeBot.

Most files are generated, not written. In the Common Crawl sample, 68% came from a plugin or template, Wix alone accounted for 41%, and 22% contained no links at all.

The case for the file is narrower. AI labs publish llms.txt files for their own developer documentation, and the proposal reports coding agents following the files into API references. For developer docs, llms.txt is useful today. For a business site or blog, it is a cheap bet with no measured return, and it changes nothing about what gets a page cited: the work covered under AI search optimization and LLM SEO. To see whether that work lands, measure your AI visibility directly.

How to write an llms.txt file

1. Pick the pages that explain you. Products or services, pricing, the guides people cite, documentation. Semrush's advice is to favor "evergreen, canonical content" and skip pages that are gated, personalized or about to expire.
2. Write the summary for a reader with no context. Say what the business does, for whom, and any fact an assistant must not get wrong.
3. Describe every link. The proposal asks for concise, clear language, "brief, informative descriptions" and no unexplained jargon.
4. Publish it at /llms.txt on the site visitors use. On a headless WordPress setup that is the front end, not the CMS.
5. Test it. The proposal's own test is to ask an agent about your content "giving it only your llms.txt as a starting point."
6. Keep it current, and keep crawl rules out of it.

On WordPress, Yoast SEO and AIOSEO can generate the file, and the guide to WordPress SEO plugins covers the AI crawler settings that sit next to it.

How Structura handles this

Our own site publishes one. The file at structurawp.com/llms.txt is generated by the site on each request: the H1 and summary, sections for the product, comparisons, documentation, the blog and support, a Learn section listing every published Learn page with its one-paragraph definition, and an Optional section linking the XML sitemap and robots.txt. Individual blog posts are left out, so the file stays a short orientation rather than a second sitemap. We treat it as the cheap bet described above, not as a ranking tactic.

The WordPress plugin covers sites that have no other source for the file. When no supported SEO plugin is active, it serves a fallback /llms.txt that lists your newest published posts and pages, up to 100, each with its excerpt or opening words. When Yoast SEO, Rank Math, SEOPress or All in One SEO is active, it stands down and leaves the file to that plugin. On a headless install it does nothing, because the public site is not WordPress, and a headless space does not generate a file for your front end; you can build one from the Content API's post list.

What matters more for being cited is the posts themselves. Each opens with a direct answer, cites sources that are checked before publishing and carries structured data, which is the material an AI system quotes whether or not it ever reads a manifest. The AI SEO page sets out which parts of AI search Structura covers.

FAQ