llms.txt: What It Is, What It Is Not, and How to Create One

The llms.txt proposal places a Markdown-formatted file at the root of a website, usually /llms.txt. It gives a short explanation of the site and a curated list of important resources. An optional llms-full.txt can provide expanded context when a concise index is not enough.

The useful mental model is a guided map for tools that choose to read it. It can summarize what the organization does, point to canonical documentation, and make high-value resources easy to locate without interpreting a complex navigation system.

It is not the same as robots.txt. It does not provide authentication, grant a content license, or universally control training and search crawlers. It is also not a confirmed ranking factor across AI or traditional search systems. Provider support and behavior can vary.

Do not let the file become a substitute for crawlable pages, a valid sitemap, clear navigation, structured data, or helpful content. If the linked pages are blocked, vague, stale, or contradictory, a perfect index cannot repair them.

Begin with one H1 containing the site or product name, followed by a short blockquote that explains the primary purpose. Add one or two paragraphs of factual context. Then create H2 sections with Markdown links to important pages and a plain-language description after each link.

Prefer canonical public pages: product overview, documentation, API reference, key guides, pricing, company information, contact, and policies. Keep the list intentionally small. A dump of every URL recreates the sitemap in a less structured format.

Build the file from the same content registry or CMS that produces navigation and the XML sitemap. Validate URLs during the build, use absolute HTTPS links, and deploy it with a text/plain or Markdown-compatible content type. Keep it outside authentication and redirect loops.

Assign an owner and update the file when products, documentation paths, pricing, or policies change. A stale AI context file can be worse than no file because it creates a confident path to obsolete information.

Publish llms.txt after the technical and editorial foundation is sound. Fix unintended blocking, inaccessible content, ambiguous brand identity, thin documentation, missing author information, and broken internal links first. Those improvements help humans, search engines, and multiple AI systems regardless of llms.txt support.

For a documentation-heavy product with an existing content registry, generating the file can be a low-effort enhancement. For a five-page brochure site with unclear copy, rewriting the core pages is more valuable than adding another discovery file.

A static llms.txt file fits naturally in S3 behind CloudFront, but the operational details still matter. Serve it from the canonical hostname at `/llms.txt`, use a plain-text content type, keep the body human-readable, and make the cache lifetime short enough that important documentation changes propagate predictably. Publish a more detailed companion only when the site can maintain it accurately.

Use a private S3 bucket with CloudFront origin access control for production rather than a public website bucket. Deploy the file with infrastructure or content releases, verify the public object after upload, and alert when it becomes missing or unexpectedly empty. The cost-conscious AWS web architecture shows how static delivery and a serverless backend can remain cleanly separated.

The file should point to useful canonical pages, not attempt to replace crawlable navigation. Include the homepage, documentation or product overview, important policies, and a few durable knowledge hubs. Revisit the selections during the AI visibility audit, and remove destinations that redirect, disappear, or no longer represent the best source.