llms.txt Explained

An llms.txt file is usually published at /llms.txt. It introduces a site, product, documentation collection, or other content area and links to a selected group of important pages. It can offer compatible agents a concise orientation point when a site has a large documentation set or complex navigation.

The strongest file is short and editorially maintained. It should identify the organization or project, explain its purpose in plain language, and link to canonical pages with useful descriptions. It is a curated map, not a dump of every URL on a domain.

Do not use llms.txt to grant or deny crawler access. That decision belongs in robots.txt for bots that document support for robots directives. Do not use it as a substitute for an XML sitemap, which helps search engines discover canonical URLs.

It is also not a special form of schema markup or an AI ranking signal. Publishing an llms.txt file cannot compensate for pages that are blocked, not indexed, thin, inaccurate, inaccessible, or difficult to navigate.

Start with a single H1 containing your company, project, or documentation name. Follow with a one- or two-sentence summary. Group a small selection of high-value links under clear Markdown H2 headings such as Product, Documentation, API, and Learn.

Use absolute HTTPS URLs, link only to pages you want represented, and review the file as part of your regular documentation release process. If your documentation includes generated reference pages, prioritize the few overview and getting-started pages that give an agent useful context.

Confirm that priority content is crawlable and indexable.

Maintain robots.txt and XML sitemaps based on each crawler’s documented behavior.

Create helpful people-first pages with descriptive headings and stable canonical URLs.

Add llms.txt only when you can keep a curated file accurate over time.