What Is llms.txt and Should Your Site Have One?

Last updated: 6 July 2026

Changelog: 27 February 2026: first published. 6 July 2026: adoption claims re-checked against vendor documentation, Google statements and Ahrefs log data.

Disclosure: some links on this page are affiliate links. If you sign up through one we may earn a commission at no extra cost to you. We only recommend tools we would genuinely use.

llms.txt (often typed as llm.txt in searches) is a proposed standard: a plain markdown file at the root of your website that gives large language models a short, curated map of your most important content. As a London SEO agency, the question we hear about it is nearly always the same: do we need one? The short answer: it costs almost nothing to add, and as of July 2026 there is no confirmed evidence that any major AI system reads it. This guide covers what the file is, where it came from, who supports it (and who does not), and how to write one properly if you decide to bother.

What is llms.txt?

llms.txt is a proposed text file, written in markdown, that sits at the root of a website (for example https://example.com/llms.txt) and gives large language models a curated index of the site’s key pages: what the site is, what it offers, and where the most useful content lives.

The thinking behind it is straightforward. LLMs work with limited context windows, and most web pages are cluttered with navigation, adverts, scripts and boilerplate that waste that context. A single tidy markdown file lets a site say: here is who we are, and here are the twenty pages that actually matter, each with a one line description. Markdown was chosen because both humans and language models read it easily.

Is it llm.txt, llm txt, or llms.txt?

The correct filename is llms.txt, with an s: it stands for “large language models”, plural. Searches for llm.txt or llm txt are common misspellings of the same proposed standard, and every variant refers to one file: a markdown document served from your site root.

The filename matters in practice. If you publish the file as llm.txt (singular), any tool that does look for the standard location will miss it, because the proposal specifies /llms.txt exactly. Keep the file lowercase, at the root of the domain rather than in a subfolder, and served as plain text.

Who created llms.txt and when?

Jeremy Howard, co-founder of Answer.AI, published the llms.txt proposal on 3 September 2024 at llmstxt.org. It was pitched as a web convention in the spirit of robots.txt and sitemap.xml: a predictable location where AI systems could find LLM-friendly information about a site.

The proposal defines a specific structure. An llms.txt file should contain, in order: an H1 with the name of the site or project (the only required element), a blockquote with a short summary, optional freeform detail, then one or more sections under H2 headings containing lists of markdown links, each with an optional one line note. A section headed “Optional” has special meaning: anything under it can be skipped when a shorter context is needed.

What is the difference between llms.txt and llms-full.txt?

llms.txt is the index: a short, curated list of links with descriptions. llms-full.txt is a companion convention where the entire content of those pages is compiled into one large markdown file, so an AI system can load everything in a single request without crawling.

The comparison in brief:

	llms.txt	llms-full.txt
Contains	Site name, summary, curated links with notes	The full text of the key pages in one markdown file
Typical size	A few kilobytes	Can run to megabytes
In the original proposal	Yes	No, a convention that grew up around it
Best suited to	Any site wanting to offer a map	Documentation-heavy sites

llms-full.txt has mostly been adopted by developer documentation platforms. Anthropic, for example, publishes both files for its own docs. For a typical business website, the plain llms.txt index is the only version worth considering.

Do any AI companies actually use llms.txt?

As of July 2026, no major AI vendor has publicly confirmed that its crawlers or assistants fetch llms.txt. That is the single most important fact about this file, and it is worth being blunt about, because a lot of coverage implies adoption that has not happened.

The position by vendor, as of July 2026:

• Google’s John Mueller wrote in April 2025 that, as far as he knew, none of the AI services had said they were using llms.txt, adding that server logs show they do not even check for it. He compared it to the old keywords meta tag.
• OpenAI documents robots.txt directives for its crawlers but has made no commitment to llms.txt.
• Anthropic publishes llms.txt and llms-full.txt for its own documentation, but has not stated that its crawlers read the file on other people’s sites.
• Meta has given no indication of support.

The strongest independent evidence comes from Ahrefs, which analysed 137,210 domains and found roughly 38,000 of them serving a valid llms.txt file: 97% of those files received zero requests in May 2026. Not from bots, not from anyone. Ahrefs also found that AI bots never requested llms.txt on domains that lacked one, which means they are not even looking for it. Plenty of sites publish the file; almost nothing fetches it.

How does llms.txt relate to robots.txt and sitemap.xml?

llms.txt complements robots.txt and sitemap.xml rather than replacing either. robots.txt controls what crawlers may access, sitemap.xml lists every URL you want indexed, and llms.txt offers a curated, human-readable summary of your best content. Only the first two are honoured by major platforms today.

File	Job	Audience	Status as of July 2026
robots.txt	Tells crawlers what they may and may not fetch	All crawlers, including AI bots	Long established and widely honoured
sitemap.xml	Lists every URL you want discovered	Search engine crawlers	Long established, supported by all major engines
llms.txt	Offers a curated map of key content	LLMs and AI assistants	Proposal only; no confirmed major consumer

One point trips people up constantly: llms.txt cannot block anything. It is an invitation, not a gate. If you want to stop AI crawlers taking your content, that is a robots.txt job (and a separate debate). An llms.txt file has no enforcement power at all.

Should your site have an llms.txt file?

Our honest verdict: llms.txt is a low cost bet with an unproven payoff, so add one if it takes minutes, and skip it if it takes hours. No AI vendor has confirmed support as of July 2026, so nobody should promise you traffic or citations from it.

The case for adding one anyway: the file is cheap, harmless when done properly, and if any assistant does start checking the standard location, early publishers benefit without lifting a finger. We build client sites on Astro as fast static HTML, and on a build like that an llms.txt is a few minutes of work, so the cost side of the equation barely registers.

The case against: it is one more file to keep accurate. A stale llms.txt pointing at dead or renamed URLs is worse than none. And every hour spent on speculative files is an hour not spent on things that measurably move AI visibility: clean crawlable HTML, answer-first copy, and pages that actually deserve citing, all of which we cover in our guide to optimising for AI Overviews.

Our own position: we sit on the “worth an hour, not a week” side of the argument, and we practise it. Luckywebs publishes its own file at luckywebs.co.uk/llms.txt, which doubles as a working example of the format described below.

How do you write an llms.txt file?

To write an llms.txt file, list your 10 to 30 most useful URLs, then present them in markdown: an H1 with your site name, a blockquote summary, and links grouped under H2 headings with a one line note each. Save it as plain text at yourdomain.com/llms.txt.

A worked generic example for a service business:

# Acme Heating Ltd

> Acme Heating is a Cambridge boiler installation and servicing firm.
> We publish plain English guides on boiler costs, servicing and
> central heating upgrades for UK homeowners.

## Services

- [Boiler installation](https://www.example.co.uk/boiler-installation/): scope, brands fitted and typical timescales
- [Boiler servicing](https://www.example.co.uk/boiler-servicing/): what an annual service includes and pricing
- [Central heating upgrades](https://www.example.co.uk/heating-upgrades/): radiators, smart controls and system conversions

## Guides

- [New boiler cost guide](https://www.example.co.uk/guides/new-boiler-cost/): itemised UK pricing, updated quarterly
- [Combi vs system boilers](https://www.example.co.uk/guides/combi-vs-system/): which suits which property type

## Optional

- [About us](https://www.example.co.uk/about/): company history and accreditations

Three rules keep the file useful. Curate hard: this is a highlights reel, not a second sitemap, so leave out tag pages, thin pages and anything you would not want quoted. Describe honestly: the one line notes are there to help a model pick the right page, not to stuff keywords. Review it whenever you restructure the site, because broken links in a curated file undermine the whole point.

What else do people ask about llms.txt?

These are the questions people ask most often about llms.txt, answered briefly. Every answer reflects the position as of July 2026: the file is still a proposal, no major AI vendor has confirmed reading it, and nothing about it changes how Google crawls, indexes or ranks a site.

Is llms.txt an official web standard?

No. llms.txt is a community proposal published in September 2024, not a ratified standard from a body like the W3C or IETF. Nothing obliges any AI company to read it, and as of July 2026 none of the major vendors has committed to doing so.

Does llms.txt improve Google rankings?

No. Google’s John Mueller has said that no AI service he knows of uses llms.txt, and he compared it to the old keywords meta tag. There is no evidence it affects organic rankings, and no credible evidence links it to AI Overviews visibility either.

Can llms.txt stop AI companies scraping my content?

No. llms.txt only offers content; it cannot restrict access. If you want to limit AI crawlers, use robots.txt directives for the specific bots (and be aware compliance is voluntary there too). The two files do opposite jobs.

Where exactly should the file live?

At the root of your domain: https://yourdomain.com/llms.txt. Subdirectory locations are not part of the proposal, and any tool checking the standard path would miss a file buried elsewhere. Serve it as plain text with UTF-8 encoding.

How long should an llms.txt file be?

Short. The value of the file is curation, so a focused list of 10 to 30 links with clear one line notes beats a dump of every URL. If you feel the urge to include everything, that job already belongs to your XML sitemap.

Do I need llms-full.txt as well?

Almost certainly not. llms-full.txt suits documentation platforms where a single compiled file of full page content is genuinely useful to developers and AI tools. For a normal business site, the llms.txt index alone covers the proposal.

---

Whether or not you publish an llms.txt, the thing worth measuring is whether AI assistants actually mention your brand. SE Ranking’s AI visibility toolkit tracks brand mentions and links across Google AI Overviews, AI Mode, ChatGPT, Gemini and Perplexity (as of July 2026), which turns the AI search question from guesswork into a report.

Start Your Free Trial