Some recommendations spread faster than anyone checks them. llms.txt is one of those. The proposal sounds sensible: a text file at the root that tells AI systems, in tidy form, what this website is about — much as robots.txt tells search engines what they may read. Since 2024 it has appeared on practically every GEO checklist.
There is one problem. The systems it was written for barely request it. Analyses spanning hundreds of millions of bot events show that GPTBot and its peers fetch HTML and largely ignore /llms.txt. The share of AI-crawler traffic that touches that address at all is negligible.
The short version: llms.txt does no harm, costs little, and helps in a place almost nobody expects. It simply is not what it is sold as — and it replaces none of the three measures that actually govern who reads and cites your content.
1. What llms.txt was meant to be — and what became of it
The idea comes out of the developer-documentation world and is well meant. Large websites are hard for a language model to parse: navigation, ad slots, cookie banners and footers bury the actual content. An llms.txt at the root was meant to summarise, in Markdown, what the site is, which pages matter and where the full texts live.
The flaw is not the idea but the assumption underneath it — that the recipients would fetch it. A standard is not created by someone proposing a format. It is created when the other side reads it. With llms.txt, that has not happened yet. What the format was meant to be, and what a clean file looks like, we covered in our introduction to llms.txt.
2. The numbers: 7.4% of the Fortune 500, and almost no requests
Two measurements matter here — adoption and use. Both come out sober.
Two readings are possible, and both hold. First: a format with under eight per cent adoption among the world's largest companies is a proposal, not a standard. Second: adoption says nothing about usefulness — you have to check whether the recipients read it. Mostly, they do not.
Careful with absolute figures. The 7.4% refers to one cut-off date in March 2026 and one specific population. Among technical-documentation providers, adoption is considerably higher — that is where the format was born. Extrapolating the figure to "all websites" is simply wrong maths.
3. Who actually reads the file today
Interestingly, llms.txt is not without effect — the effect just sits elsewhere than the checklists claim. Today it is requested mainly by:
- Agentic browsers and AI assistants that are on your page right now on behalf of a user — not by training crawlers.
- Developer tooling that wants to make documentation machine-readable.
- Audit tools such as Lighthouse, which treat its presence as a signal.
That is a smaller group, but not an unimportant one. When an AI assistant reads your page on behalf of a prospect, that is the single most valuable moment there is — it does not get closer to a buying decision. It is simply a different argument from "so that ChatGPT knows you exist".
4. The crawlers that matter — and how they differ
Anyone who wants to steer visibility in AI answers first needs to know who is actually knocking. The tokens are public, and they do different jobs:
| Token | Provider | Purpose |
|---|---|---|
GPTBot | OpenAI | Collects content for model training |
OAI-SearchBot | OpenAI | Builds the index behind ChatGPT search — the basis for citations |
ChatGPT-User | OpenAI | Fetches a page on demand when a user asks for it |
ClaudeBot | Anthropic | The primary crawler; anthropic-ai and claude-web are also observed in logs |
PerplexityBot | Perplexity | Indexes for Perplexity search |
Perplexity-User | Perplexity | Fetches a page on demand for one specific user query |
Google-Extended | Governs only use for Gemini and Vertex AI — not Google Search | |
CCBot | Common Crawl | Public web dataset used widely as training data |
The row that causes most confusion in practice is Google-Extended. Blocking it does not remove you from Google Search, nor from AI Overviews — those run through ordinary Googlebot. All you block is use for Gemini training. We have more than once seen this line added out of caution and then blamed for traffic drops that came from somewhere else entirely.
5. robots.txt: the control that actually bites
If llms.txt is barely requested, what does control access? Still robots.txt. It is unglamorous, old, and honoured by the declared bots of the major providers. Two caveats belong with that:
- It is voluntary. robots.txt is a request, not a lock. Anyone who wants to ignore it can. In 2025 Perplexity was publicly accused of using undeclared crawlers to evade blocks — the accusation shows both that it happens and that it gets noticed.
- It governs reading, not knowing. Whatever a model learned before the block stays learned.
Important in practice: a named bot group in robots.txt replaces the rules from User-agent: * entirely — it does not add to them. Create a group for AI crawlers without repeating your general blocks inside it, and you have just lifted those blocks for exactly those bots. It is the single most common error we find in audits.
6. The distinction almost everyone misses
The most important distinction in this topic appears on no checklist, because it is inconvenient: training and citation are two different operations with two different crawlers — and most companies want different answers to each.
| Question | Who handles it | Typical SMB answer |
|---|---|---|
| May my content train a model? | GPTBot, ClaudeBot, CCBot, Google-Extended |
Usually yes — the content is public anyway, and being known helps |
| May I be cited in answers? | OAI-SearchBot, PerplexityBot, Googlebot |
Absolutely yes. This is the channel customers arrive through |
| May an assistant read my page right now? | ChatGPT-User, Perplexity-User |
Yes — there is a real prospect sitting in front of it |
Blocking "all AI bots" answers all three questions with no — closing the channel your customers arrive through in order to prevent a training process that rarely harms you. It is the most expensive mistake in this field. The group that loses most from a blanket block is the newest one: agentic AI browsers such as Atlas and Comet, which visit on a user's behalf.
7. What works instead
What gets cited is what a system understands quickly and can pass on in good conscience. That is less a question of files than of how your pages are built:
- The answer sits directly under the question. A heading phrased as a question, then two or three sentences that stand on their own. That block is what ends up in the citation.
- Figures with a date and a source. A number without provenance does not get cited — a system cannot verify it.
- Structured data that matches the page: Organization, Article, FAQ. To a machine, that is the page's abstract.
- Verifiable authorship. An author with a name, a role, and a page that shows why they would know.
- A visible update date — machine-readable, not only in prose.
- Reachable without detours. Content that only appears after scripts run does not exist for many crawlers.
None of this is new, and that is precisely the point. The measures that produce AI visibility are almost identical to those that produce ordinary findability. If someone sells you a separate "GEO strategy", ask how it differs from doing the job properly — and whether generative engine optimization and demonstrable authority under E-E-A-T appear in it at all.
8. Checklist: visibility in AI answers in 7 steps
OAI-SearchBot and PerplexityBot should not be blocked — the clicks come through them.9. Verdict: not wrong, just overrated
llms.txt is not a bad idea. It is an idea whose recipients have not started listening yet. Publishing one does nothing wrong — putting it at the top of your list confuses a preparation with an effect.
The three things that genuinely govern who reads and cites your content in 2026 are unglamorous: a considered robots.txt that separates training from citation. Content whose key statements are citable without context. And evidence a system can check. Everything else is optional.
If you do one thing: open your robots.txt and check whether a named group for AI bots exists. If it does, check that your general blocks are repeated inside it. That single check finds more real errors than any llms.txt you write afterwards. If you want a second opinion on where you stand, start with our AI Visibility Score.
10. FAQ: the most common questions on llms.txt and AI crawlers
Do I need an llms.txt?
Need: no. The major AI crawlers barely request it. It is useful for agentic assistants working on a user's behalf, and for developer tooling. If you have half an hour to spare, publish one — but only once your robots.txt is clean.
How many companies have an llms.txt?
Of the Fortune 500, exactly 37 had published one by 31 March 2026 — 7.4 per cent. Among technical-documentation providers adoption is considerably higher; that is where the format originated.
Do I block Google by blocking Google-Extended?
No. Google-Extended governs use for Gemini and Vertex AI only. Google Search and AI Overviews run through ordinary Googlebot and are unaffected.
Do AI providers honour robots.txt?
The major providers state publicly that their declared bots do, and as a rule they do. You should not rely on it: robots.txt is a request, not a technical lock. In 2025 Perplexity was accused of using undeclared crawlers to evade blocks. If you genuinely need to block, you need a block at server level.
Should I block AI crawlers entirely?
In most cases no — and if you do, decide separately. Blocking training crawlers is a legitimate stance. Blocking citation crawlers means disappearing from AI answers in which your competitors then stand alone.
What is the difference between llms.txt and robots.txt?
robots.txt says whether you may read, and is honoured. llms.txt wants to say what matters, and is barely requested. One is an access rule, the other a table of contents — and only the first has recipients.
How do I know whether I appear in AI answers?
The honest way: ask the ten questions your customers ask in ChatGPT, Perplexity and Google, and note who gets cited. That takes an hour and tells you more than any tool. Your server logs will additionally show which of the tokens above are visiting you. A two-minute version of the same check is described in Does ChatGPT know your business?.
Sources and status
The adoption figure (37 of 500 Fortune 500 companies, cut-off 31 March 2026) and the finding on low request rates come from an analysis of bot traffic data published in summer 2026. The crawler tokens and their roles follow provider statements and a 2026 summary overview. The 2025 accusation against Perplexity was made by Cloudflare; it is reproduced here as an accusation, not as an established fact.
We have deliberately not repeated the circulating percentage claims about "visibility gains through llms.txt" — no verifiable measurement basis could be found for any of them. All statements current as of 27 August 2026.