Table of Contents
Almost nothing reads it: across 137,210 domains measured in May 2026, 28% published an llms.txt file and 97% of those files were fetched zero times during that month. The bots performing those fetches were predominantly SEO crawlers and coding agents rather than the AI search engines on which the file is customarily sold.
Key Takeaways
- 97% of published llms.txt files received no requests whatsoever during May 2026, a figure drawn from the 137,210 domains examined in the Ahrefs server-log study.
- AI retrieval bots accounted for 1.1% of the requests. Slackbot, the link-preview bot of a chat application, fetched these files more frequently than PerplexityBot managed across the same period.
- No AI bot has ever sought a file that was absent. Requests directed at missing
/llms.txtURLs proved to be 98% human in origin. - Google states in writing that Search does not use the file, while Chrome’s Lighthouse simultaneously ships an audit flagging its absence.
- Publication is warranted where developers are your customers, or where your CMS generates the file at no cost. In no circumstance should an agency be paid to produce one.
The reason six different answers to this question exist is that in late May 2026 Google managed to argue both sides of it within the space of a single week.
A guide on optimising for generative AI features was published by Google Search carrying a section titled “mythbusting”, which informed site owners that special machine-readable files are not required for appearance in AI search. Days later an llms.txt check was shipped by the Chrome team inside Lighthouse, accompanied by documentation explaining that in the file’s absence agents may spend longer crawling a site in order to understand it.
One part of Google advises omission; another audits you for the omission. A means of settling the question without reliance on anyone’s opinion now exists: the server logs of 137,210 domains were pulled by Ahrefs and the parties actually fetching the file were counted.
What llms.txt Is, and What It Is Not
llms.txt is a single markdown file placed at the root of a website, at /llms.txt. It holds a short description of what the site is together with a curated list of links to its most important pages, so that a language model or an agent may orient itself without crawling everything. Jeremy Howard, co-founder of Answer.AI and fast.ai, proposed the format in September 2024.
Two characteristics are worth stating in the negative. It is not a robots.txt-style directive: despite the filename, nothing is controlled and nothing is blocked by llms.txt, which constitutes a suggestion rather than an instruction. Nor is it the practice of publishing markdown copies of every page, a separate tactic carrying separate problems with which this one is constantly conflated.
The “AI visibility” framing was attached later by the SEO industry as adoption spread, on the speculation that the file would begin to be rewarded by AI platforms. That speculation is what the log data tests.
Who Actually Reads llms.txt
Of the 137,210 domains studied, 28% published a valid llms.txt, and of those roughly 38,000 files 97% received zero requests during May 2026, from bots and humans alike.
The remaining 3%, approximately 1,100 domains, absorbed every request measured. That pool is where the interest lies.
That chart is best read from the bottom upward, because the bottom is the part being sold to you.
The AI Categories Add Up to 19.5%, and Coding Agents Are Most of It
AI retrieval bots account for 1.1% of requests. These are the crawlers fetching pages in order to answer live questions within AI search products: OAI-SearchBot, PerplexityBot, and Claude’s search crawler. Between them a couple of hundred fetches were made across thousands of sites. Slackbot, the link-preview bot within a chat application, fetched llms.txt files more frequently than PerplexityBot did.
Combining all four AI categories produces 19.5%. That is the generous figure, though its composition is worth noting: AI agents and coding infrastructure at 10.5% and training crawlers at 5.3% perform most of the work. GPTBot is the largest single fetcher at 4.51%, with Claude-Code second, ahead of every AI search bot on the list.
Nothing Goes Looking for a File That Is Not There
Then there is the finding that ends the argument. Requests to /llms.txt URLs returning a 404 were also examined by Ahrefs, and the AI bot share of those was zero. Missing files drew 98% human traffic, mostly people typing the URL into a browser in order to check on a competitor.
Nothing goes looking. Where an AI tool fetches your llms.txt, it is because a link, an index or a user instruction indicated that the file was present.
Why Google Says One Thing and Its Own Tool Does Another
Google’s position on llms.txt is consistent from the Search side and muddled everywhere else.
It was stated by Gary Illyes at Search Central Live in July 2025 that llms.txt is not supported by Google and is not planned.
Google’s own AI optimisation guide puts it in writing: the creation of machine-readable files, AI text files, markup or markdown is not required for appearance in Google Search, including its generative capabilities, because Search does not use them. The idea has separately been compared by Mueller to the keywords meta tag, the signal that was stuffed until Google ceased reading it.
Pressed on the contradiction with the Lighthouse audit, Mueller’s answer was that llms.txt is “not done for search”. It was described as a temporary crutch, perhaps to save some tokens, for AI coding tools reading developer documentation.
The log data agrees with him. Claude-Code out-fetched every AI retrieval bot, every AI assistant and every training crawler with the exception of GPTBot. The audience for this file is coding agents reading API documentation, which is precisely the audience for which the format was designed and almost never the audience to which it is sold.
For the record, the Lighthouse audit that began the argument produced 22 requests across the whole dataset. Roughly one fetch in a thousand.
What This Means If You Run a Small Business Website
Almost every guide to llms.txt is written about documentation sites. What follows is the version for a business that sells something.
Publish It If You Sell to Developers
Where developers are your customers, or where your documentation constitutes the product itself, publication is warranted, that being the single case the measured evidence supports.
Leave It On If Your Platform Makes It for Free
Where your platform generates the file at no cost, it should be left in place. Wix already does so, while Framer and Lovable scan for it, and within a year the file may be as ordinary a CMS default as a sitemap, a file requiring no work of your own carrying no cost at all.
Skip It If You Are a Service Business
Where you operate a clinic, a villa, a studio or a shop in the hope that the file will produce a citation from ChatGPT, the effort is not warranted and no payment should be made for one. The bots determining AI citations sent 1.1% of the requests here, and they have never once sought a file that was absent.
We have written separately about what the evidence does support for AI visibility, and none of it is a text file.
One action is worth taking in either case, and it carries no cost: establishing whether anything has ever fetched yours.
How to Check Whether Anything Has Fetched Yours
Your hosting control panel should be opened, the raw access logs located, and a search performed for llms.txt, most shared hosts retaining 30 days of them. A 200 indicates that something fetched it, and the user agent beside it should be checked before any celebration, because on this evidence it is four times more likely to be an SEO crawler than an AI tool.
A 404 is the more useful result, indicating that something requested a file you never published, which is the only genuine signal in this topic and the one nobody checks. Fetched is also not read: a bot may request the file and ignore every word within it, so every figure above constitutes a ceiling on consumption rather than a measurement of it.
What Goes In One, If You Publish It
The format is deliberately plain markdown: an H1 carrying the site name, a blockquote summarising what the site is, then H2 sections holding lists of links each with a short description. An optional section marks links that may be skipped.
Two rules matter considerably more than the syntax itself.
Link to It
Agents fetch llms.txt where something directs them to it rather than speculatively, which means an unlinked file sitting at your root is precisely the 97% case described above.
Treat It Like Code, Not Copy
The largest single research crawler in the Ahrefs dataset identified itself as prompt-injection-survey/1.0. llms.txt is being systematically studied by somebody as a means of feeding instructions to agents built to trust it. The file should be version-controlled, edit access restricted, the contents kept to plain links and descriptions with nothing instruction-shaped within them, and anything a platform generates on your behalf reviewed.
Common Questions
Does llms.txt help my website appear in ChatGPT, Perplexity or Google AI Overviews?
No. It is stated in Google’s own documentation that machine-readable files such as llms.txt are not used by Search for any of its features, generative ones included. The server-log evidence points the same way: across 137,210 domains, AI retrieval bots such as OAI-SearchBot and PerplexityBot accounted for 1.1% of all requests to these files, and no AI bot ever requested one from a site that had not published it. Where citation in AI answers is the objective, this file is not the lever.
How many websites actually publish an llms.txt file in 2026?
Ahrefs found that 28% of the 137,210 domains studied published a valid llms.txt as of May 2026, which is approximately 38,000 sites. Ahrefs cautions that its analytics customers skew more technical and SEO-aware than the web at large, so 28% should be treated as an upper bound rather than a general adoption rate. The figure has risen rapidly, driven by CMS defaults and speculation rather than by any AI platform confirming that the file is read.
Which bots and AI tools actually fetch llms.txt files?
Coding agents, mostly. Of the 3% of files receiving any request at all, AI agents and agentic infrastructure sent 10.5%, with GPTBot the largest single fetcher at 4.51% and Anthropic’s Claude-Code second. SEO audit tools constituted the biggest category overall at 21.7%, ahead of every AI category individually. AI assistants sent 2.5% and AI retrieval bots 1.1%. This is a developer-tooling file carrying a marketing story.
Should a small business pay an agency to create an llms.txt file?
No, and a proposal including one as a paid line item is worth questioning. The file requires a few minutes to write and most website platforms, Wix among them, now generate it automatically at no cost. No measured visibility benefit exists for a service business, and Google’s documentation states plainly that no special file is required. Where an agency is charging for llms.txt, the question of what else on that proposal was chosen the same way is worth asking.
Is there any risk in publishing an llms.txt file?
A small one, and not the one commonly expected. Agents are built to trust this file, which renders it a target: the largest research crawler in the Ahrefs data was studying llms.txt specifically as a prompt-injection surface. The file should be kept to plain links and descriptions, anything phrased as an instruction to a model avoided, links restricted to resources under the publisher’s control, version control applied, and whatever the CMS generates reviewed.
Will llms.txt matter more in the future than it does now?
Possibly, and that is the honest answer rather than a hedge. The one group measurably reading these files today is agents, and Google has stated repeatedly that search is expected to become more agentic. Where agents end up mediating AI search rather than retrieval bots fetching pages directly, this file could begin to matter through that layer. None of that is true yet, so the sensible position is to keep the cost at zero until it is.
For those wishing to establish what is actually worth doing on a given site, that is the purpose of our audit, and the services page lists what follows from it.
Sources
- llms.txt specification, Jeremy Howard, Answer.AI, proposed September 2024
- Google: Optimizing your website for generative AI features, Search Central
- Chrome for Developers: Lighthouse agentic browsing llms.txt audit
- Ahrefs: We analysed 137K sites, 97% of llms.txt files never get read, Louise Linehan and Xibeijia Guan, 15 June 2026, 137,210 domains, May 2026 server logs
- Gary Illyes, Search Central Live, July 2025, on Google not supporting llms.txt
- John Mueller, 2026, on llms.txt as a “temporary crutch” for AI coding tools
