Does Anything Actually Read Your llms.txt? The 2026 Server-Log Evidence
Almost nothing reads it. Across 137,210 domains, 28% publish an llms.txt and 97% of those files were fetched zero times in a month. Here is who does read them.
Who reads llms.txt, according to server logs
Table of Contents9 sections
Your web developer, an SEO tool or a Lighthouse report has probably told you that your site is missing an llms.txt file, and that you need one for AI search.
The short version: almost nothing reads it. Across 137,210 domains measured in May 2026, 28% published an llms.txt file, and 97% of those files were fetched zero times that month. Most fetches that did happen came from SEO crawlers and coding agents, and the AI search engines barely showed up.
So why do you find six different answers to this question online? Because in late May 2026, Google managed to argue both sides of it in the same week:
- Google Search published a guide on optimising for generative AI features, with a "mythbusting" section telling you, the site owner, that you don't need special machine-readable files to appear in AI search.
- A few days later, the Chrome team shipped an llms.txt check inside Lighthouse, with documentation saying that without the file, agents may spend longer crawling your site to understand it.
One part of Google tells you to skip it. Another part audits you for skipping it. I found that funny.
Luckily you and I don't have to take anyone's opinion on this anymore, because Ahrefs pulled the server logs of 137,210 domains and counted who actually fetches the file.
Key Takeaways
- 97% of published llms.txt files got no requests at all in May 2026, across the 137,210 domains in the Ahrefs server-log study.
- AI retrieval bots sent 1.1% of the requests. Slackbot, a chat app's link-preview bot, fetched these files more often than PerplexityBot did over the same period.
- No AI bot has ever gone looking for a missing file. Requests to missing
/llms.txtURLs were 98% human. - Google says in writing that Search doesn't use the file, while Chrome's Lighthouse ships an audit that flags it when it's missing.
- Publish one if developers are your customers, or if your CMS makes it for free. Don't pay an agency to write one, ever.
What llms.txt Is, and What It Is Not
llms.txt is a single markdown file at the root of a website, at /llms.txt. You can open one in a browser tab like any other page.
It holds a short description of the site and a hand-picked list of links to its most important pages, so a language model or an agent can find its way around without crawling everything. Jeremy Howard, co-founder of Answer.AI and fast.ai, proposed the format in September 2024.
Two things it isn't:
- It isn't a robots.txt-style directive. Despite the name, llms.txt doesn't control or block anything; it's a suggestion, and nobody has to follow it.
- It isn't the practice of publishing markdown copies of every page, which is a separate tactic with separate problems, though people mix the two up all the time.
If someone pitches you one, ask which they mean.
The SEO industry added the "AI visibility" story later, as more sites published the file, betting that AI platforms would start rewarding it. That bet is what the log data tests. So who actually fetches the file?
(What does show up in AI answers is ordinary work: a current Google profile, consistent listings and pages that answer real questions, which is what our Local SEO service covers.)
Who Actually Reads llms.txt
Here's how many published one. Of the 137,210 domains studied, 28% published a valid llms.txt. Of those roughly 38,000 files, 97% received zero requests in May 2026, from bots and humans alike.
The other 3%, about 1,100 domains, took every request measured. Who were they? That's where it gets interesting.
| Who sent the request | Share of llms.txt requests |
|---|---|
| SEO audit tools | 21.7% |
| Unidentified bots | 14.9% |
| General web crawlers | 13.1% |
| Technology profiling tools | 11.6% |
| AI agents and coding infrastructure | 10.5% |
| GEO and AEO scoring tools | 5.8% |
| AI training crawlers | 5.3% |
| llms.txt discoverability bots | 3.6% |
| Social preview bots | 2.9% |
| Research bots | 2.7% |
| AI assistants | 2.5% |
| AI retrieval bots | 1.1% |
Read that table from the bottom up. The bottom is the part someone is selling you.
The AI Categories Add Up to 19.5%, and Coding Agents Are Most of It
AI retrieval bots sent 1.1% of requests. These are the crawlers that fetch pages to answer live questions inside AI search products:
- OAI-SearchBot
- PerplexityBot
- Claude's search crawler
Between them, they made a couple of hundred fetches across thousands of sites. Slackbot, the link-preview bot inside a chat app, fetched llms.txt files more often than PerplexityBot did.
That one made me laugh. It might make you laugh too.
Notice something? Add all four AI categories together and you get 19.5%. That's the generous number.
Now look at what's in it: AI agents and coding infrastructure at 10.5%, and training crawlers at 5.3%, do most of the work. GPTBot is the largest single fetcher at 4.51%, with Claude-Code second, ahead of every AI search bot on the list.
Nothing Goes Looking for a File That Is Not There
Then comes the finding that settles it. Ahrefs also looked at requests to /llms.txt URLs that returned a 404, and the AI bot share of those was zero. Missing files drew 98% human traffic, mostly people typing the URL into a browser to check on a competitor.
Nothing goes looking for your file. If an AI tool fetches your llms.txt, it's because a link, an index or a user's instruction told it the file was there.
Why Google Says One Thing and Its Own Tool Does Another
On the Search side, Google's position is clear to me. Everywhere else, you'll find it muddled.
Gary Illyes said at Search Central Live in July 2025 that Google doesn't support llms.txt and has no plans to.
Google's own AI optimisation guide puts it in writing: you don't need machine-readable files, AI text files, markup or markdown to appear in Google Search, generative features included, because Search doesn't use them. Mueller has also compared the idea to the keywords meta tag, the signal everyone stuffed until Google stopped reading it.
| Google Search | Chrome Lighthouse | |
|---|---|---|
| Position | Not supported and not planned | An experimental agentic browsing audit checks for the file |
| What it says | No special files are needed for generative AI features | Agents may crawl longer without it |
| What it produced in the logs | Nothing | 22 requests in the month, about 1 in every 1,000 fetches |
Asked about the contradiction with the Lighthouse audit, Mueller said llms.txt is "not done for search". He called it a temporary crutch, maybe to save some tokens, for AI coding tools reading developer documentation.
The log data backs him up. I checked twice. Claude-Code out-fetched every AI retrieval bot, every AI assistant and every training crawler except GPTBot.
So who actually fetches the file? Coding agents reading API documentation. That's exactly who the format was designed for, and almost never who it gets sold to.
For the record, the Lighthouse audit that started the argument produced 22 requests across the whole dataset. Roughly one fetch in a thousand. I wouldn't lose sleep over that audit, and you shouldn't either.
What This Means If You Run a Small Business Website
Almost every guide to llms.txt is written about documentation sites.
This is the version for a business that sells something. Is that you? Then read on.
| Your site | Verdict | Why |
|---|---|---|
| Software documentation, API reference or developer tools | Publish it | Coding agents are the one measured readership |
| Your customers use coding agents to shortlist suppliers | Publish it and link to it | Agents fetch only when directed |
| Your CMS generates it automatically, as Wix does | Leave it on | It costs nothing |
| A service business hoping for ChatGPT or Perplexity citations | Skip it | AI retrieval bots sent 1.1% of requests and never ask for a missing file |
| Any of these, through an agency invoice | Don't pay for it | The file takes ten minutes |
Publish It If You Sell to Developers
If developers are your customers, or your documentation is the product, publish it. That's the one case the measured evidence supports, and I'd do it today.
Leave It On If Your Platform Makes It for Free
If your platform makes the file for you, leave it there.
Wix already does, Framer and Lovable scan for it, and within a year it may be as ordinary a CMS default as a sitemap. It costs you nothing. Your time stays free too.
Skip It If You Are a Service Business
Do you run a clinic, a villa, a studio or a shop, and hope the file will get you cited in ChatGPT? Don't bother, and don't pay anyone for one. The bots that decide AI citations sent 1.1% of the requests here, and they've never once gone looking for a missing file.
We've written separately for you about what the evidence does support for AI visibility, and none of it is a text file.
The one sure spot in ChatGPT right now is a paid one: an ad under its answers, live in Indonesia since September.
Whatever you decide, there's one free thing worth doing: find out whether anything has ever fetched yours. You can do it in five minutes.
How to Check Whether Anything Has Fetched Yours
Open your hosting control panel. Find the raw access logs and search for llms.txt. Most shared hosts keep 30 days of them.
Did you get a 200? Then something fetched it. Check the user agent beside it before you celebrate, because on this evidence it's four times more likely to be an SEO crawler than an AI tool.
A 404 is better news for you. It means something asked for a file you never published, and that's the only real signal in this whole topic. Nobody checks for it.
And fetched isn't the same as read: a bot can request the file and ignore every word in it. So every figure above is a ceiling. Your real number is probably lower.
What Goes In One, If You Publish It
The format is plain markdown on purpose: an H1 with the site name, a blockquote summing up what the site is, then H2 sections with lists of links, each with a short description. An optional section marks links an agent can skip. You can write yours in ten minutes:
| Part | What goes in it |
|---|---|
H1 | Your business name |
| Blockquote | One or two sentences on what the business does and where |
H2 sections | Links to your main pages, each with a short description |
| Optional section | Links an agent may skip |
If your site is built around a pillar page and its cluster, those pages are the obvious first links.
Two rules matter a lot more than the syntax. I'd follow both.
Link to It
Agents fetch llms.txt when something points them to it. They don't go looking on the off chance.
So an unlinked file at your root is the 97% case. Link yours.
Treat It Like Code, Not Copy
The largest single research crawler in the Ahrefs dataset called itself prompt-injection-survey/1.0. Someone is studying llms.txt, systematically, as a way to feed instructions to agents that are built to trust it. So keep your file under version control and limit who can edit it.
Keep it to plain links and descriptions, with nothing in it that reads like an instruction. And check anything your platform generates. I'd read it once a month.
Common Questions
Does llms.txt help my website appear in ChatGPT, Perplexity or Google AI Overviews?
No. Google's own documentation says Search doesn't use machine-readable files like llms.txt, generative features included, and no AI bot ever asked for one from a site that hadn't published it. If an AI tool fetches your llms.txt, it's because a link, an index or a user's instruction told it the file was there.
How many websites actually publish an llms.txt file in 2026?
Ahrefs found that 28% of the 137,210 domains it studied published a valid llms.txt as of May 2026, about 38,000 sites. Its analytics customers are more technical and SEO-aware than the web at large, so I'd treat 28% as an upper bound for your market.
Which bots and AI tools actually fetch llms.txt files?
Coding agents reading API documentation, which is exactly who the format was designed for, and almost never who it gets sold to. AI agents and agent infrastructure sent 10.5% of requests, with GPTBot the largest single fetcher at 4.51% and Claude-Code second. SEO audit tools were the biggest category at 21.7%, AI assistants 2.5% and AI retrieval bots 1.1%.
Should a small business pay an agency to create an llms.txt file?
No, and if your proposal lists one as a paid line item, I'd question it. The file takes a few minutes to write, and platforms like Wix now make it for free. There's no measured visibility benefit for a service business, and Google says plainly that you don't need a special file.
Is there any risk in publishing an llms.txt file?
A small one: agents are built to trust this file, and the largest research crawler in the Ahrefs data was studying llms.txt as a prompt-injection surface. Keep yours to plain links and descriptions, with nothing that reads like an instruction to a model. Keep it under version control and check whatever your CMS generates.
Will llms.txt matter more in the future than it does now?
Maybe, and I mean that as a real answer. The only group that measurably reads these files today is agents, and if agents end up sitting between people and AI search, the file could start to matter through that layer. None of that has happened yet, so keep what you spend on it at zero.
Want to know what's actually worth doing on your own site? That's what our marketing audit is for, and monthly local SEO is the ongoing version of the work. If a free scan is what flagged your llms.txt in the first place, read what a free crawler scan can't see.
Sources
- llms.txt specification, Jeremy Howard, Answer.AI, proposed September 2024
- Google: Optimizing your website for generative AI features, Search Central
- Chrome for Developers: Lighthouse agentic browsing llms.txt audit
- Ahrefs: We analysed 137K sites, 97% of llms.txt files never get read, Louise Linehan and Xibeijia Guan, 15 June 2026, 137,210 domains, May 2026 server logs
- Gary Illyes, Search Central Live, July 2025, on Google not supporting llms.txt
- John Mueller, 2026, on llms.txt as a "temporary crutch" for AI coding tools
