Do you need an llms.txt?

Short answer: Most sites don't. Google Search says it ignores llms.txt [1], OpenAI and Anthropic don't say their crawlers read it [2][3], and of about 38,000 sites with a valid llms.txt in Ahrefs' May 2026 logs, 97% got no requests for it all month [4]. It does real work for software documentation that developers point coding assistants at. If your SEO plugin already makes one, leaving it on does no harm in Google; keep the pages it lists current for the tools that do read it.

We make an SEO audit tool that does not check for llms.txt, and we keep one on this site anyway, at the cost of a line per new page.

What is llms.txt?

llms.txt is a proposed file, written in Markdown and placed at the root of a site (yoursite.com/llms.txt), that gives AI tools a short summary of the site and links to its most useful pages. Jeremy Howard published the proposal in September 2024 and revised it in August 2026, and still describes it as a proposal open to community input [5][6].

The format has one required part, a heading with the site's name. After it come a short summary set as a quote, optional notes, and sections of links, each link optionally followed by a few words on what the page holds [5]. We write ours by hand, and it is built like this:

The proposal also suggests a plain Markdown copy of each page at the same address with .md added [5]. A related file, llms-full.txt, which puts a whole documentation site into one file, is generated by documentation platforms such as Mintlify; the current version of the proposal doesn't define it [5][7].

Does Google or ChatGPT read it?

Google Search does not: its guide to optimizing for AI features lists llms.txt among the things a site doesn't need, and adds:

Googlebot may still fetch the file, as it fetches many kinds of files on a site, and Google says that does not mean the file is treated in a special way [1]. Ahrefs counted about 900 such Googlebot fetches in May 2026 [4].

OpenAI's and Anthropic's pages on their crawlers explain what each one does and how to control it with robots.txt (the file that tells crawlers which pages they may fetch), and neither says its crawlers read other sites' llms.txt [2][3]. Both companies publish an llms.txt for their own developer documentation [8][9]. That helps developers' coding tools find those docs; it is not evidence that ChatGPT or Claude reads the file on your site. Ahrefs found no major AI platform that has committed to reading llms.txt [4].

Who does read it?

Mostly bots, and rarely AI search crawlers. Among them are OpenAI's training crawler, Anthropic's coding assistant and tools that test for the file. Ahrefs looked at May 2026 server logs from 137,210 sites that use its web analytics. 28% published an llms.txt, and about 38,000 had a valid one [4]:

• 97% of those files got no requests in the month.
• 96% of the requests that did arrive came from bots, not people.
• Named AI tools made 19.5% of the fetches. OpenAI's GPTBot, which collects training data, made the most, and Claude Code, Anthropic's coding assistant, the second most.
• The search crawlers behind ChatGPT, Perplexity and Claude made only a couple of hundred fetches between them, across thousands of sites. Slackbot fetched the files more often than Perplexity's crawler did.
• About 12% of fetches came from AEO tools (services that track how AI answers mention a site), llms.txt checkers and researchers.

No AI bot in the logs asked for an llms.txt on a site that didn't have one. Ahrefs' explanation:

Ahrefs notes that its customers are more technical than most site owners, so the 28% is an upper bound, and that a fetch does not show anything used what it fetched [4]. It is one company's data for one month, and Ahrefs sells SEO tools.

When is it worth having?

It is worth having on a software documentation site. A developer can give a coding assistant the docs' llms.txt to point it at the right pages, and Mintlify, a documentation host, generates the file and keeps it current automatically [7].

For a local business, a shop or a services site, few tools are likely to ask for it. Yoast, which adds an llms.txt to WordPress sites in one click, says the file helps AI assistants such as ChatGPT, Claude and Gemini understand a site [10]. Yoast gives no data for that, and the logs above show the search crawlers behind ChatGPT and Perplexity rarely request these files [4].

Chrome's Lighthouse, a page-testing tool, includes an llms.txt audit, but a site without the file is marked Not Applicable rather than failed [11]. Lighthouse belongs to Chrome's developer tools, not to Google Search.

If you keep one

Keep it accurate or switch it off: Ahrefs warns that an out-of-date or tampered file misleads every agent that reads it [4].

• List the pages you most want an assistant to read: what you sell, prices, contact details and help pages.
• If a plugin writes it, open yoursite.com/llms.txt once and read what it lists.
• Update it when those pages change. Ours is written by hand, and we edited it four times in its first five days, each time we added or changed a page. This post meant one more line.
• If an agency offers to make one for a fee, ask what it expects the file to do.

Ours stays up because a coding agent or a checker may ask for it and the upkeep is a line per page. Nothing on our site links to it, so going by Ahrefs' logs, few tools will. Our audit does not check for llms.txt, because a missing one costs a site nothing. It does check robots.txt, which decides whether AI crawlers may fetch your pages at all; our guide to AI crawlers shows how to check yours.