Does Your Website Really Need an LLMS.txt File?

The world of SEO is evolving quickly (not dying as some would suggest), and with the rise of large language models (LLMs) such as ChatGPT, Claude and Gemini, a new question is emerging: should your website have an llms.txt file?

The idea behind llms.txt is simple. Just as robots.txt guides search engine crawlers, llms.txt was proposed as a way to provide AI systems with structured, LLM-friendly information about your website.

But does it actually work in practice, or is it just another well-intentioned idea that never quite lands?

What Is an llms.txt File?

Proposed in September 2024 by Jeremy Howard, llms.txt is essentially a plain-text, markdown-style file that websites can use to make themselves easier for LLMs to understand.

The idea is that instead of crawling heavy web pages, an AI bot could reference llms.txt to find:

• A quick description of your site
• Links to important sections
• Context about what your site covers

This, in theory, would make it easier for these LLMs and therefore encourage them to return to
your website – great in principle. Right?

What Google and Others Say

So far, no major AI company has adopted the format. Google’s Senior Search Analyst & Search Relations Team Lead, John Mueller, even commented on Reddit:

“AFAIK none of the AI services have said they’re using LLMs.TXT (and you can tell when you look at your server logs that they don’t even check for it). To me, it’s comparable to the keywords meta tag – this is what a site-owner claims their site is about … (Is the site really like that? Well, you can check it. At that point, why not just check the site directly?)”

In other words, just because you say what your site is about doesn’t mean LLMs will take your word for it. They already have the ability to crawl and interpret your site directly.

Our Research Into llms.txt

Sure, we could take this analysis at face value, but itʼs always worth digging deeper and testing ourselves. We created our own llms.txt to see how a range of BOTs would respond and interact with it. Following on from John Mueller’s comments I had low expectations, but I didn’t expect as little interactions.

For context here are the parameters of our research:

• We created our llms.txt on the 24th July
• We carried out our research for exactly 1 month
• We analysed 132,000 entries from our server logs

When we dug into our logs to check whether llms.txt was being crawled, the results were pretty telling: not a single AI service or LLM bot has requested it.

Instead, what we did find were the following patterns:

• 4 macOS Chrome hits → though on closer inspection, two were clearly tagged as Zoombot, which is Zoomʼs link preview crawler rather than a real user.
• 4 Windows Chrome hits → all of these turned out to be BW/1.3 bots, masked as Chrome traffic but really just persistent crawlers using fake UAs.
• 1 SEMrush SiteAuditBot → a legitimate SEO crawler that openly declares itself, which is expected.
• 1 Dataprovider.com crawler

What This Means Right Now

So, despite the theoretical promise of llms.txt as a structured, LLM-friendly entry point, in reality, our logs show zero adoption by LLMs. This echos John Muellerʼs comment that AI services simply arenʼt checking for it.

This aligns the practice with the current skepticism in the industry: for now, llms.txt is closer to the keywords meta tag of old SEO, a well-intentioned standard without practical uptake. Until major AI services openly announce support, optimising content for LLMs will need to focus on broader Generative Engine Optimisation (GEO) strategies rather than hoping a single file will drive visibility.

What Can Be Done to Encourage LLMs to Read My Website’s Content?

The quick answer: get in touch with us for a chat about how we can support your GEO strategy. But since youʼre here for practical tips, here are a few key steps you can take yourself:

Use structured data

This will help both search engines and LLMs to understand your content contextually. Adding FAQ, HowTo, Article, and Product schema can make your content easier to surface in AI Overviews.

Improve your internal and external linking

Within SEO getting a great backlink is a huge win an one we rightly shout about. Unfortunately a well optimised internal linking strategy doesnʼt seem to get the same credit. This will help LLMʼs crawl, read and interoperate your content.

Update content regularly

Itʼs no hidden secret that Google uses the ‘freshnessʼ of your content as a ranking factor for traditional SEO. This factor appears to be heightened with the introduction of GEO, as they learn they
want to make sure they are using the most up to date data to train their models. As such, the fresher the content the better chance you have of not only begin used to train the LLMʼs but also to be cited by them.

Maintain a good technical structure

Having come from a Technical SEO background, I will continue to stand on my soapbox and preach the importance of a well-structured website and the huge benefits youʼll see from fixing issues. From the basics of a good heading hierarchy to ensuring clean URL structures, optimised internal linking, and crawlable navigation. Every technical improvement makes your content easier to understand for both search engines and LLMs. The stronger your foundation, the more effectively your site can be indexed, summarised, and recommended.

Frequently Asked Questions

What is llms.txt?

llms.txt is a proposed file format suggested in 2024 by Jeremy Howard. It is similar in concept to robots.txt but designed for AI. The idea is to provide LLMs with a simplified, structured overview of a website, including key information, links, and context, without requiring them to crawl full pages.

Do AI services currently support llms.txt?

As of 2025, no major AI companies (including OpenAI, Anthropic, or Google) have announced support for llms.txt. Server log analysis shows that AI crawlers are not requesting the file.

Should I add an llms.txt file to my website?

Thereʼs no harm in experimenting with llms.txt, but it is not currently recognised by AI systems. For now, investing in broader GEO strategies – such as publishing clear, authoritative, and well-structured content, is a more effective way to improve visibility in AI-driven search results.

Back to blog list