Does llms.txt matter? An honest look for SEOs and site owners

5 min read

Short answer: it's cheap to add and harmless, but it isn't what decides whether ChatGPT, Perplexity, Claude or Google's AI features cite you. Crawler access, server-rendered content and clear pages matter far more. Add llms.txt after those are right, not instead of them.

What llms.txt is

llms.txt is a proposed convention (published in 2024, not a formal standard) for a Markdown file at /llms.txt that gives language models a concise map of your site: a title, a one-paragraph summary, and lists of links to the pages that matter most, optionally with Markdown versions of those pages.

# Acme

> Acme makes invoicing software for freelancers in the EU.

## Docs

- [Getting started](https://acme.example/docs/start): set up in 5 minutes
- [API reference](https://acme.example/docs/api)

## Optional

- [Blog](https://acme.example/blog/)

The idea is sensible: HTML pages are full of navigation, scripts and layout that a model has to wade through, while a short Markdown index says "here's what we are and where the good stuff is".

What it isn't

• Not an access control. It doesn't allow or block anything. That's robots.txt (and your CDN). See AI crawler user agents & robots.txt templates.
• Not a ranking signal in Google. Google representatives have said Search doesn't use llms.txt, and AI Overviews draw on the normal Google index.
• Not a sitemap replacement. Search crawlers, including the ones AI search engines use, still discover pages through links and XML sitemaps.

Do AI search engines use it?

As of this writing, none of the major AI search products (ChatGPT search, Perplexity, Claude, Google AI Overviews, Copilot) has publicly documented using llms.txt to choose what to crawl or cite. Server logs that site owners have shared show only occasional fetches of the file. That may change, and some developer tools and coding assistants do read llms.txt (or the fuller llms-full.txt) when pointed at a documentation site. That's where it's most useful today.

So should you add one?

It depends on your site:

• Developer docs, APIs, SDKs: yes. Coding assistants and agents genuinely benefit from a clean index and Markdown versions of pages.
• SaaS, ecommerce, publishers: optional. It takes 20 minutes, costs nothing to serve, and positions you if adoption grows. Just don't prioritise it above the basics.

What actually decides AI visibility

1. The AI search crawlers are allowed in robots.txt (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot…), separately from training crawlers.
2. Your CDN or firewall doesn't block them anyway.
3. The content is in the server-rendered HTML. Most AI crawlers don't run JavaScript.
4. The page is indexable (no noindex) and Googlebot/Bingbot can reach it. Google's AI features and Copilot depend on those indexes.
5. Pages are easy to quote: a clear H1, question-style headings with direct answers, structured data, dates.

The AI visibility checker tests all of these for a URL (and checks for llms.txt too), with copy-paste fixes. Our technical GEO checklist goes deeper.

If you add one: a few tips

• Serve it as text/plain or text/markdown at the site root, with a 200 status (not a redirect to an HTML page).
• Keep it short: the summary should say what you are and who it's for in one or two sentences.
• Link to canonical, indexable URLs only, and keep it updated when pages move. A stale llms.txt is worse than none.
• Don't put anything in it you wouldn't publish on the site.

You can see ours at crawlsentry.com/llms.txt.