TL;DR

llms.txt is a proposal by Jeremy Howard, published in September 2024, for a Markdown file at the root of a website that gives AI agents a concise summary of the site and links to clean, LLM-readable pages. It is a reasonable idea and cheap to implement, and Google's Lighthouse tool now checks for it, but there is no public evidence that it improves visibility in AI search: Google says no special AI text files are needed for AI Overviews, and Google's John Mueller has called its effect speculative. For brands, the bigger wins are the basics: let AI crawlers in, publish clear entity and product pages, and add structured data that matches what is on the page.

What llms.txt is

Jeremy Howard proposed llms.txt on 3 September 2024 at llmstxt.org. The problem he set out to solve is practical: websites are built for human browsers, full of navigation, scripts and layout, while language models work best with concise text and have limited context windows. His proposal is a single Markdown file at /llms.txt that tells an AI agent, in a few lines, what the site is and where the most useful information lives.

The format is deliberately simple:

  • An H1 heading with the name of the project or site. This is the only required section.
  • A short blockquote summarising the project.
  • Optional paragraphs with further detail.
  • H2 sections containing "file lists": Markdown links to key pages, each with an optional note.
  • A section called "Optional" for secondary links that an agent can skip if it needs less context.

The proposal also suggests offering clean Markdown versions of pages at the same address with .md appended. Howard positions the file as complementary to existing standards: robots.txt tells automated tools what access is acceptable, sitemap.xml lists indexable pages for search engines, and llms.txt offers curated context for an agent working on a user's behalf.

"mainly useful for inference rather than training" — Jeremy Howard, llmstxt.org

That distinction matters. llms.txt is designed for the moment an assistant or coding tool is actively looking something up, not as a way to influence what goes into a model's training data.

llms.txt

a proposed Markdown file at a website's root that summarises the site and links to LLM-friendly versions of its key content.

What llms.txt is not: adoption and limits

The current llmstxt.org page states that thousands of sites now publish the file, that documentation platforms generate it automatically and that several AI labs publish one for their developer documentation. In May 2026, Google's Lighthouse auditing tool added an llms.txt check in its agentic browsing audits, as reported by PPC Land; it flags server errors and treats a missing file as not applicable, because the file is optional.

None of this shows that llms.txt helps a brand appear in AI answers, and the evidence points the other way for search:

  • Google's documentation on AI features and your website says site owners do not need to create new machine-readable files, AI text files or special markup to appear in AI Overviews or AI Mode.
  • Google's John Mueller, responding on Reddit in 2026, said no AI platform has required the file, and earlier noted that AI services had not said they use it. He compared it to the keywords meta tag: a site owner's claim about content that can be checked directly.
  • In September 2025, Search Engine Journal concluded that no AI platform was using it, and pointed out a trust problem: a separate file could say something different from the page users see.
  • A ProGEO.ai study cited by PPC Land found that in March 2026, 7.4% of Fortune 500 companies had an llms.txt file, compared with 92.8% for robots.txt and 53.8% using JSON-LD structured data.

"I don't think anyone knows – it's purely speculative for now." — John Mueller, Google

The fair conclusion: llms.txt is a low-cost, low-risk addition that may help developer-focused sites and AI agents navigating documentation. It is not a ranking lever, and anyone selling it as the key to AI visibility is overclaiming.

How to make a brand genuinely machine-readable

What AI systems can reliably use today is the same content humans see: crawlable HTML, clear facts and consistent entity information. In our work with European SMEs, the gaps we see most often are basic: unclear company descriptions, inconsistent names across markets, and key facts locked in images or PDFs. A practical checklist:

  1. Do not block the crawlers you want. Mueller describes this as the most basic step. Review robots.txt, CDN and firewall rules for search and AI crawlers. Aleyda Solis's AI Search Optimization Checklist notes that Google-Extended does not govern AI Overviews eligibility, which relies on Googlebot.
  2. Serve content as real text. Make sure important content is in server-rendered or reliably rendered HTML, not only in images, PDFs or scripts.
  3. Build a clear entity page. One "About" page that states who you are, what you do, where you operate, your legal name, registration and VAT details, leadership and contact information, in plain sentences.
  4. Add Organization structured data. Google's Organization documentation recommends placing it on the home page or a page describing the organisation, with properties such as name, url, logo, sameAs links to your official profiles, address, contactPoint, legalName and vatID. Google describes the VAT code as an important trust signal for users.
  5. Keep markup honest. Google's AI features guidance says structured data should match the visible text on the page. Never mark up claims the page does not make.
  6. Make products and services explicit. Give each offer its own page with name, description, specifications, price or pricing model, availability and markets served. Keep Merchant Center and Business Profile data current.
  7. Stay consistent everywhere. Use the same company name, description and key facts across your site, LinkedIn, directories and marketplaces, in every language you operate in.
  8. Then, optionally, add llms.txt. If you have documentation, an API or a large knowledge base, a well-curated llms.txt is a sensible extra. Keep it consistent with your pages.
Key takeaways
  • llms.txt, proposed by Jeremy Howard in 2024, is a Markdown summary file intended for AI agents at inference time.
  • There is no public evidence that it improves AI search visibility; Google says no AI text files are needed for AI Overviews.
  • Lighthouse now checks for it, but a missing file is treated as not applicable.
  • Real machine-readability comes from crawl access, text-based content, clear entity pages and honest structured data.
  • Add llms.txt as a low-cost extra, not as a strategy.

FAQ

Should we add an llms.txt file to our website?
It does little harm and takes little time, so it is reasonable if you have documentation or a large content library. Just do not expect it to change your visibility in Google or AI assistants; prioritise crawl access, clear content and structured data first.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers what they may access. llms.txt is a proposed, optional summary of your site with links to key content, intended to help AI agents understand it. It does not control access.
What structured data matters most for a brand?
Start with Organization markup on your home or about page, including name, url, logo, sameAs links to official profiles, address, contact details and legal identifiers such as your VAT number. Then add markup for products, services or articles where relevant, always matching visible content.

Sources

Make your brand easy for AI to understand

Talk to JRK about the entity pages, structured data and technical access that help AI systems describe your business accurately.

Book a free growth session →

Want this working for your brand?

Book a free 30-minute growth session. We'll look at your funnel and tell you what we'd do first — honestly.

Book a growth session →