# llms.txt Explained: The New Standard for AI Crawlers What llms.txt is, why it matters for AI search, and how to add one to your startup's site, with a copy-ready structure and common mistakes to avoid. Date: 2026-08-07 | Author: Kelly | 7 min read --- # llms.txt Explained: The New Standard for AI Crawlers `llms.txt` is a plain-text Markdown file you place at the root of your domain (`yoursite.com/llms.txt`) that gives AI systems a clean, curated map of your most important content. Think of it as a table of contents written for large language models: instead of forcing them to crawl and parse your entire JavaScript-heavy site, you hand them a tidy, high-signal summary of who you are and which pages matter. It's an emerging convention, not yet an enforced standard, but adopting it early is a cheap, low-risk way to make your startup more legible to AI search. Here's what it is, how it differs from `robots.txt`, and exactly how to add one. --- ## What Problem Does llms.txt Solve? Modern sites are built for browsers, not language models. Navigation, cookie banners, client-side rendering, and ad markup create noise that wastes an LLM's limited context window and buries your actual content. `llms.txt` solves this by offering a **clean, Markdown-native entry point**: a short description of your site plus curated links to the content you most want cited. There are two common files: - **`llms.txt`**, a concise index: a one-paragraph description of your site and a curated, categorized list of links to key pages. - **`llms-full.txt`**, the expanded version: the actual full text of your most important pages concatenated into one document, so a model can ingest everything in a single fetch. --- ## llms.txt vs robots.txt vs sitemap.xml These three files are complementary, not interchangeable: | File | Audience | Purpose | |---|---|---| | `robots.txt` | All crawlers | Permission, what may or may not be crawled | | `sitemap.xml` | Search crawlers | Discovery, a machine list of every URL | | `llms.txt` | AI / LLMs | Curation, a human-readable map of your *best* content and what your site is | `robots.txt` controls access, `sitemap.xml` lists everything, and `llms.txt` highlights what matters most in a format models can read directly. You want all three. --- ## The Structure of a Good llms.txt The convention is simple Markdown. A solid `llms.txt` looks like this: ``` # Your Company Name > One-sentence description of what you do and for whom. A short paragraph expanding on your positioning, your ICP, and how to contact you. ## Core Pages - [Services](https://yoursite.com/md/services.md): What you offer. - [Pricing](https://yoursite.com/md/pricing.md): How you charge. ## Guides - [Guide title](https://yoursite.com/md/blog/slug.md): One-line summary. ## Frequently Asked Questions ### The question? The plain-text answer. ``` A few principles that make it effective: - **Link to clean Markdown mirrors** (e.g. `/md/...`) rather than full HTML pages when you can, less noise for the model - **Curate, don't dump**, your 20 best pages beat all 500 - **Write the description in plain, factual language**, this is often the first thing a model reads about you - **Include an FAQ block**, question-and-answer pairs are highly extractable *Common mistake: auto-generating `llms.txt` from your entire sitemap. That recreates the noise problem you were trying to solve. Curate to high-value pages.* --- ## How to Add llms.txt to Your Site 1. **Draft the content** using the structure above. Keep the top description tight and factual. 2. **Curate your link list**, services, pricing, your best 10–20 guides, and an FAQ. 3. **Generate clean Markdown mirrors** of those pages if you can (many frameworks can emit a `.md` version of each page at build time). Point your links at those. 4. **Place the file at the root** so it resolves at `yoursite.com/llms.txt`. Serve it as `text/plain; charset=utf-8`. 5. **(Optional) Add `llms-full.txt`** with the full text of your priority pages concatenated. 6. **Keep it fresh**, regenerate whenever you publish. A stale index misrepresents you. On this very site, `llms.txt` and `llms-full.txt` are regenerated automatically on every deploy from our live content, so the map an AI reads is never out of date. That "auto-maintained" part matters more than the initial setup. --- ## Does llms.txt Actually Work Yet? Honest answer: adoption is early and no major engine has publicly confirmed it as a ranking input. So why bother now? - **It's near-zero cost** to add and maintain - **It future-proofs** you as the convention gains traction - **It forces useful hygiene**, clean Markdown mirrors and a curated content map help every AI crawler, llms.txt or not - **The downside is nil**, a small static file that only helps machines that choose to read it Treat it like structured data in its early days: not yet mandatory, clearly directional, and cheap insurance. --- ## Common Questions ### Where does the llms.txt file go? At the root of your domain, reachable at `https://yoursite.com/llms.txt`. Serve it as plain text (`text/plain; charset=utf-8`). ### What's the difference between llms.txt and llms-full.txt? `llms.txt` is a curated index (description + links). `llms-full.txt` contains the actual full text of your priority pages in one file, so a model can ingest everything in a single request. ### Will llms.txt hurt my normal SEO? No. It's an additive, opt-in file for AI systems. It doesn't affect Googlebot's ranking of your HTML pages, and it doesn't replace `robots.txt` or `sitemap.xml`. ### How often should I update llms.txt? Every time you publish or materially change key pages. The best setups regenerate it automatically at build/deploy so it's always current. --- ## Key Takeaways - `llms.txt` is a **curated, Markdown map** of your best content for AI systems - It **complements** `robots.txt` (access) and `sitemap.xml` (discovery), keep all three - **Curate, don't dump**; link to clean Markdown mirrors; include an FAQ block - Add `llms-full.txt` for a single-fetch full-text version - **Automate regeneration** so the file never goes stale - It's cheap, low-risk, future-facing hygiene, part of a broader effort to [get cited by ChatGPT and Perplexity](/blog/ai-search-optimization-startups-chatgpt-perplexity) --- Want an `llms.txt` that's generated and kept current automatically, alongside the rest of your AI-search setup? [Run Kelly's free SEO audit](/ai-growth-team) to see what's missing today, then let Kelly maintain it for you every week.