What Is llms.txt? How to Write One and Whether It Helps AI Visibility
What is llms.txt?
An llms.txt file is a plain-text Markdown file hosted at the root of a website (/llms.txt) designed specifically to provide large language models, AI search engines, and autonomous coding agents with a structured, token-efficient directory of a domain's most authoritative content. Originally proposed by Jeremy Howard and the Answer.AI research team in late 2024, the specification standardizes how websites present curated documentation, product specifications, and knowledge bases to machine consumers without forcing AI web scrapers to parse complex HTML document object models, heavy CSS stylesheets, or client-side JavaScript execution trees.
Traditional search engine crawlers like Googlebot were built over three decades to process arbitrary HTML markup, execute JavaScript bundles, and extract hyperlink topologies across deeply nested page hierarchies. While effective for indexation, modern conversational artificial intelligence systems (such as OpenAI ChatGPT Search, Perplexity AI, Anthropic Claude, and Google Gemini) interact with web content through a fundamentally different technical process: Retrieval-Augmented Generation (RAG). In a RAG workflow, an autonomous agent queries external endpoints, converts text passages into high-dimensional vector embeddings, retrieves relevant document chunks, and passes those chunks into the limited context window of a neural language model.
Parsing raw web pages presents severe technical friction for RAG systems. A typical enterprise web page might weigh 3 to 6 megabytes, containing thousands of lines of navigation menus, tracking pixels, cookie consent banners, layout wrappers, and third-party scripts, while the actual authoritative factual content occupies fewer than 500 words. When an AI crawler ingests that raw HTML, up to 95% of its compute and token budget is wasted discarding layout chrome. Furthermore, client-side single-page applications built with modern frontend frameworks frequently suffer from client-side ghosting, where headless scrapers that do not run JavaScript see completely blank pages. The llms.txt standard solves this structural mismatch by providing a direct, machine-readable Markdown roadmap.
Strips layout overhead, navigation headers, and script bundles, reducing token consumption by up to 90% compared to raw HTML scraping.
Provides autonomous coding tools (like Cursor, Claude Code, and Windsurf) with direct markdown links to authoritative API references and guides.
Allows site operators to explicitly declare which pages contain ground-truth product data, pricing, and architecture specifications.
Unlike a traditional XML sitemap that lists thousands of raw URLs without semantic context or qualitative ranking, an llms.txt file acts like an executive briefing document. It delivers a concise summary of the organization, delineates major knowledge categories, and annotates each hyperlink with a descriptive summary that helps an LLM decide whether retrieving that specific document will satisfy the user prompt.
How do you create an llms.txt file?
Creating an llms.txt file requires adhering to a strict, standardized Markdown format defined by the community specification. The file must be formatted in clean UTF-8 text and should follow five distinct structural components:
- Top-Level H1 Heading: The exact name of your organization, project, or software platform. This establishes the primary entity name for the model's knowledge graph.
- Blockquote Summary: A concise 25 to 50 word executive summary placed immediately below the H1 heading. This paragraph must clearly state what the platform does, its target audience, and its core technical capabilities.
- Primary H2 Category Sections: Distinct groupings for your documentation, such as Core Features, API References, Architecture Guides, or Tutorials.
- Annotated Markdown Bullet Links: Every entry must use standard Markdown link syntax (
- [Title](URL): Description) followed by a colon and a standalone explanatory sentence describing the contents of the target URL. - Optional Resources Section: An
## OptionalH2 section placed at the bottom for secondary materials, community forums, or non-essential blog archives that agents can ignore when operating under constrained context budgets.
# ThriveStack citedby Documentation & Architecture Index
> ThriveStack citedby is an answer engine optimization (AEO) and generative engine optimization (GEO) platform that tracks AI visibility across 10 conversational search engines, audits technical crawlability, and attributes live pipeline revenue.
## Core Platform
- [Platform Overview](https://www.thrivestack.ai/platform): Architecture, multi-engine tracking, and attribution pipeline.
- [AI Search Visibility Analytics](https://www.thrivestack.ai/citedby): Real-time Share of Model tracking across ChatGPT, Perplexity, Gemini, Claude, and Copilot.
- [Revenue Attribution Engine](https://www.thrivestack.ai/research/revenue-attribution): Direct correlation between AI citations, trial signups, and closed-won pipeline.
- [Pricing](https://www.thrivestack.ai/pricing): Usage-based pricing tiers without per-seat or per-credit penalties.
## Research & Empirical Benchmarks
- [Best AEO Tools Benchmark 2026](https://www.thrivestack.ai/research/best-aeo-tool): Side-by-side technical evaluation of ThriveStack citedby, Profound, and Peec AI.
- [AI Citation Sources 2026](https://www.thrivestack.ai/research/ai-citation-sources-2026): Empirical data analyzing 680M+ citations and earned media distribution.
- [Server Logs for AI Search](https://www.thrivestack.ai/research/server-logs-ai-search-citation-gaps): Log analysis of 842,000 crawler requests across GPTBot, ClaudeBot, and PerplexityBot.
## Technical Standards & Knowledge
- [AEO vs SEO vs GEO Analysis](https://www.thrivestack.ai/research/aeo-vs-seo-vs-geo): Comprehensive strategic breakdown of the three search optimization disciplines.
- [LLMs-full.txt Specification](https://www.thrivestack.ai/citedby/glossary/what-is-llms-full-txt): Concatenated full documentation corpus for single-shot context windows.
- [Robots.txt for AI](https://www.thrivestack.ai/citedby/glossary/what-is-robots-txt-for-ai): Granular permissions and crawler directives for search retrieval bots vs training scrapers.
## Optional
- [Company Blog](https://www.thrivestack.ai/blog): Research articles, technical guides, and GTM playbooks.
- [AEO & GEO Myths Fact-Checked](https://www.thrivestack.ai/citedby/aeo-geo-myths): 71 common claims tested against 350+ primary research papers.When authoring descriptions for each link, avoid marketing superlatives like "revolutionary" or "market-leading." Large language models prioritize factual statements containing concrete nouns, technical capabilities, integration targets, and clear functional distinctions. For example, writing "- [Pricing](/pricing): Usage-based pricing starting at $35/month without seat limits" gives an AI engine actionable data to answer transactional customer questions accurately.
Where do you publish it?
An llms.txt file must be published at the exact root directory of your primary domain:
Autonomous agents and web crawlers are hardcoded to inspect /llms.txt at the site root, exactly as search bots look for /robots.txt and /sitemap.xml. Hosting the file in a subdirectory (such as /docs/llms.txt or /assets/llms.txt) prevents automated discovery tools from finding it.
HTTP Headers & MIME Type Requirements
Serving the file correctly requires configuring your web server or edge CDN with the appropriate HTTP response headers:
- Content-Type Header: The file should be served as
text/markdown; charset=utf-8or fallback totext/plain; charset=utf-8. Never serve it astext/htmlorapplication/octet-stream, as automated parsers will reject binary or HTML responses. - HTTP Status Code: Must return an unambiguous
200 OKstatus code. Avoid redirects (such as 301 or 302 hops to an external storage bucket or docs subdomain) whenever possible, as several headless fetchers do not follow multi-hop redirects. - Access-Control-Allow-Origin (CORS): Set
Access-Control-Allow-Origin: *so that client-side agent extensions running in developer browsers or IDE panels (like Cursor or Windsurf) can retrieve the file directly without cross-origin blocking errors. - Cache-Control Directives: Set reasonable cache headers (e.g.,
Cache-Control: public, max-age=3600) to ensure fast edge delivery while allowing updates to propagate within an hour.
HTML Head Link Discovery Tag
In addition to hosting the file at the root, you can signal its presence to browser extensions and scrapers by inserting an alternate link tag inside the <head> element of your website templates:
<link rel="alternate" type="text/markdown" title="LLMs.txt" href="/llms.txt" />llms.txt vs llms-full.txt vs robots.txt
Understanding technical answer engine optimization requires distinguishing between three separate files hosted at your domain root. While non-technical observers frequently confuse them, each file serves an entirely different purpose, targets different software actors, and uses different syntax standards.
| Feature / Dimension | llms.txt | llms-full.txt | robots.txt |
|---|---|---|---|
| Primary Purpose | Curated directory and roadmap of key documentation links with summaries | Complete, unabridged documentation corpus concatenated into a single document | Crawler access control, allow/disallow permissions, and sitemap declaration |
| Target Consumer | AI search agents, RAG indexers, developer tool extensions | Large-context LLMs (1M+ tokens), Claude Code, Cursor, offline reasoning tools | All web spiders (Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot) |
| File Format | Markdown (.txt / .md) | Markdown (.txt / .md) | Robots Exclusion Protocol plain text |
| Typical File Size | 2 KB to 25 KB | 200 KB to 5 MB+ | 500 bytes to 10 KB |
| Standard Status | Community proposal (Answer.AI / llmstxt.org) | Community proposal extension | IETF Official Internet Standard (RFC 9309) |
| Deep Dive Guide | Current Page | Read llms-full.txt Guide → | Read robots.txt for AI Guide → |
To understand their interaction, consider their operational sequence during an automated crawl:
- First, an AI crawler inspects robots.txt to check whether it has permission to access the domain. If your
robots.txtblocksUser-agent: GPTBotorUser-agent: ClaudeBot, the bot aborts immediately and will never read yourllms.txtfile. - Next, if permitted, an AI assistant or coding agent looks for
llms.txtat the site root to discover curated technical documentation endpoints without crawling thousands of pagination links. - Finally, if an agent possesses a modern large context window (such as Claude 3.5 Sonnet or Gemini 1.5 Pro) and requires the entire code documentation for end-to-end reasoning, it retrieves llms-full.txt in a single HTTP request, bypassing multi-hop retrieval entirely.
Does llms.txt help AI search visibility?
No major AI engine has confirmed using llms.txt for live answer generation
As of 2026, neither OpenAI (ChatGPT Search), Google (AI Overviews, Gemini), Perplexity AI, Anthropic (Claude), nor Microsoft (Copilot) has officially confirmed that /llms.txt is utilized as an indexation signal, ranking factor, or primary retrieval target for answering consumer search queries.
In our comprehensive empirical research evaluating 71 claims across 350+ primary research papers, we fact-checked the claim that deploying an llms.txt file immediately boosts answer citations. You can explore the full technical teardown in our fact-check analysis: Does llms.txt Help AI Search Visibility & AI Crawler Discovery?
Server log analysis of over 842,000 crawler requests across enterprise SaaS domains reveals why: commercial answer engines rely on live web retrieval pipelines (RAG) that query established search indexes (such as Google, Bing, and internal web caches). When a buyer asks ChatGPT "What is the best revenue attribution tool for B2B SaaS?", ChatGPT does not send a bot to scan /llms.txt across fifty candidate domains in real time. Instead, it queries its search index, retrieves ranking web pages and third-party validation sources (like G2, Reddit, and trade journals), extracts semantic text snippets, and synthesizes an answer within 800 milliseconds.
Furthermore, server access logs demonstrate that GPTBot, ClaudeBot, and PerplexityBot request /llms.txt on fewer than 0.8% of crawled domains. The vast majority of bot requests target HTML pages, XML sitemaps, and standard REST/GraphQL API endpoints. Believing that publishing an /llms.txt file will automatically get your company cited in ChatGPT answers is a misconception that confuses developer tooling conventions with search retrieval algorithms.
Where llms.txt Actually Delivers Value Today
While llms.txt does not directly manipulate consumer search rankings, it provides tangible utility in three specific technical workflows:
1. Developer IDE Coding Assistants
AI code generation tools (such as Cursor, Windsurf, GitHub Copilot, and Claude Code) actively ingest /llms.txt when users point them at documentation URLs. If your developer platform provides clean markdown links, engineers building with your SDK experience fewer syntax errors and integration failures.
2. Custom Enterprise RAG Pipelines & Agents
Internal research agents and buyer procurement teams building custom LangChain or LlamaIndex workflows frequently check for /llms.txt first to assemble competitor evaluation dossiers without writing custom web scrapers.
3. Future-Proofing for Standardized AI Web Protocols
Because maintaining an llms.txt file takes minimal engineering effort (often under 30 minutes to create and automated via build scripts), deploying one represents a zero-downside insurance policy against future search index changes.
Best Practices & Common Pitfalls
Teams that decide to deploy an llms.txt file must maintain it with the same engineering rigor applied to XML sitemaps and API schemas. A broken or outdated file causes more harm than having no file at all, as language models that encounter 404 links or hallucinated parameters will drop the domain from their reasoning context. Follow these four operational standards:
Avoid Token Bloat: Keep Total Size Under 25 KB
Do not dump every blog post, tag page, or changelog snippet into llms.txt. The primary file should remain an annotated table of contents containing 20 to 60 essential links. If your total file exceeds 50 KB, autonomous agents with strict retrieval budgets may truncate the payload before reaching your critical product endpoints.
Automate Generation in CI/CD Deploy Pipelines
Manual files rot quickly as documentation URLs change and product features evolve. Hook your llms.txt generation into your build workflow (e.g. GitHub Actions, Next.js static generation, or Vite build scripts) so the file updates automatically whenever documentation is merged to production.
Ensure Linked Endpoints Return Clean Markdown
When an agent follows a link inside your llms.txt, it expects to receive clean, readable text. If the target URL requires client-side JavaScript execution to render its content, the agent will encounter client-side ghosting. Support content negotiation by serving raw Markdown when scrapers pass Accept: text/markdown headers.
Verify Robots.txt Consistency
Ensure that URLs listed in your llms.txt are not disallowed in your robots.txt. Crawlers that verify permissions before following links will flag contradictory directives as an error condition.
Frequently Asked Questions About llms.txt
Is llms.txt an official web standard approved by the W3C or IETF?
No. Unlike robots.txt (which is governed by the IETF under RFC 9309) and XML sitemaps (standardized by major search engines), llms.txt is a community proposal initiated by Answer.AI. While dozens of prominent AI developer companies and documentation frameworks have adopted it voluntarily, it carries no formal standards body certification.
Can I block AI bots from training on my content using llms.txt?
No. The llms.txt file is strictly an information discovery mechanism, not an access control protocol. To prevent AI crawlers from scraping your site for model training, you must configure disallow directives in your robots.txt file or deploy edge Web Application Firewall (WAF) challenge rules.
Should I create both llms.txt and llms-full.txt?
Yes, if you operate a technical product, API platform, or documentation-heavy SaaS. Provide /llms.txt as the lightweight table of contents for quick discovery, and compile /llms-full.txt as the comprehensive single-document reference for frontier models capable of digesting 1M+ token context payloads.
What is the fastest way to get cited in ChatGPT and Perplexity today?
Focus on earning third-party authority signals, structured JSON-LD entity schema, and comprehensive coverage across Wikipedia, Reddit, and high-domain-authority industry publications. In empirical 2026 citation research, 84% of citations pull from earned media rather than company websites. Learn more in our AI Citation Sources 2026 Report.
llmstxt.org Specification
The community standard defining Markdown proposal structures for LLM web ingestion.
Articles & Research Referencing LLMs.txt
Get cited across ChatGPT, Perplexity & Gemini with citedby
Optimize your brand’s AI visibility score, track Share of Model across buyer prompts, and turn zero-click search into your highest-converting pipeline source.
Explore ThriveStack citedby