{"pageUrl":"https://promagen.com/what-is-llms-txt","lastModified":"2026-05-10","provenanceHash":"sha256:8f416dae0f37a50d7db55f73756a150d5c0ea344ef83d205729bd9723a71bc79","provenanceNote":"llms.txt spec is published at https://llmstxt.org/. As of 2026 no major AI engine publicly documents external llms.txt ingestion as a guaranteed crawl or retrieval signal. Anthropic publishes a documentation-index llms.txt for its own developer docs (claude.com/docs/llms.txt) but its crawler documentation does not commit to external llms.txt as a privileged input. Treat all vendor-support claims for external llms.txt ingestion as emerging or unconfirmed unless official vendor docs explicitly say otherwise.","claims":[{"id":"claim-llms-txt-purpose","statement":"llms.txt is a markdown file at site root that provides AI engines with a curated, human-readable summary of the site, distinct from robots.txt (access control) and sitemap.xml (URL inventory).","evidenceUrl":"https://llmstxt.org/","lastVerified":"2026-05-10","hash":"sha256:c163dd4753639c258bd981eed3e9d79c3acb6b14594bfdcdd288db7283740111"},{"id":"claim-anthropic-publishes-llms-txt","statement":"Anthropic publishes a documentation-index llms.txt file at claude.com/docs/llms.txt for its own developer documentation. This is a public adoption of the file format for first-party docs; it is distinct from a commitment that Claude/ClaudeBot ingests external sites' llms.txt files as a privileged retrieval signal.","evidenceUrl":"https://claude.com/docs/llms.txt","lastVerified":"2026-05-10","hash":"sha256:d774ca7a5c3694f8fdd9ea34ccfa01d348e8141cec82d0edf2b384ffcce40ded"},{"id":"claim-vendor-support-unconfirmed","statement":"Official Anthropic crawler documentation describes ClaudeBot, Claude-User, and Claude-SearchBot using robots.txt as the access-control signal and does not document llms.txt as a guaranteed external crawl or retrieval input. Treat vendor support for external llms.txt ingestion at the major AI engines (OpenAI, Anthropic, Perplexity, Google) as emerging or unconfirmed unless official vendor docs explicitly say otherwise.","evidenceUrl":"https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler","lastVerified":"2026-05-10","hash":"sha256:906253373da7ae658f285b1ce575fc66d7a41eef553c96b9845ded180f32bfc2"},{"id":"claim-three-artefacts-coexist","statement":"robots.txt, sitemap.xml, and llms.txt serve different layers and do not replace each other. A site that takes AI visibility seriously publishes all three.","evidenceUrl":"https://promagen.com/sentinel/weekly","lastVerified":"2026-05-10","hash":"sha256:8170a83537f47b8deefaa1b6e64ab4f24be7980318b2992642a9d9926b4c01e9"}]}