Generate an llms.txt file, plus the crawler rules that are actually read.
write the markdown file that tells an assistant what your site is and which pages matter, and get the matching robots.txt rules for GPTBot, ClaudeBot and PerplexityBot in the same pass. free, runs in your browser, nothing is uploaded. the honest caveat about llms.txt adoption is in the writing below, not buried.
add your site name and one page to see the file
add these to the robots.txt you already have rather than replacing it, and keep any existing
Sitemap: line if yours is not at /sitemap.xml. some frameworks
generate /sitemap-index.xml instead.
What an llms.txt file actually is
llms.txt is a proposed convention, not a standard anyone enforces. it is a plain markdown file you put at the root of your domain so it resolves at /llms.txt, and it does one job: it hands an assistant a short, curated map of what your site is and which pages are worth reading. the proposal came out of the AI research community in late 2024 and the format is deliberately tiny, because the whole point is that a model can read the entire thing before deciding what to fetch.
the shape is fixed enough to parse and loose enough to write by hand. one # heading with the site or brand name. one > blockquote that says in a single sentence what this is. an optional line of plain context, which is a good place for the numbers you want repeated. then ## section headings, each with links underneath in the form - [Title](https://example.com/page): what is on it. that is the entire format, and this page writes it for you.
the description after each link is the part people skip and the part that carries the value. - [Pricing](https://example.com/pricing) tells a model nothing it could not have guessed from the URL. - [Pricing](https://example.com/pricing): three plans, billed per seat, free tier capped at two projects is a fact it can repeat back to someone who asked. write the descriptions as if they are the answer, because if this file is ever used, they are.
Nobody has promised to read it
the honest state of it: adoption is not universal and no major AI provider has committed to reading llms.txt. OpenAI, Anthropic and Google have not announced support for it. in April 2025 Google's John Mueller publicly compared llms.txt to the keywords meta tag, which was not a compliment, since that tag was ignored for two decades and then dropped. if someone is selling llms.txt as the way to rank inside ChatGPT, they are guessing and charging for it.
it is still worth the twenty minutes. the file costs nothing to write and nothing to host, and it is a genuinely useful forcing function: choosing the twenty pages that belong in it is the same exercise as deciding what your site is for, and most sites fail that exercise quietly for years. if the convention does get picked up you already have one, and if it does not, you spent an afternoon getting clear about your own site. that is not a bad trade at zero cost.
the robots.txt half below is doing real work regardless of what happens to llms.txt, because crawler directives genuinely are read. GPTBot and OAI-SearchBot for ChatGPT, ClaudeBot for Claude, PerplexityBot for Perplexity: the major ones document that they honour robots.txt, and there have been credible reports of some crawlers ignoring it, so treat it as a strong signal rather than a lock. either way, if your robots.txt still carries a blanket disallow from a staging launch, you are blocking the exact crawlers you have been writing for. more on the surrounding work in the AI search playbook.
The difference between llms.txt, robots.txt and a sitemap
people conflate the three constantly, and they do different jobs. robots.txt is permissions: which crawlers may fetch which paths. it is old, it is honoured by the main crawlers, and it says nothing at all about what your pages mean. it is the only one of the three with any enforcement behind it.
a sitemap is a complete machine list of every URL you want indexed, in XML, usually generated automatically and often thousands of lines long. it exists for coverage, so nothing gets missed. no human reads one and no assistant is going to summarise your business from it.
llms.txt is neither of those. it is a short editorial summary: what this site is, and the handful of pages that answer the questions people actually ask. the value is in the selection. list every page and you have written a worse sitemap, so keep it under about twenty five links, put a real description on each one, and it stays the thing it was meant to be. all three can coexist, and none of them replaces having pages worth citing.
A tidy file is not a strategy. Being the answer is.
if you want ChatGPT and Perplexity naming you when someone asks who to hire, the work sits in what you publish, not in one text file at the root. bring the site to the breakdown call.
Common questions
What is an llms.txt file?
a plain markdown file at the root of a domain, served at /llms.txt, that gives an AI assistant a short map of the site: the name, one sentence on what it is, then a curated list of pages with a description on each. it is a proposed convention rather than an official standard, and it is written for a model to read in full.
Do ChatGPT, Claude and Perplexity actually read it?
none of them has said so. OpenAI, Anthropic and Google have not committed to reading llms.txt, and in April 2025 Google's John Mueller compared it to the old keywords meta tag. treat it as cheap insurance, not a ranking tactic. the robots.txt rules this page also writes are the half that is definitely read.
Where do I put the file once it is generated?
at the root of your domain, so it loads at https://yoursite.com/llms.txt, the same place robots.txt lives. on most static sites that means dropping it in the public folder and deploying. it is served as plain text and there is nothing to submit or register anywhere.
How many pages should I list?
about twenty five is a sensible ceiling and ten is often plenty. the file is a map, not a sitemap, and its only real advantage over your sitemap is that a person chose what went in it. list everything and you have built a worse version of something you already have.
Is this the same as a sitemap?
no. a sitemap is a complete XML list of URLs for search crawlers to index, generated automatically and never read by a person. llms.txt is a short hand-written summary in markdown with a description on every link. keep both, they answer different questions.
Is anything I type here sent anywhere?
no. the file is built in your browser with javascript, there is no account and nothing is uploaded. the download button hands back the text your own browser just assembled from what you typed. you can go offline after the page loads and it still works.