--- title: "llms.txt: what it is, who actually reads it, and what four months of logs from a site with 4 million bot visits show" description: "llms.txt is the file that is supposed to give AI a map of a website. On Quotidiano Motori, the Italian online motoring daily we run, AI bots opened it 131 times in four months, while in one week they requested the Markdown versions of the pages 29,561 times. Google writes that new files are not needed." url: https://www.thinkai.it/en/blog/llms-txt-what-it-is-who-reads-it/ date: 2026-10-07 modified: 2026-10-13 author: "Caterina Di Iorgi" image: https://www.thinkai.it/wp-content/uploads/2026/10/Flusso-digitale-verso-LLMstxt.jpg categories: ["English"] type: post lang: it --- # llms.txt: what it is, who actually reads it, and what four months of logs from a site with 4 million bot visits show In brief - llms.txt is a file proposed in September 2024 to give AI systems a Markdown map of a website. Google writes that its AI features need no new files. - On Quotidiano Motori, the Italian online motoring daily we run, AI and search-engine bots made 4 million requests in four months. They opened the llms.txt file 131 times, about once a day. In one week the file was requested 491 times, almost always by SEO tools and by llms.txt directories. - In the same week, the Markdown versions of the pages were requested 29,561 times, sixty times more, 42% of them by ChatGPT’s search bot. AI reads the pages, not the map. When a company asks how to get read by AI, the first piece of advice it gets is almost always the same: **add an llms.txt file**. It takes ten minutes, and plugins generate it. The question nobody asks is whether anyone then reads it. We looked at the logs of [Quotidiano Motori](https://www.quotidianomotori.com/), the Italian online motoring daily we run, where search-engine and AI bots make about **a million requests a month**: the file has been there since 2026, along with a Markdown version of every page. The numbers say who reads what. Index - [What llms.txt is](https://www.thinkai.it/en/blog/llms-txt-what-it-is-who-reads-it/#What_llmstxt_is) - [The bots we count, and the ones we keep out](https://www.thinkai.it/en/blog/llms-txt-what-it-is-who-reads-it/#The_bots_we_count_and_the_ones_we_keep_out) - [Who reads llms.txt](https://www.thinkai.it/en/blog/llms-txt-what-it-is-who-reads-it/#Who_reads_llmstxt) - [What they read instead: the pages, in Markdown](https://www.thinkai.it/en/blog/llms-txt-what-it-is-who-reads-it/#What_they_read_instead_the_pages_in_Markdown) - [What to do, in order of weight](https://www.thinkai.it/en/blog/llms-txt-what-it-is-who-reads-it/#What_to_do_in_order_of_weight) - [Frequently asked questions](https://www.thinkai.it/en/blog/llms-txt-what-it-is-who-reads-it/#Frequently_asked_questions) ## What llms.txt is It is a text file in Markdown format, placed at the root of the site like robots.txt and sitemap.xml. It was [proposed by Jeremy Howard on September 3, 2024](https://llmstxt.org/): it holds the site’s name, a one-line summary, and a list of links to the pages that matter, with a note for each. The idea is to give an **AI agent** a short map that fits in its working memory, and to leave the details behind the links. The same proposal asks for a second, less-quoted thing: a **Markdown version of every page**, at the same address with the .md extension, linked from the HTML page. The file was born for software documentation, where coding agents use it to find a library’s instructions. The updated version of the proposal, from August 2026, says that OpenAI, Anthropic, and Google publish it for the **documentation of their own tools**. That is a fact to read carefully: publishing the file for developers does not mean their bots look for it on other people’s sites. ## The bots we count, and the ones we keep out The log we use counts only bots recognized as **search engines and AI systems**: Google, Bing, Apple, Amazon, Meta, OpenAI, Anthropic, Perplexity, and the others. These are the bots we want to let in. The rest, image harvesters, content copiers, scanners hunting for configuration files, are kept out by rules set at three levels: robots.txt, the server, and Cloudflare, and they are not in the count. Between June 6 and October 6, 2026, the admitted bots made **4,005,693 requests**: Bing 1.29 million, Google 846,000, Apple 512,000, Amazon 426,000, Meta 235,000, Claude’s search bot 211,000, ChatGPT-User 211,000, ChatGPT’s search bot 88,000, Perplexity 39,000. Over the same period, AI systems sent the site **1,443 human visits**, 1,072 of them from ChatGPT. The ratio of bot reads to human visits is nearly three thousand to one: that is the context for everything that follows. ## Who reads llms.txt In four months, AI and search-engine bots requested llms.txt and its extended version **131 times**, about once a day. The requests came from Google (about 35), Amazon (about 30), Meta (14), Apple (11), ChatGPT-User (9), OpenAI’s SearchBot (5); one each from ClaudeBot, Claude-SearchBot, and Bing. Out of 4 million requests, the file accounts for **0.003%**. Widening the view to all visitors, human or not, the file was requested **491 times** in the past seven days. The top two entries are an SEO tool (171 requests) and a public directory of llms.txt files (112); then come validators, research scanners, a service that detects the technologies a site uses, and manual requests made with curl. The AI bots that week: Meta 12, Amazon 6, Google 4; **OpenAI, Anthropic, and Perplexity: zero**. Today, those who read llms.txt are mostly the tools that sell llms.txt. Google says so in writing. In its documentation for site owners, under the entry on AI features in Search, it writes that [there is no need to create new files](https://developers.google.com/search/docs/appearance/ai-features) readable by machines, text files, or AI markup: the usual requirements apply, indexable content, crawling allowed, readable text. ## What they read instead: the pages, in Markdown The figure that changes the picture lies in the other half of the proposal. Every article on Quotidiano Motori declares its own **Markdown version** in its code, with the *rel=”alternate” type=”text/markdown”* link the proposal recommends. In the same seven days those versions were requested **29,561 times**, about 4,200 a day, and served correctly in 98% of cases. ChatGPT’s search bot alone requested 12,370 of them, **one every fifty seconds**; then Bing 5,808, Amazon 3,922, Baidu 1,581, Apple 1,423, Meta 1,027, Claude’s search bot 129, ChatGPT-User 73, Perplexity 8. **Sixty times** the requests for llms.txt, and from different bots: the ones that ignore the map happily read the clean pages. The reason lies in how search bots work: they land on a page because someone asked a question, find the link to the plain-text version, and take it, free of menus, ads, and scripts. **llms.txt is an index; the page is the content**. What they then do with that text, whether they cite it or discard it, the log does not say: it only says what they asked for. ## What to do, in order of weight First: **the pages**. AI bots arrive almost only there, and Google asks for only that: readable text, crawling allowed, no accidental blocks at the hosting or CDN level. It is the same work that decides whether a brand appears in the answers, described in the [rules of Generative Engine Optimization](https://www.thinkai.it/en/blog/what-is-geo-generative-engine-optimization/). Second: **the Markdown version** of every page, linked from the HTML. In our numbers it is the only technical addition that bots actually use, and it costs a plugin or a server rule. Third: **llms.txt**. It costs little, does no harm, and some agents look for it; publish it knowing that today it is read by tools more than by AI. Fourth, and before all the others: **knowing who comes in**. Without a bot log you work by guesswork, and on the sites AI reads the most [more than half of traffic is not human](https://www.thinkai.it/en/blog/ai-agents-web-traffic-not-human/). What AI then says about the brand, having read those pages, is [the measurement an audit starts from](https://www.thinkai.it/en/light-audit/). ## Frequently asked questions **What is the llms.txt file?** It is a Markdown text file at the root of a website, with the site’s name, a one-line summary, and a list of links to the important pages. Jeremy Howard proposed it in September 2024 to give AI agents a short map of a site. The same proposal asks for a Markdown version of every page. **Does Google use llms.txt?** No. In its documentation for site owners, Google writes that appearing in its AI features does not require creating new machine-readable files, text files, or AI markup. The usual requirements apply: an indexable page, crawling allowed, content as text. **Is it worth publishing llms.txt?** Yes, because it costs little and does no harm, but without expecting results. On Quotidiano Motori, AI bots opened it 131 times in four months, while in a single week they requested the Markdown versions of the pages 29,561 times. The work that counts is on the pages.