AI crawlers & llms.txt: staying visible in AI answers

AI search changes how your website gets found

ChatGPT, Claude, Perplexity and Google’s AI Overviews answer questions directly — citing websites as sources. You can directly influence two technical prerequisites: are AI crawlers allowed to read your pages at all, and is the content technically accessible to them? Whether a page is actually selected or cited also depends on each provider’s selection and ranking systems. This page explains the technical controls — including the young llms.txt convention.

What is llms.txt?

llms.txt is a Markdown file at the root of your website (/llms.txt), analogous to robots.txt or sitemap.xml. It contains a curated table of contents for AI systems: your site’s name as a heading, a short description as a blockquote, followed by link lists to your most important pages — each with one sentence on what it covers.

The convention was proposed by Answer.AI in September 2024 and is deliberately simple: curate instead of dumping everything. A good llms.txt covers the 10 to 30 pages that genuinely explain your offering — not your entire sitemap.

An honest assessment: llms.txt is a young community convention, not an official standard. Google has publicly stated that it does not read the file, and the major AI providers have not committed to using it. Documented use today is concentrated in developer tools such as AI code editors and in documentation platforms. Our recommendation therefore: a curated file is relatively straightforward to provide and makes content easier for those tools to access — but there is currently no proven effect on AI search results. The more direct technical lever is bot access via robots.txt (below).

live24h applies the convention itself: www.live24h.eu/llms.txt .

What are AI crawlers?

AI crawlers are automated visitors that fetch web pages for AI systems. What matters is their purpose — it determines what a block in robots.txt means for you:

• Search: builds the index for AI search answers and citations. A block prevents that search crawler from collecting the page for this purpose; allowing it does not guarantee inclusion or citation.
• Training: collects content for training future models. Blocking a dedicated training bot does not automatically block the same provider’s separate search bot.
• Live fetch: retrieves a page when a user specifically asks about it. According to providers, robots.txt rules may not apply, or may apply only in part, to some of these requests.

The most important bots at a glance:

The table describes the documented purposes of these user-agent tokens. Whether a provider follows robots.txt in every case is a separate question; live fetches in particular may behave differently.

Why robots.txt is the more direct technical lever

For bots that follow it, robots.txt works as an opt-out: bots that are not restricted there may crawl. For your AI visibility, that means:

• Blocking search bots restricts discoverability. Blocking OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot or Bingbot prevents collection through that search crawler. Conversely, allowing it does not guarantee inclusion or citation.
• Blocking dedicated training bots is a legitimate trade-off. Blocking GPTBot, ClaudeBot or opting out via Google-Extended does not automatically block the separate search crawlers — here you decide deliberately about use for training or grounding.
• A common configuration risk is a copy-pasted block list that unintentionally excludes search bots, or a blanket Disallow: /. Either can restrict discoverability in conventional and AI-assisted search.

So check specifically which bots your robots.txt affects — and whether that matches your intent.

How live24h helps

The SEO check covers technical AI-access signals. live24h monitors robots.txt, meta robots and canonical URLs on your most important pages — and also looks at the AI perspective: is an llms.txt present and plausibly structured? Does your robots.txt block AI bots, and if so: search bots (a discoverability risk) or training bots (a deliberate choice)? The results appear as advisory findings with context, without degrading your check status. The SEO checks guide shows how to set them up.

The WordPress plugin creates your llms.txt. The live24h plugin for WordPress generates a curated llms.txt straight from your site information — title, description, most important pages — and respects existing files instead of overwriting them. Setup is described in the WordPress plugin guide .

That keeps the decision with you: enable technical discoverability, control training use deliberately and review both regularly.