What Is ai.txt?

ai.txt is a proposed plain-text file, served from a site's root like /ai.txt, that declares how AI systems may use the site's content — separating permissions for text, images, and other media, and distinguishing AI training from other uses. The best-known proposal came from Spawning in 2023, positioned as an opt-out layer for generative AI training that robots.txt was never designed to express.

What problem does ai.txt try to solve?

Robots.txt, standardized as RFC 9309 in 2022, answers one question: which URLs may a named crawler fetch. It cannot express purpose — "crawl for search, but not for training" — except crudely, by blocking purpose-specific user agents (GPTBot for training, Google-Extended for Gemini grounding). ai.txt proposed a media-type and usage-based grammar instead: allow text for indexing, deny images for generative training, and so on, in one declarative file.

How does ai.txt compare to its neighbors?

The table exposes the split personality of AI-crawler governance: ai.txt and noai are restrictive signals, llms.txt is an invitational one, and only robots.txt has both standardization and public compliance commitments from OpenAI, Google, and Anthropic.

What should site owners actually do?

Treat ai.txt as optional signaling, not protection. The enforceable-in-practice stack today is: purpose-specific robots.txt rules (the lever engines document), CDN-level bot management for crawlers that ignore declarations, and emerging paid-access schemes like pay-per-crawl where monetization matters. Sites pursuing AI visibility rather than restriction should invest the same effort in the opposite direction — clean crawlability and an llms.txt file — as part of a broader GEO setup.

Example

A stock-photo marketplace deploys ai.txt denying image training, but scraper logs show dataset crawlers still fetching files. Adding explicit robots.txt blocks for the offending user agents plus CDN bot rules stops the traffic — illustrating where declaration ends and enforcement begins.