AI Directives allow you to express preferences about how AI systems, search engines, and automated tools should interact with your content. Two standards are supported:
Both can be used together in your robots.txt file and are enforced through the robots.txt protocol.
Important: AI directives rely on voluntary compliance by crawlers and AI systems. They are not enforced by web servers and should be combined with other protection methods for sensitive content.
The Content-Usage directive follows the IETF AI Preferences specification, providing a standardized way to express automation preferences.
| Category | Description | Example Use Case |
|---|---|---|
train-ai | Foundation Model Production | Training large language models |
y - Allow this category of usen - Disallow this category of useUser-agent: *
Content-Usage: <category>=<value>[, <category>=<value>]
Content-Usage: /path/ <category>=<value>[, <category>=<value>]
User-agent: *
Allow: /
Content-Usage: train-ai=n
User-agent: *
Allow: /
Content-Usage: bots=y, train-ai=n
User-agent: *
Allow: /
Content-Usage: train-ai=n
Content-Usage: /docs/ train-ai=y
Content-Usage: /api/ train-ai=n
Object Format (Recommended) - Type-safe with autocomplete:
export default defineNuxtConfig({
robots: {
groups: [
{
userAgent: '*',
allow: '/',
contentUsage: {
'train-ai': 'n'
}
}
]
}
})
Content-Signal is Cloudflare's implementation based on IETF aipref-contentsignals.
| Category | Description | Example Use Case |
|---|---|---|
search | Search Applications | Indexing for search results and snippets |
ai-input | AI Input | RAG, grounding, generative AI search answers |
ai-train | AI Training | Training or fine-tuning AI models |
yes - Allow this category of useno - Disallow this category of useUser-agent: *
Content-Signal: <category>=<value>[, <category>=<value>]
Content-Signal: /path/ <category>=<value>[, <category>=<value>]
User-agent: *
Allow: /
Content-Signal: ai-train=no, search=yes
User-agent: *
Allow: /
Content-Signal: ai-train=no, ai-input=no, search=yes
User-agent: *
Allow: /
Content-Signal: ai-train=no, search=yes
Content-Signal: /docs/ ai-input=yes
Content-Signal: /api/ ai-train=no, ai-input=no, search=no
Object Format (Recommended) - Type-safe with autocomplete:
export default defineNuxtConfig({
robots: {
groups: [
{
userAgent: '*',
allow: '/',
contentSignal: {
'ai-train': 'no',
'search': 'yes'
}
}
]
}
})
You can use both Content-Usage and Content-Signal in the same robots.txt for comprehensive coverage:
User-agent: *
Allow: /
Content-Usage: bots=y, train-ai=n
Content-Signal: ai-train=no, search=yes
export default defineNuxtConfig({
robots: {
groups: [
{
userAgent: '*',
allow: '/',
contentUsage: {
'train-ai': 'n'
},
contentSignal: {
'ai-train': 'no',
'search': 'yes'
}
}
]
}
})
export default defineNuxtConfig({
robots: {
groups: [
{
userAgent: '*',
allow: '/',
contentUsage: ['bots=y, train-ai=n'],
contentSignal: ['ai-train=no, search=yes']
}
]
}
})
User-agent: *
Allow: /
Content-Usage: train-ai=n
User-agent: *
Allow: /
Content-Signal: ai-train=no
User-agent: *
Allow: /
Content-Usage: train-ai=n
Content-Usage: /docs/ train-ai=y
User-agent: *
Allow: /
Content-Signal: ai-train=no
Content-Signal: /docs/ ai-train=yes