AI Directives
AI Directives allow you to express preferences about how AI systems, search engines, and automated tools should interact with your content. Two standards are supported:
- Content-Usage - IETF standard with broader automation categories
- Content-Signal - Cloudflare's widely-deployed implementation focused on AI use cases
You can publish both directives in robots.txt.
Content-Usage (IETF aipref-vocab)
The Content-Usage directive follows the IETF AI Preferences specification, providing a standardized way to express automation preferences.
Categories
| Category | Description | Example Use Case |
|---|---|---|
bots | Automated Processing | General bot access and crawling |
train-ai | Foundation Model Production | Training large language models |
ai-output | AI Output | AI generated responses and content |
search | Search | Indexing for search results |
Values
y- Allow this category of usen- Disallow this category of use
Syntax
User-agent: *
Content-Usage: <category>=<value>[, <category>=<value>]
Content-Usage: /path/ <category>=<value>[, <category>=<value>]Examples
Block AI Training Globally:
User-agent: *
Allow: /
Content-Usage: train-ai=nAllow Bots, Block AI Training:
User-agent: *
Allow: /
Content-Usage: bots=y, train-ai=nPath-Specific Rules:
User-agent: *
Allow: /
Content-Usage: train-ai=n
Content-Usage: /docs/ train-ai=y
Content-Usage: /api/ train-ai=nProgrammatic Configuration
Object Format (Recommended) - Type-safe with autocomplete:
export default defineNuxtConfig({
robots: {
groups: [
{
userAgent: '*',
allow: '/',
contentUsage: {
'train-ai': 'n'
}
}
]
}
})Content-Signal (Cloudflare/IETF aipref-contentsignals)
Content-Signal is Cloudflare's implementation based on IETF aipref-contentsignals.
Categories
| Category | Description | Example Use Case |
|---|---|---|
search | Search Applications | Indexing for search results and snippets |
ai-input | AI Input | RAG, grounding, generative AI search answers |
ai-train | AI Training | Training or fine-tuning AI models |
Values
yes- Allow this category of useno- Disallow this category of use
Syntax
User-agent: *
Content-Signal: <category>=<value>[, <category>=<value>]
Content-Signal: /path/ <category>=<value>[, <category>=<value>]Examples
Block AI Training, Allow Search:
User-agent: *
Allow: /
Content-Signal: ai-train=no, search=yesBlock All AI Usage:
User-agent: *
Allow: /
Content-Signal: ai-train=no, ai-input=no, search=yesPath-Specific Rules:
User-agent: *
Allow: /
Content-Signal: ai-train=no, search=yes
Content-Signal: /docs/ ai-input=yes
Content-Signal: /api/ ai-train=no, ai-input=no, search=noProgrammatic Configuration
Object Format (Recommended) - Type-safe with autocomplete:
export default defineNuxtConfig({
robots: {
groups: [
{
userAgent: '*',
allow: '/',
contentSignal: {
'ai-train': 'no',
'search': 'yes'
}
}
]
}
})HTTP headers
Set robots.contentSignalHeader: true to publish a site-wide policy as an HTTP response header.
The option defaults to false and requires robots.header: true.
Use the User-agent: * policy from the programmatic example.
You can also put that policy in the robots.txt file that Robots merges.
Replace the example URL with a deployed page. Check for Content-Signal: ai-train=no, search=yes.
curl -i https://example.com/Robots leaves unspecified permissions out of the header. An explicit response header takes precedence.
If any group targets a specific user-agent, or a policy contains path exceptions or conflicting permissions, Robots keeps the policy in robots.txt.
Set explicit route headers when those routes need an HTTP policy too.
On Cloudflare, enabling the option also writes the fallback to _headers for prerendered assets.
An explicit static policy or header removal takes precedence.
Configure the response header on your host if you deploy static pages elsewhere.
Cloudflare allows 100 static header rules. If adding the fallback pushes the final file over that limit, the build fails and keeps the policy rules. Combine rules that share a header or reduce other rules, then build again.
Set robots.header: false to disable both the automatic Content-Signal and X-Robots-Tag headers.
Cloudflare Markdown for Agents preserves the origin's header. If the origin sends no header, Cloudflare sends a policy that allows training, search, and AI input. See Cloudflare's Content Signals policy.
Vendor-Specific AI Tokens
While Content-Usage and Content-Signal are emerging standards, some major AI providers offer specific User-Agent tokens to opt-out of AI training while maintaining search visibility.
These are highly effective as they are strictly adhered to by their respective companies.
Google-Extended
Google-Extended is a standalone token that allows you to control whether your content is used to help improve Google's generative AI APIs and services (Gemini, Vertex AI).
- Does NOT affect your site's ranking in Google Search.
- Does NOT stop Googlebot from crawling your site for indexing.
User-agent: Google-Extended
Disallow: /Applebot-Extended
Applebot-Extended allows you to opt-out of having your website content used to train Apple's foundation models that power generative AI features across Apple products.
- Does NOT affect your site's ranking in Apple Search (Spotlight, Siri).
- Does NOT stop Applebot from crawling your site.
User-agent: Applebot-Extended
Disallow: /Dataset Crawlers
Some crawlers are specifically designed to build massive datasets used for training many different AI models. Blocking these can be a broad-stroke approach to AI protection.
- CCBot (Common Crawl): Used by many open-source and commercial models (including early GPT versions).
- Bytespider: Crawler for ByteDance (TikTok) AI models.
- Diffbot: AI-powered knowledge extraction.
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Diffbot
Disallow: /Using Both Together
You can use both Content-Usage and Content-Signal in the same robots.txt for comprehensive coverage:
User-agent: *
Allow: /
Content-Usage: bots=y, train-ai=n
Content-Signal: ai-train=no, search=yesexport default defineNuxtConfig({
robots: {
groups: [
{
userAgent: '*',
allow: '/',
contentUsage: {
'train-ai': 'n'
},
contentSignal: {
'ai-train': 'no',
'search': 'yes'
}
}
]
}
})export default defineNuxtConfig({
robots: {
groups: [
{
userAgent: '*',
allow: '/',
contentUsage: ['bots=y, train-ai=n'],
contentSignal: ['ai-train=no, search=yes']
}
]
}
})Examples
Block All AI Training
User-agent: *
Allow: /
Content-Usage: train-ai=nUser-agent: *
Allow: /
Content-Signal: ai-train=noDocumentation-Only Training
User-agent: *
Allow: /
Content-Usage: train-ai=n
Content-Usage: /docs/ train-ai=yUser-agent: *
Allow: /
Content-Signal: ai-train=no
Content-Signal: /docs/ ai-train=yes