AI Directives

AI Directives allow you to express preferences about how AI systems, search engines, and automated tools should interact with your content. Two standards are supported:

• Content-Usage - IETF standard with broader automation categories
• Content-Signal - Cloudflare's widely-deployed implementation focused on AI use cases

Both can be used together in your robots.txt file and are enforced through the robots.txt protocol.

Content-Usage (IETF aipref-vocab)

The Content-Usage directive follows the IETF AI Preferences specification, providing a standardized way to express automation preferences.

Categories

Values

• y - Allow this category of use
• n - Disallow this category of use

Syntax

User-agent: *
Content-Usage: <category>=<value>[, <category>=<value>]
Content-Usage: /path/ <category>=<value>[, <category>=<value>]

Examples

Block AI Training Globally:

User-agent: *
Allow: /
Content-Usage: train-ai=n

Allow Bots, Block AI Training:

User-agent: *
Allow: /
Content-Usage: bots=y, train-ai=n

Path-Specific Rules:

User-agent: *
Allow: /
Content-Usage: train-ai=n
Content-Usage: /docs/ train-ai=y
Content-Usage: /api/ train-ai=n

Programmatic Configuration

Object Format (Recommended) - Type-safe with autocomplete:

export default defineNuxtConfig({
  robots: {
    groups: [
      {
        userAgent: '*',
        allow: '/',
        contentUsage: {
          'train-ai': 'n'
        }
      }
    ]
  }
})

Content-Signal (Cloudflare/IETF aipref-contentsignals)

Content-Signal is Cloudflare's implementation based on IETF aipref-contentsignals.

Categories

Values

• yes - Allow this category of use
• no - Disallow this category of use

Syntax

User-agent: *
Content-Signal: <category>=<value>[, <category>=<value>]
Content-Signal: /path/ <category>=<value>[, <category>=<value>]

Examples

Block AI Training, Allow Search:

User-agent: *
Allow: /
Content-Signal: ai-train=no, search=yes

Block All AI Usage:

User-agent: *
Allow: /
Content-Signal: ai-train=no, ai-input=no, search=yes

Path-Specific Rules:

User-agent: *
Allow: /
Content-Signal: ai-train=no, search=yes
Content-Signal: /docs/ ai-input=yes
Content-Signal: /api/ ai-train=no, ai-input=no, search=no

Programmatic Configuration

Object Format (Recommended) - Type-safe with autocomplete:

export default defineNuxtConfig({
  robots: {
    groups: [
      {
        userAgent: '*',
        allow: '/',
        contentSignal: {
          'ai-train': 'no',
          'search': 'yes'
        }
      }
    ]
  }
})

Vendor-Specific AI Tokens

While Content-Usage and Content-Signal are emerging standards, some major AI providers offer specific User-Agent tokens to opt-out of AI training while maintaining search visibility.

These are highly effective as they are strictly adhered to by their respective companies.

Google-Extended

Google-Extended is a standalone token that allows you to control whether your content is used to help improve Google's generative AI APIs and services (Gemini, Vertex AI).

• Does NOT affect your site's ranking in Google Search.
• Does NOT stop Googlebot from crawling your site for indexing.

User-agent: Google-Extended
Disallow: /

Applebot-Extended

Applebot-Extended allows you to opt-out of having your website content used to train Apple's foundation models that power generative AI features across Apple products.

• Does NOT affect your site's ranking in Apple Search (Spotlight, Siri).
• Does NOT stop Applebot from crawling your site.

User-agent: Applebot-Extended
Disallow: /

Dataset Crawlers

Some crawlers are specifically designed to build massive datasets used for training many different AI models. Blocking these can be a broad-stroke approach to AI protection.

• CCBot (Common Crawl): Used by many open-source and commercial models (including early GPT versions).
• Bytespider: Crawler for ByteDance (TikTok) AI models.
• Diffbot: AI-powered knowledge extraction.

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Diffbot
Disallow: /

Using Both Together

You can use both Content-Usage and Content-Signal in the same robots.txt for comprehensive coverage:

User-agent: *
Allow: /
Content-Usage: bots=y, train-ai=n
Content-Signal: ai-train=no, search=yes

export default defineNuxtConfig({
  robots: {
    groups: [
      {
        userAgent: '*',
        allow: '/',
        contentUsage: {
          'train-ai': 'n'
        },
        contentSignal: {
          'ai-train': 'no',
          'search': 'yes'
        }
      }
    ]
  }
})

export default defineNuxtConfig({
  robots: {
    groups: [
      {
        userAgent: '*',
        allow: '/',
        contentUsage: ['bots=y, train-ai=n'],
        contentSignal: ['ai-train=no, search=yes']
      }
    ]
  }
})

Examples

Block All AI Training

User-agent: *
Allow: /
Content-Usage: train-ai=n

User-agent: *
Allow: /
Content-Signal: ai-train=no

Documentation-Only Training

User-agent: *
Allow: /
Content-Usage: train-ai=n
Content-Usage: /docs/ train-ai=y

User-agent: *
Allow: /
Content-Signal: ai-train=no
Content-Signal: /docs/ ai-train=yes