Index of every page on this site: llms.txt

AI Directives allow you to express preferences about how AI systems, search engines, and automated tools should interact with your content. Two standards are supported:

  • Content-Usage - IETF standard with broader automation categories
  • Content-Signal - Cloudflare's widely-deployed implementation focused on AI use cases

You can publish both directives in robots.txt.

Test AI crawler blocking - Our Robots.txt Generator includes presets for GPTBot, ClaudeBot, and other AI crawlers.
Important: AI directives rely on voluntary compliance by crawlers and AI systems. They are not enforced by web servers and should be combined with other protection methods for sensitive content.

Content-Usage (IETF aipref-vocab)

The Content-Usage directive follows the IETF AI Preferences specification, providing a standardized way to express automation preferences.

Categories

CategoryDescriptionExample Use Case
botsAutomated ProcessingGeneral bot access and crawling
train-aiFoundation Model ProductionTraining large language models
ai-outputAI OutputAI generated responses and content
searchSearchIndexing for search results

Values

  • y - Allow this category of use
  • n - Disallow this category of use

Syntax

robots.txt
User-agent: *
Content-Usage: <category>=<value>[, <category>=<value>]
Content-Usage: /path/ <category>=<value>[, <category>=<value>]

Examples

Block AI Training Globally:

robots.txt
User-agent: *
Allow: /
Content-Usage: train-ai=n

Allow Bots, Block AI Training:

robots.txt
User-agent: *
Allow: /
Content-Usage: bots=y, train-ai=n

Path-Specific Rules:

robots.txt
User-agent: *
Allow: /
Content-Usage: train-ai=n
Content-Usage: /docs/ train-ai=y
Content-Usage: /api/ train-ai=n

Programmatic Configuration

Object Format (Recommended) - Type-safe with autocomplete:

nuxt.config.ts
export default defineNuxtConfig({
  robots: {
    groups: [
      {
        userAgent: '*',
        allow: '/',
        contentUsage: {
          'train-ai': 'n'
        }
      }
    ]
  }
})

Content-Signal (Cloudflare/IETF aipref-contentsignals)

Content-Signal is Cloudflare's implementation based on IETF aipref-contentsignals.

Categories

CategoryDescriptionExample Use Case
searchSearch ApplicationsIndexing for search results and snippets
ai-inputAI InputRAG, grounding, generative AI search answers
ai-trainAI TrainingTraining or fine-tuning AI models

Values

  • yes - Allow this category of use
  • no - Disallow this category of use

Syntax

robots.txt
User-agent: *
Content-Signal: <category>=<value>[, <category>=<value>]
Content-Signal: /path/ <category>=<value>[, <category>=<value>]

Examples

Block AI Training, Allow Search:

robots.txt
User-agent: *
Allow: /
Content-Signal: ai-train=no, search=yes

Block All AI Usage:

robots.txt
User-agent: *
Allow: /
Content-Signal: ai-train=no, ai-input=no, search=yes

Path-Specific Rules:

robots.txt
User-agent: *
Allow: /
Content-Signal: ai-train=no, search=yes
Content-Signal: /docs/ ai-input=yes
Content-Signal: /api/ ai-train=no, ai-input=no, search=no

Programmatic Configuration

Object Format (Recommended) - Type-safe with autocomplete:

nuxt.config.ts
export default defineNuxtConfig({
  robots: {
    groups: [
      {
        userAgent: '*',
        allow: '/',
        contentSignal: {
          'ai-train': 'no',
          'search': 'yes'
        }
      }
    ]
  }
})

HTTP headers

Set robots.contentSignalHeader: true to publish a site-wide policy as an HTTP response header. The option defaults to false and requires robots.header: true. Use the User-agent: * policy from the programmatic example. You can also put that policy in the robots.txt file that Robots merges.

Replace the example URL with a deployed page. Check for Content-Signal: ai-train=no, search=yes.

curl -i https://example.com/

Robots leaves unspecified permissions out of the header. An explicit response header takes precedence. If any group targets a specific user-agent, or a policy contains path exceptions or conflicting permissions, Robots keeps the policy in robots.txt. Set explicit route headers when those routes need an HTTP policy too.

On Cloudflare, enabling the option also writes the fallback to _headers for prerendered assets. An explicit static policy or header removal takes precedence. Configure the response header on your host if you deploy static pages elsewhere.

Cloudflare allows 100 static header rules. If adding the fallback pushes the final file over that limit, the build fails and keeps the policy rules. Combine rules that share a header or reduce other rules, then build again.

Set robots.header: false to disable both the automatic Content-Signal and X-Robots-Tag headers.

Cloudflare Markdown for Agents preserves the origin's header. If the origin sends no header, Cloudflare sends a policy that allows training, search, and AI input. See Cloudflare's Content Signals policy.

Vendor-Specific AI Tokens

While Content-Usage and Content-Signal are emerging standards, some major AI providers offer specific User-Agent tokens to opt-out of AI training while maintaining search visibility.

These are highly effective as they are strictly adhered to by their respective companies.

Google-Extended

Google-Extended is a standalone token that allows you to control whether your content is used to help improve Google's generative AI APIs and services (Gemini, Vertex AI).

  • Does NOT affect your site's ranking in Google Search.
  • Does NOT stop Googlebot from crawling your site for indexing.
robots.txt
User-agent: Google-Extended
Disallow: /

Applebot-Extended

Applebot-Extended allows you to opt-out of having your website content used to train Apple's foundation models that power generative AI features across Apple products.

  • Does NOT affect your site's ranking in Apple Search (Spotlight, Siri).
  • Does NOT stop Applebot from crawling your site.
robots.txt
User-agent: Applebot-Extended
Disallow: /

Dataset Crawlers

Some crawlers are specifically designed to build massive datasets used for training many different AI models. Blocking these can be a broad-stroke approach to AI protection.

  • CCBot (Common Crawl): Used by many open-source and commercial models (including early GPT versions).
  • Bytespider: Crawler for ByteDance (TikTok) AI models.
  • Diffbot: AI-powered knowledge extraction.
robots.txt
User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Diffbot
Disallow: /

Using Both Together

You can use both Content-Usage and Content-Signal in the same robots.txt for comprehensive coverage:

robots.txt
User-agent: *
Allow: /
Content-Usage: bots=y, train-ai=n
Content-Signal: ai-train=no, search=yes
export default defineNuxtConfig({
  robots: {
    groups: [
      {
        userAgent: '*',
        allow: '/',
        contentUsage: {
          'train-ai': 'n'
        },
        contentSignal: {
          'ai-train': 'no',
          'search': 'yes'
        }
      }
    ]
  }
})

Examples

Block All AI Training

User-agent: *
Allow: /
Content-Usage: train-ai=n

Documentation-Only Training

User-agent: *
Allow: /
Content-Usage: train-ai=n
Content-Usage: /docs/ train-ai=y
Was this page helpful?