NuxtSEO

$ nuxt-seo tools

Robots.txt Generator & Tester

Generate robots.txt with AI crawler presets. Test rules instantly.

Rules

1.
(none)
IETF
Cloudflare

"No preference" excludes from output

robots.txt
# Generated by Nuxt SEO
# https://nuxtseo.com/tools/robots-txt-generator
User-agent: *
Allow: /

Test Your Rules

Access granted!
Matched: User-agent: * Allow: /

Common User Agents

Search Engines

Googlebot
Google Search (28% of bot traffic)
Bingbot
Microsoft Bing Search
DuckDuckBot
DuckDuckGo Search
YandexBot
Yandex Search (Russia)
Baiduspider
Baidu Search (China)

AI Crawlers

GPTBot
OpenAI model training (7.5% of bot traffic, most blocked bot)
ChatGPT-User
ChatGPT live browsing
OAI-SearchBot
ChatGPT Search feature
ClaudeBot
Anthropic model training
Claude-Web
Claude live browsing
anthropic-ai
Anthropic data collection
CCBot
Common Crawl dataset (Frequently blocked, used by many AI labs)
Google-Extended
Gemini/Bard training (separate from Search)
PerplexityBot
Perplexity AI search
Bytespider
TikTok/ByteDance AI
Amazonbot
Amazon Alexa training
cohere-ai
Cohere model training
Meta-ExternalAgent
Meta AI training
meta-externalfetcher
Meta data fetching
Applebot-Extended
Apple AI training (not Search)

Social Platforms

facebookexternalhit
Facebook/Meta link previews
Twitterbot
Twitter/X link previews
LinkedInBot
LinkedIn link previews
Slackbot
Slack link previews
Discordbot
Discord link previews
WhatsApp
WhatsApp link previews
TelegramBot
Telegram link previews

Source: Cloudflare 2025 Bot Traffic Report

Directives

Core Directives

User-agent: *

applies to all crawlers

Disallow: /

block entire site

Allow: /

explicitly allow (for exceptions)

Crawl-delay: 10

wait 10s between requests (Bing/Yandex)

Sitemap: URL

specify sitemap location

Content-Usage

IETF

Uses y/n values

Content-Usage: search=y

allow search indexing

Content-Usage: train-ai=n

disallow AI model training

Combine: search=y, train-ai=n

Content-Signal

Cloudflare

Uses yes/no values

search=yes

allow search indexing

ai-input=no

disallow live AI answers

ai-train=no

disallow model training

Pattern Matching

*

matches any sequence

$

matches end of URL

/*.pdf

all .pdf files

/*.php$

URLs ending in .php

Common Patterns

Disallow: /admin/Block /admin/ directory
Disallow: /*?Block URLs with query strings
Disallow: /*.json$Block all .json files
Disallow: /private/*Block everything under /private/
Allow: /api/publicAllow specific path (exception)

Block AI Training

GPTBot: most blocked bot 2024 →

$ top AI crawlers to block

GPTBotClaudeBotCCBotGoogle-ExtendedBytespiderPerplexityBot

Google-Extended = Gemini training (not Search)

$ content preference headers

IETFContent-Usage: search=y, train-ai=n
CloudflareContent-Signal: search=yes, ai-train=no

ai-input = live answers   ai-train = model training

Learn More

Robots.txt Guide

Complete guide to robots.txt syntax, common patterns, and security considerations.

XML Sitemaps

How sitemaps work with robots.txt to guide crawler discovery.

Meta Robots Tags

Page-level control when robots.txt isn't enough.

Need Dynamic Robots.txt?

Generate robots.txt rules dynamically based on routes, environments, or user conditions with the Nuxt Robots module.

Explore Nuxt Robots