Index of every page on this site: llms.txt
Nuxt SEO
Nuxt SEO

Menu

$ nuxt-seo tools

Free Robots.txt Generator

Write and test crawler rules, add sitemap references, then download the finished robots.txt file.

Rules

1.
Disallow paths
(none)
Allow paths
Sitemap URLs
IETF
Cloudflare

"No preference" excludes from output

robots.txt
# Generated by Nuxt SEO
# https://nuxtseo.com/tools/robots-txt-generator
User-agent: *
Allow: /

Run these tools from your editor

The Nuxt SEO MCP server exposes the same checks to Claude, Codex, Cursor, and other MCP clients.

Open MCP setup
Was this tool helpful?

Your feedback helps us improve

Test Your Rules

Allowed by these rules
Matched: User-agent: * Allow: /

Directives

Core Directives

User-agent: *

applies to all crawlers

Disallow: /

block entire site

Allow: /

explicitly allow (for exceptions)

Crawl-delay: 10

wait 10s between requests (Bing/Yandex)

Sitemap: URL

specify sitemap location

Content-Usage IETF

Uses y/n values

Content-Usage: search=y

allow search indexing

Content-Usage: train-ai=n

disallow AI model training

Combine: search=y, train-ai=n

Content-Signal Cloudflare

Uses yes/no values

search=yes

allow search indexing

ai-input=no

disallow live AI answers

ai-train=no

disallow model training

Pattern Matching

*

matches any sequence

$

matches end of URL

/*.pdf

all .pdf files

/*.php$

URLs ending in .php

Common questions

What is a robots.txt generator?

A robots.txt generator writes crawler rules and sitemap references in the expected text format. Crawlers choose whether to follow these rules, so the file cannot enforce access control.

How do I block AI bots with robots.txt?

Add User-agent rules for AI crawlers like GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), and Google-Extended (Gemini training). Set Disallow: / under each to block them. Note: AI companies may not respect robots.txt for training data already collected.

What is the meta robots nofollow directive?

The nofollow directive tells search engines not to follow links on a page. It belongs in a meta robots tag or link rel attribute, not robots.txt. Robots.txt only describes crawling access.

Where do I put my robots.txt file?

Place robots.txt in your website's root directory (e.g., example.com/robots.txt). It must be accessible at this exact URL. Search engines check this location automatically. Most web frameworks have built-in ways to generate this file.

Can robots.txt block pages from Google?

Robots.txt prevents crawling but not indexing. Blocked pages can still appear in search results if linked from other sites. To fully remove pages from search results, use the meta robots noindex tag or X-Robots-Tag HTTP header instead.

How do I test my robots.txt file before publishing?

Use the Rule Tester above: enter a URL path and pick a user-agent to see if your current rules would allow or block it. You can also select "Test Live Robots.txt" to import an already-published robots.txt file and test changes against it before you deploy.

Generated one file. Nuxt SEO Pro audits crawl and indexing issues across the rest of your site.

Monitor in Pro