---
title: "Free Robots.txt Generator · Write and Test Crawler Rules · Nuxt SEO"
canonical_url: "https://nuxtseo.com/tools/robots-txt-generator"
last_updated: "2026-05-05T14:01:10.949Z"
meta:
  description: "Write, test, and download a robots.txt file with crawler presets, sitemap references, and optional AI-use signals."
  "og:description": "Write, test, and download a robots.txt file with crawler presets, sitemap references, and optional AI-use signals."
  "og:title": "Free Robots.txt Generator · Write and Test Crawler Rules · Nuxt SEO"
---

Nuxt SEO on GitHub

$ nuxt-seo tools

# Free Robots.txt Generator

Write and test crawler rules, add sitemap references, then download the finished robots.txt file.

## **Rules **

1\.

Disallow paths

*(none) *

Allow paths

Sitemap URLs

Content-Usage [IETF](https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/)

search

train-ai

Content-Signal [Cloudflare](https://blog.cloudflare.com/content-signals-policy/)

search

ai-input

ai-train

"No preference" excludes from output

```
# Generated by Nuxt SEO
# https://nuxtseo.com/tools/robots-txt-generator
User-agent: *
Allow: /
```

**Run these tools from your editor**

The Nuxt SEO MCP server exposes the same checks to Claude, Codex, Cursor, and other MCP clients.

[**Open MCP setup **](https://nuxtseo.com/docs/nuxt-seo/guides/mcp)

**Was this tool helpful? **

Your feedback helps us improve

## Test Your Rules

URL path

User agent

**Allowed by these rules**

Matched: User-agent: \* Allow: /

## Directives

### **Core Directives **

`User-agent: *`

applies to all crawlers

`Disallow: /`

block entire site

`Allow: /`

explicitly allow (for exceptions)

`Crawl-delay: 10`

wait 10s between requests (Bing/Yandex)

`Sitemap: URL`

specify sitemap location

### **Content-Usage **[IETF](https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/)

Uses `y`/`n` values

`Content-Usage: search=y`

allow search indexing

`Content-Usage: train-ai=n`

disallow AI model training

Combine: `search=y, train-ai=n`

### **Content-Signal **[Cloudflare](https://blog.cloudflare.com/content-signals-policy/)

Uses `yes`/`no` values

`search=yes`

allow search indexing

`ai-input=no`

disallow live AI answers

`ai-train=no`

disallow model training

### **Pattern Matching **

`*`

matches any sequence

`$`

matches end of URL

`/*.pdf`

all .pdf files

`/*.php$`

URLs ending in .php

## Common questions

<details>

<summary>What is a robots.txt generator?</summary>



A robots.txt generator writes crawler rules and sitemap references in the expected text format. Crawlers choose whether to follow these rules, so the file cannot enforce access control.

</details>

<details>

<summary>How do I block AI bots with robots.txt?</summary>



Add User-agent rules for AI crawlers like GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), and Google-Extended (Gemini training). Set Disallow: / under each to block them. Note: AI companies may not respect robots.txt for training data already collected.

</details>

<details>

<summary>What is the meta robots nofollow directive?</summary>



The nofollow directive tells search engines not to follow links on a page. It belongs in a meta robots tag or link rel attribute, not robots.txt. Robots.txt only describes crawling access.

</details>

<details>

<summary>Where do I put my robots.txt file?</summary>



Place robots.txt in your website's root directory (e.g., example.com/robots.txt). It must be accessible at this exact URL. Search engines check this location automatically. Most web frameworks have built-in ways to generate this file.

</details>

<details>

<summary>Can robots.txt block pages from Google?</summary>



Robots.txt prevents crawling but not indexing. Blocked pages can still appear in search results if linked from other sites. To fully remove pages from search results, use the meta robots noindex tag or X-Robots-Tag HTTP header instead.

</details>

<details>

<summary>How do I test my robots.txt file before publishing?</summary>



Use the Rule Tester above: enter a URL path and pick a user-agent to see if your current rules would allow or block it. You can also select "Test Live Robots.txt" to import an already-published robots.txt file and test changes against it before you deploy.

</details>

Generated one file. Nuxt SEO Pro audits crawl and indexing issues across the rest of your site.

[**Monitor in Pro **](https://nuxtseo.com/pro)