Free Robots.txt Checker · Validate Syntax & AI Bot Blocking · Nuxt SEO

[NuxtSEO](https://nuxtseo.com/ "Home")

- [Modules](https://nuxtseo.com/docs/nuxt-seo/getting-started/introduction)
- [Tools](https://nuxtseo.com/tools)
- [Pro](https://nuxtseo.com/pro)
- [Learn SEO](https://nuxtseo.com/learn-seo/nuxt) [Releases](https://nuxtseo.com/releases)

[1.4K](https://github.com/harlan-zw/nuxt-seo)

[Nuxt SEO on GitHub](https://github.com/harlan-zw/nuxt-seo)

$ nuxt-seo tools

# Free Robots.txt Checker

Check your robots.txt for syntax errors, verify you are correctly blocking AI bots (GPTBot, Claude), and validate IETF/Cloudflare signals.

URL

Paste robots.txt

Check

Try:openai.com

github.com

nytimes.com

## How to Check Your Robots.txt

To check your site's robots.txt file, simply append `/robots.txt` to your domain (e.g., `example.com/robots.txt`). This file is the first thing search engines look for when visiting your site.

Our **robots.txt checker** helps you identify if this file is correctly formatted and whether it's accidentally blocking important pages. Use the "URL" tab to fetch your live file or the "Paste" tab to test changes before you deploy them.

#### What to look for:

- Does it exist at the root directory?
- Is there a User-agent declaration for each block?
- Are important assets (CSS/JS) allowed?
- Is your sitemap URL included?

## Common Robots.txt Errors

#### Missing User-agent

Rules like `Disallow: /admin/` must be preceded by a `User-agent:` line, otherwise bots will ignore them.

#### Relative URLs

Sitemap directives require absolute URLs. Using `Sitemap: /sitemap.xml` is invalid and will be ignored by search engines.

#### Blocking Resources

Blocking `/_nuxt/` or `/assets/` prevents Google from rendering your page correctly, hurting your SEO.

#### Path Case Sensitivity

Robots.txt paths are case-sensitive. `Disallow: /Admin/` will not block access to `/admin/`.

#### Conflicting Rules

Having both `Allow: /` and `Disallow: /` for the same agent can lead to unpredictable bot behavior.

## AI Bot Blocking in Robots.txt

As of 2026, managing AI crawlers is critical for site owners. Modern AI bots like **GPTBot** (OpenAI), **ClaudeBot** (Anthropic), and **CCBot** (Common Crawl) fetch data to train LLMs.

Standard robots.txt directives control access, but new standards like **IETF Content-Usage** and **Cloudflare Content-Signal** allow you to explicitly opt-out of AI training even if search indexing is allowed.

[Generate AI Rules →](https://nuxtseo.com/tools/robots-txt-generator)

#### IETF Content-Usage

A machine-readable standard to declare if content can be used for training.

`Content-Usage: search=y, train-ai=n`

#### Cloudflare Content-Signal

Used to protect your site from bots that might ignore standard robots.txt.

`Content-Signal: ai-train=no`

## Frequently Asked Questions

<details>

<summary>01 ### What does a robots.txt validator check?

</summary>

A robots.txt validator checks your file for syntax errors, invalid directives, and common mistakes. It verifies User-agent declarations, Allow/Disallow rules, sitemap references, and newer directives like Content-Usage for AI opt-out signals.

</details>

<details>

<summary>02 ### How do I know if my robots.txt is blocking AI bots?

</summary>

Enter your site URL above to analyze your robots.txt. This tool specifically checks for AI crawler rules (GPTBot, ClaudeBot, CCBot, Google-Extended, etc.) and shows which bots are blocked vs allowed. It also detects Content-Usage headers for AI training opt-out.

</details>

<details>

<summary>03 ### What is the robots.txt syntax?

</summary>

Each rule block starts with User-agent: followed by the bot name (* for all). Then add Disallow: /path/ to block or Allow: /path/ to permit access. Rules are case-sensitive for paths. Add Sitemap: URL at the end to reference your sitemap. Lines starting with # are comments.

</details>

<details>

<summary>04 ### Why is Google still crawling pages I blocked?

</summary>

Robots.txt only suggests crawl behavior - it doesn't enforce it. Also, Disallow prevents crawling but not indexing. Pages can still appear in search if linked from other sites. For true blocking, use meta robots noindex or password protection.

</details>

<details>

<summary>05 ### How do I test if a specific URL is blocked?

</summary>

After validating your robots.txt, use the path tester feature to check any URL against your rules. Enter a path like /admin/ and select a user-agent to see if it would be blocked or allowed based on your current rules.

</details>

Part of [Nuxt SEO](https://nuxtseo.com/) — the all-in-one technical SEO toolkit for Nuxt. Sitemaps, robots.txt, meta tags, Schema.org, and OG images in one install.

### References

[Robots Exclusion Protocol The original standard for robots.txt.](https://www.robotstxt.org/orig.html) [Google: Robots.txt Specification Google's implementation and extensions.](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) [IETF: AI Preferences Vocab New proposal for machine-readable AI signals.](https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/) [Cloudflare: Content Signals Cloudflare's AI bot control signals.](https://blog.cloudflare.com/content-signals-policy/)

## Learn More

[<h2>Robots.txt Generator</h2>Create a valid robots.txt file with AI bot blocking presets.](https://nuxtseo.com/tools/robots-txt-generator) [<h2>Robots.txt Guide</h2>Complete guide to robots.txt syntax, common patterns, and security.](https://nuxtseo.com/learn-seo/vue/controlling-crawlers/robots-txt) [<h2>Robots.txt Module</h2>Generate robots.txt automatically in your Nuxt application.](https://nuxtseo.com/docs/robots/getting-started/introduction) [<h2>Meta Tag Checker</h2>Check your page-level meta robots and Open Graph tags.](https://nuxtseo.com/tools/meta-tag-checker)

[GitHub](https://github.com/harlan-zw/nuxt-seo) [ Discord](https://discord.com/invite/275MBUBvgP)

### [NuxtSEO](https://nuxtseo.com/ "Home")

- [Getting Started](https://nuxtseo.com/docs/nuxt-seo/getting-started/introduction)
- [MCP](https://nuxtseo.com/docs/nuxt-seo/guides/mcp)

Modules

- [Robots](https://nuxtseo.com/docs/robots/getting-started/introduction)
- [Sitemap](https://nuxtseo.com/docs/sitemap/getting-started/introduction)
- [OG Image](https://nuxtseo.com/docs/og-image/getting-started/introduction)
- [Schema.org](https://nuxtseo.com/docs/schema-org/getting-started/introduction)
- [Link Checker](https://nuxtseo.com/docs/link-checker/getting-started/introduction)
- [SEO Utils](https://nuxtseo.com/docs/seo-utils/getting-started/introduction)
- [Site Config](https://nuxtseo.com/docs/site-config/getting-started/introduction)
- [Skew Protection](https://nuxtseo.com/docs/skew-protection/getting-started/introduction)
- [AI Ready](https://nuxtseo.com/docs/ai-ready/getting-started/introduction)

### [NuxtSEO Pro](https://nuxtseo.com/pro "Nuxt SEO Pro")

- [Getting Started](https://nuxtseo.com/pro)
- [Dashboard](https://nuxtseo.com/pro/dashboard)
- [Pro MCP](https://nuxtseo.com/pro/docs/getting-started/mcp-setup)

### [Learn SEO](https://nuxtseo.com/learn-seo "Learn SEO")

Nuxt

- [Mastering Meta](https://nuxtseo.com/learn-seo/nuxt/mastering-meta)
- [Controlling Crawlers](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers)
- [Launch & Listen](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen)
- [Routes & Rendering](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering)
- [Staying Secure](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security)

Vue

- [Vue SEO Guide](https://nuxtseo.com/learn-seo/vue)
- [Mastering Meta](https://nuxtseo.com/learn-seo/vue/mastering-meta)
- [Controlling Crawlers](https://nuxtseo.com/learn-seo/vue/controlling-crawlers)
- [SPA SEO](https://nuxtseo.com/learn-seo/vue/spa)
- [SSR Frameworks](https://nuxtseo.com/learn-seo/vue/ssr-frameworks)
- [SEO Checklist](https://nuxtseo.com/learn-seo/checklist)
- [Pre-Launch Warmup](https://nuxtseo.com/learn-seo/pre-launch-warmup)
- [Backlinks & Authority](https://nuxtseo.com/learn-seo/backlinks)

### [Tools](https://nuxtseo.com/tools "SEO Tools")

- [Social Share Debugger](https://nuxtseo.com/tools/social-share-debugger)
- [Robots.txt Generator](https://nuxtseo.com/tools/robots-txt-generator)
- [Meta Tag Checker](https://nuxtseo.com/tools/meta-tag-checker)
- [HTML to Markdown](https://nuxtseo.com/tools/html-to-markdown)
- [XML Sitemap Validator](https://nuxtseo.com/tools/xml-sitemap-validator)
- [Schema.org Validator](https://nuxtseo.com/tools/schema-validator)
- [Keyword Idea Generator](https://nuxtseo.com/tools/keyword-generator)
- [Keyword Research](https://nuxtseo.com/tools/keyword-research)
- [SERP Analyzer](https://nuxtseo.com/tools/serp-analyzer)
- [Domain Rankings](https://nuxtseo.com/tools/domain-rankings)

Copyright © 2023-2026 Harlan Wilton - [MIT License](https://github.com/harlan-zw/nuxt-seo/blob/main/license) · [mdream](https://mdream.dev)