Robots.txt Validator · Check Syntax & AI Blocking · Nuxt SEO [NuxtSEO](https://nuxtseo.com/ "Home") - [Modules](https://nuxtseo.com/docs/nuxt-seo/getting-started/introduction) - [Tools](https://nuxtseo.com/tools) - [Pro](https://nuxtseo.com/pro) - [Learn SEO](https://nuxtseo.com/learn-seo/nuxt) [Releases](https://nuxtseo.com/releases) [1.4K](https://github.com/harlan-zw/nuxt-seo) [Nuxt SEO on GitHub](https://github.com/harlan-zw/nuxt-seo) $ nuxt-seo tools # Robots.txt Validator & AI Bot Checker Check your robots.txt for syntax errors, verify you are correctly blocking AI bots (GPTBot, Claude), and validate IETF/Cloudflare signals. URL Paste robots.txt Check Try:openai.com github.com nytimes.com ## AI Crawler Signals Standard robots.txt directives (Allow/Disallow) control crawl access, but don't explicitly handle what the data can be used for (like AI training). #### IETF Content-Usage A proposed machine-readable standard to declare if content can be used for search indexing or AI training. `Content-Usage: search=y, train-ai=n` #### Cloudflare Content-Signal Used by Cloudflare to protect your site from bots that might ignore standard robots.txt but respect these signals. `Content-Signal: ai-train=no` ## Why Validate? - **Fix Syntax Errors**Malformed directives can lead to bots ignoring your rules. A **robots.txt checker** ensures your file is valid. - **Block AI Bots & Scrapers**Verify you are correctly blocking **GPTBot**, **ClaudeBot**, **CCBot**, and other LLM crawlers to stop AI training. - **Nuclear Option**Checking if you need to **disallow all** robots? Validate that your `Disallow: /` rule is working correctly. ## Frequently Asked Questions
01 ### What does a robots.txt validator check? A robots.txt validator checks your file for syntax errors, invalid directives, and common mistakes. It verifies User-agent declarations, Allow/Disallow rules, sitemap references, and newer directives like Content-Usage for AI opt-out signals.
02 ### How do I know if my robots.txt is blocking AI bots? Enter your site URL above to analyze your robots.txt. This tool specifically checks for AI crawler rules (GPTBot, ClaudeBot, CCBot, Google-Extended, etc.) and shows which bots are blocked vs allowed. It also detects Content-Usage headers for AI training opt-out.
03 ### What is the robots.txt syntax? Each rule block starts with User-agent: followed by the bot name (* for all). Then add Disallow: /path/ to block or Allow: /path/ to permit access. Rules are case-sensitive for paths. Add Sitemap: URL at the end to reference your sitemap. Lines starting with # are comments.
04 ### Why is Google still crawling pages I blocked? Robots.txt only suggests crawl behavior - it doesn't enforce it. Also, Disallow prevents crawling but not indexing. Pages can still appear in search if linked from other sites. For true blocking, use meta robots noindex or password protection.
05 ### How do I test if a specific URL is blocked? After validating your robots.txt, use the path tester feature to check any URL against your rules. Enter a path like /admin/ and select a user-agent to see if it would be blocked or allowed based on your current rules.
Part of [Nuxt SEO](https://nuxtseo.com/) — the all-in-one technical SEO toolkit for Nuxt. Sitemaps, robots.txt, meta tags, Schema.org, and OG images in one install. ### References [Robots Exclusion Protocol The original standard for robots.txt.](https://www.robotstxt.org/orig.html) [Google: Robots.txt Specification Google's implementation and extensions.](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) [IETF: AI Preferences Vocab New proposal for machine-readable AI signals.](https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/) [Cloudflare: Content Signals Cloudflare's AI bot control signals.](https://blog.cloudflare.com/content-signals-policy/) ### Related [Robots.txt Generator](https://nuxtseo.com/tools/robots-txt-generator) [Robots.txt Module](https://nuxtseo.com/docs/robots/getting-started/introduction) [Meta Tag Checker](https://nuxtseo.com/tools/meta-tag-checker) [GitHub](https://github.com/harlan-zw/nuxt-seo) [ Discord](https://discord.com/invite/275MBUBvgP) ### [NuxtSEO](https://nuxtseo.com/ "Home") - [Getting Started](https://nuxtseo.com/docs/nuxt-seo/getting-started/introduction) - [MCP](https://nuxtseo.com/docs/nuxt-seo/guides/mcp) Modules - [Robots](https://nuxtseo.com/docs/robots/getting-started/introduction) - [Sitemap](https://nuxtseo.com/docs/sitemap/getting-started/introduction) - [OG Image](https://nuxtseo.com/docs/og-image/getting-started/introduction) - [Schema.org](https://nuxtseo.com/docs/schema-org/getting-started/introduction) - [Link Checker](https://nuxtseo.com/docs/link-checker/getting-started/introduction) - [SEO Utils](https://nuxtseo.com/docs/seo-utils/getting-started/introduction) - [Site Config](https://nuxtseo.com/docs/site-config/getting-started/introduction) - [Skew Protection](https://nuxtseo.com/docs/skew-protection/getting-started/introduction) - [AI Ready](https://nuxtseo.com/docs/ai-ready/getting-started/introduction) ### [NuxtSEO Pro](https://nuxtseo.com/pro "Home") - [Getting Started](https://nuxtseo.com/pro) - [Dashboard](https://nuxtseo.com/pro/dashboard) - [Pro MCP](https://nuxtseo.com/docs/nuxt-seo-pro/mcp/installation) ### [Learn SEO](https://nuxtseo.com/learn-seo "Learn SEO") Nuxt - [Mastering Meta](https://nuxtseo.com/learn-seo/nuxt/mastering-meta) - [Controlling Crawlers](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers) - [Launch & Listen](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen) - [Routes & Rendering](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering) - [Staying Secure](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security) Vue - [Vue SEO Guide](https://nuxtseo.com/learn-seo/vue) - [Mastering Meta](https://nuxtseo.com/learn-seo/vue/mastering-meta) - [Controlling Crawlers](https://nuxtseo.com/learn-seo/vue/controlling-crawlers) - [SPA SEO](https://nuxtseo.com/learn-seo/vue/spa) - [SSR Frameworks](https://nuxtseo.com/learn-seo/vue/ssr-frameworks) - [SEO Checklist](https://nuxtseo.com/learn-seo/checklist) - [Pre-Launch Warmup](https://nuxtseo.com/learn-seo/pre-launch-warmup) - [Backlinks & Authority](https://nuxtseo.com/learn-seo/backlinks) ### [Tools](https://nuxtseo.com/tools "SEO Tools") - [Social Share Debugger](https://nuxtseo.com/tools/social-share-debugger) - [Robots.txt Generator](https://nuxtseo.com/tools/robots-txt-generator) - [Meta Tag Checker](https://nuxtseo.com/tools/meta-tag-checker) - [HTML to Markdown](https://nuxtseo.com/tools/html-to-markdown) - [XML Sitemap Validator](https://nuxtseo.com/tools/xml-sitemap-validator) - [Schema.org Validator](https://nuxtseo.com/tools/schema-validator) - [Keyword Research Pro](https://nuxtseo.com/tools/keyword-research) - [SERP Analyzer Pro](https://nuxtseo.com/tools/serp-analyzer) - [Domain Rankings Pro](https://nuxtseo.com/tools/domain-rankings) Copyright © 2023-2026 Harlan Wilton - [MIT License](https://github.com/harlan-zw/nuxt-seo/blob/main/license) · [mdream](https://mdream.dev)