$ nuxt-seo tools
Check your robots.txt for syntax errors, verify you are correctly blocking AI bots (GPTBot, Claude), and validate IETF/Cloudflare signals.
Standard robots.txt directives (Allow/Disallow) control crawl access, but don't explicitly handle what the data can be used for (like AI training).
A proposed machine-readable standard to declare if content can be used for search indexing or AI training.
Content-Usage: search=y, train-ai=nUsed by Cloudflare to protect your site from bots that might ignore standard robots.txt but respect these signals.
Content-Signal: ai-train=noDisallow: / rule is working correctly.A robots.txt validator checks your file for syntax errors, invalid directives, and common mistakes. It verifies User-agent declarations, Allow/Disallow rules, sitemap references, and newer directives like Content-Usage for AI opt-out signals.
Enter your site URL above to analyze your robots.txt. This tool specifically checks for AI crawler rules (GPTBot, ClaudeBot, CCBot, Google-Extended, etc.) and shows which bots are blocked vs allowed. It also detects Content-Usage headers for AI training opt-out.
Each rule block starts with User-agent: followed by the bot name (* for all). Then add Disallow: /path/ to block or Allow: /path/ to permit access. Rules are case-sensitive for paths. Add Sitemap: URL at the end to reference your sitemap. Lines starting with # are comments.
Robots.txt only suggests crawl behavior - it doesn't enforce it. Also, Disallow prevents crawling but not indexing. Pages can still appear in search if linked from other sites. For true blocking, use meta robots noindex or password protection.
After validating your robots.txt, use the path tester feature to check any URL against your rules. Enter a path like /admin/ and select a user-agent to see if it would be blocked or allowed based on your current rules.