--- title: "Robots.txt Validator · Syntax & Crawler Checker · Nuxt SEO" description: "Check robots.txt syntax, test one crawler and path, and review AI crawler preferences declared in the file." canonical_url: "https://nuxtseo.com/tools/robots-txt-validator" last_updated: "2026-10-14T05:20:54.424Z" meta: "og:description": "Check robots.txt syntax, test one crawler and path, and review AI crawler preferences declared in the file." "og:image": "https://nuxtseo.com/_og/d/c_Tool%2Ctitle_~Um9ib3RzLnR4dCBWYWxpZGF0b3IgwrcgU3ludGF4ICYgQ3Jhd2xlciBDaGVja2Vy%2Cdescription_~Q2hlY2sgcm9ib3RzLnR4dCBzeW50YXgsIHRlc3Qgb25lIGNyYXdsZXIgYW5kIHBhdGgsIGFuZCByZXZpZXcgQUkgY3Jhd2xlciBwcmVmZXJlbmNlcyBkZWNsYXJlZCBpbiB0aGUgZmlsZS4%2Cicon_bot%2Cp_Ii90b29scy9yb2JvdHMtdHh0LXZhbGlkYXRvciI%2Cs_1eSQfgCzAkI0Kqoi.png" "og:image:height": "600" "og:image:type": image/png "og:image:width": "1200" "og:title": "Robots.txt Validator · Syntax & Crawler Checker · Nuxt SEO" "twitter:image": "https://nuxtseo.com/_og/d/c_Tool%2Ctitle_~Um9ib3RzLnR4dCBWYWxpZGF0b3IgwrcgU3ludGF4ICYgQ3Jhd2xlciBDaGVja2Vy%2Cdescription_~Q2hlY2sgcm9ib3RzLnR4dCBzeW50YXgsIHRlc3Qgb25lIGNyYXdsZXIgYW5kIHBhdGgsIGFuZCByZXZpZXcgQUkgY3Jhd2xlciBwcmVmZXJlbmNlcyBkZWNsYXJlZCBpbiB0aGUgZmlsZS4%2Cicon_bot%2Cp_Ii90b29scy9yb2JvdHMtdHh0LXZhbGlkYXRvciI%2Cs_1eSQfgCzAkI0Kqoi.png" --- > Index of every page on this site: [llms.txt](https://nuxtseo.com/llms.txt) $ nuxt-seo tools # Free Robots.txt Validator Check robots.txt syntax, test a crawler against one path, and review any AI crawler preferences. **Site or robots.txt URL** Try: ## How to Check Your Robots.txt To check your site's robots.txt file, simply append `/robots.txt` to your domain (e.g., `example.com/robots.txt`). This file is the first thing search engines look for when visiting your site. The **robots.txt validator** reports syntax issues and matched rules. Use the URL tab to fetch a live file or the Paste tab to test changes before deployment. #### **What to look for: ** - Does it exist at the root directory? - Is there a User-agent declaration for each block? - Are important assets (CSS/JS) allowed? - Is your sitemap URL included? ## Common Robots.txt Errors #### **Missing User-agent ** Rules like `Disallow: /admin/` must be preceded by a `User-agent:` line, otherwise bots will ignore them. #### **Relative URLs ** Sitemap directives require absolute URLs. Using `Sitemap: /sitemap.xml` is invalid and will be ignored by search engines. #### **Blocking Resources ** Blocking `/_nuxt/` or `/assets/` can stop crawlers from loading files needed to render the page. #### **Path Case Sensitivity ** Robots.txt paths are case-sensitive. `Disallow: /Admin/` will not block access to `/admin/`. #### **Conflicting Rules ** Having both `Allow: /` and `Disallow: /` for the same agent can lead to unpredictable bot behavior. ## AI Bot Blocking in Robots.txt Crawlers such as **GPTBot**, **ClaudeBot**, and **CCBot** publish user-agent tokens that sites can address in robots.txt. The IETF **Content-Usage** draft and Cloudflare **Content-Signal** express content-use preferences. Support and enforcement vary by crawler. [**Generate AI Rules → **](https://nuxtseo.com/tools/robots-txt-generator) #### **IETF Content-Usage ** A draft vocabulary for machine-readable content-use preferences. `Content-Usage: search=y, train-ai=n` #### **Cloudflare Content-Signal ** A preference signal for search, AI input, and model training use. `Content-Signal: ai-train=no` ## Common questions
What does a robots.txt validator check? A robots.txt validator parses your file for syntax issues and common mistakes in User-agent, Allow, Disallow, and Sitemap directives. This checker also reports Content-Usage and Content-Signal preferences when present.
How do I know if my robots.txt is blocking AI bots? Enter your site URL, then review the matched robots.txt rule for each listed crawler. A blocked result reports what the file requests; each crawler decides how it handles that request. Content-use preferences are shown separately.
What is the robots.txt syntax? Each rule block starts with User-agent: followed by the bot name (\* for all). Then add Disallow: /path/ to block or Allow: /path/ to permit access. Rules are case-sensitive for paths. Add Sitemap: URL at the end to reference your sitemap. Lines starting with # are comments.
Why is Google still crawling pages I blocked? Google generally caches robots.txt for up to 24 hours. Fetch errors can extend caching, so live rules may differ from its cached copy. A Disallow rule controls crawling, but does not prevent indexing from external links. Use authentication to prevent access. To keep a crawlable page out of search, use noindex.
How do I test if a specific URL is blocked? After validating your robots.txt, use the path tester feature to check any URL against your rules. Enter a path like /admin/ and select a user-agent to see if it would be blocked or allowed based on your current rules.
Is a robots.txt checker the same as a robots.txt validator? The terms overlap. This checker parses common robots.txt syntax, reports issues it recognizes, and lets you test one crawler and path against the parsed rules.
Validated your robots.txt. Nuxt SEO Pro shows which pages are actually crawled and indexed, with Search Console built in. [**Monitor in Pro **](https://nuxtseo.com/pro) Part of [**Nuxt SEO **](https://nuxtseo.com/). One install covers sitemaps, robots.txt, meta tags, Schema.org, and OG images. ### **References** [**Robots Exclusion Protocol** The original standard for robots.txt.](https://www.robotstxt.org/orig.html) [**Google: Robots.txt Specification** Google's implementation and extensions.](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) [**IETF: AI Preferences Vocab** New proposal for machine-readable AI signals.](https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/) [**Cloudflare: Content Signals** Cloudflare's AI bot control signals.](https://blog.cloudflare.com/content-signals-policy/) ## Learn More [

**Robots.txt Generator**

Create a valid robots.txt file with AI bot blocking presets.](https://nuxtseo.com/tools/robots-txt-generator) [

**Robots.txt Guide**

Complete guide to robots.txt syntax, common patterns, and security.](https://nuxtseo.com/learn-seo/vue/controlling-crawlers/robots-txt) [

**Robots.txt Module**

Generate robots.txt automatically in your Nuxt application.](https://nuxtseo.com/docs/robots/getting-started/introduction) [

**Meta Tag Checker**

Check your page-level meta robots and Open Graph tags.](https://nuxtseo.com/tools/meta-tag-checker) ```json {"@context":"https://schema.org","@type":"FAQPage","mainEntity":[{"@type":"Question","name":"What does a robots.txt validator check?","acceptedAnswer":{"@type":"Answer","text":"A robots.txt validator parses your file for syntax issues and common mistakes in User-agent, Allow, Disallow, and Sitemap directives. This checker also reports Content-Usage and Content-Signal preferences when present."}},{"@type":"Question","name":"How do I know if my robots.txt is blocking AI bots?","acceptedAnswer":{"@type":"Answer","text":"Enter your site URL, then review the matched robots.txt rule for each listed crawler. A blocked result reports what the file requests; each crawler decides how it handles that request. Content-use preferences are shown separately."}},{"@type":"Question","name":"What is the robots.txt syntax?","acceptedAnswer":{"@type":"Answer","text":"Each rule block starts with User-agent: followed by the bot name (* for all). Then add Disallow: /path/ to block or Allow: /path/ to permit access. Rules are case-sensitive for paths. Add Sitemap: URL at the end to reference your sitemap. Lines starting with # are comments."}},{"@type":"Question","name":"Why is Google still crawling pages I blocked?","acceptedAnswer":{"@type":"Answer","text":"Google generally caches robots.txt for up to 24 hours. Fetch errors can extend caching, so live rules may differ from its cached copy. A Disallow rule controls crawling, but does not prevent indexing from external links. Use authentication to prevent access. To keep a crawlable page out of search, use noindex."}},{"@type":"Question","name":"How do I test if a specific URL is blocked?","acceptedAnswer":{"@type":"Answer","text":"After validating your robots.txt, use the path tester feature to check any URL against your rules. Enter a path like /admin/ and select a user-agent to see if it would be blocked or allowed based on your current rules."}},{"@type":"Question","name":"Is a robots.txt checker the same as a robots.txt validator?","acceptedAnswer":{"@type":"Answer","text":"The terms overlap. This checker parses common robots.txt syntax, reports issues it recognizes, and lets you test one crawler and path against the parsed rules."}}]} ``` ```json {"@context":"https://schema.org","@graph":[{"@id":"https://nuxtseo.com/#website","@type":"WebSite","description":"Nuxt SEO is a collection of hand-crafted Nuxt Modules to help you rank higher in search engines.","name":"Nuxt SEO","url":"https://nuxtseo.com/","publisher":{"@id":"https://nuxtseo.com/#identity"}},{"@id":"https://nuxtseo.com/tools/robots-txt-validator#webpage","@type":"WebPage","description":"Check robots.txt syntax, test one crawler and path, and review AI crawler preferences declared in the file.","name":"Robots.txt Validator · Syntax & Crawler Checker","url":"https://nuxtseo.com/tools/robots-txt-validator","isPartOf":{"@id":"https://nuxtseo.com/#website"},"potentialAction":[{"@type":"ReadAction","target":["https://nuxtseo.com/tools/robots-txt-validator"]}],"primaryImageOfPage":{"@id":"https://nuxtseo.com/#logo"}},{"@id":"https://nuxtseo.com/#identity","@type":"Organization","name":"Nuxt SEO","url":"https://nuxtseo.com","logo":{"@id":"https://nuxtseo.com/#logo"}},{"@id":"https://nuxtseo.com/tools/robots-txt-validator#/schema/software-application/1","@type":"SoftwareApplication","applicationCategory":"DeveloperApplication","description":"Check robots.txt syntax, test one crawler and path, and review AI crawler preferences declared in the file.","name":"Robots.txt Validator · Syntax & Crawler Checker","operatingSystem":"Web","offers":{"@type":"Offer","price":"0","priceCurrency":"USD"}},{"@id":"https://nuxtseo.com/#logo","@type":"ImageObject","caption":"Nuxt SEO","contentUrl":"https://nuxtseo.com/logo.png","inLanguage":"en","url":"https://nuxtseo.com/logo.png"}]} ``` ## Sitemap See the full [sitemap](https://nuxtseo.com/sitemap.md) for all pages.