---
title: "Free Robots.txt Validator · Check Syntax and Crawler Rules · Nuxt SEO"
canonical_url: "https://nuxtseo.com/tools/robots-txt-validator"
last_updated: "2026-05-05T14:01:10.900Z"
meta:
  description: "Check robots.txt syntax, test one crawler and path, and review AI crawler preferences declared in the file."
  "og:description": "Check robots.txt syntax, test one crawler and path, and review AI crawler preferences declared in the file."
  "og:title": "Free Robots.txt Validator · Check Syntax and Crawler Rules · Nuxt SEO"
---

Nuxt SEO on GitHub

$ nuxt-seo tools

# Free Robots.txt Validator

Check robots.txt syntax, test a crawler against one path, and review any AI crawler preferences.

**Site or robots.txt URL**

Try:

## How to Check Your Robots.txt

To check your site's robots.txt file, simply append `/robots.txt` to your domain (e.g., `example.com/robots.txt`). This file is the first thing search engines look for when visiting your site.

The **robots.txt validator** reports syntax issues and matched rules. Use the URL tab to fetch a live file or the Paste tab to test changes before deployment.

#### **What to look for: **

- Does it exist at the root directory?
- Is there a User-agent declaration for each block?
- Are important assets (CSS/JS) allowed?
- Is your sitemap URL included?

## Common Robots.txt Errors

#### **Missing User-agent **

Rules like `Disallow: /admin/` must be preceded by a `User-agent:` line, otherwise bots will ignore them.

#### **Relative URLs **

Sitemap directives require absolute URLs. Using `Sitemap: /sitemap.xml` is invalid and will be ignored by search engines.

#### **Blocking Resources **

Blocking `/_nuxt/` or `/assets/` can stop crawlers from loading files needed to render the page.

#### **Path Case Sensitivity **

Robots.txt paths are case-sensitive. `Disallow: /Admin/` will not block access to `/admin/`.

#### **Conflicting Rules **

Having both `Allow: /` and `Disallow: /` for the same agent can lead to unpredictable bot behavior.

## AI Bot Blocking in Robots.txt

Crawlers such as **GPTBot**, **ClaudeBot**, and **CCBot** publish user-agent tokens that sites can address in robots.txt.

The IETF **Content-Usage** draft and Cloudflare **Content-Signal** express content-use preferences. Support and enforcement vary by crawler.

[**Generate AI Rules → **](https://nuxtseo.com/tools/robots-txt-generator)

#### **IETF Content-Usage **

A draft vocabulary for machine-readable content-use preferences.

`Content-Usage: search=y, train-ai=n`

#### **Cloudflare Content-Signal **

A preference signal for search, AI input, and model training use.

`Content-Signal: ai-train=no`

## Common questions

<details>

<summary>What does a robots.txt validator check?</summary>



A robots.txt validator parses your file for syntax issues and common mistakes in User-agent, Allow, Disallow, and Sitemap directives. This checker also reports Content-Usage and Content-Signal preferences when present.

</details>

<details>

<summary>How do I know if my robots.txt is blocking AI bots?</summary>



Enter your site URL, then review the matched robots.txt rule for each listed crawler. A blocked result reports what the file requests; each crawler decides how it handles that request. Content-use preferences are shown separately.

</details>

<details>

<summary>What is the robots.txt syntax?</summary>



Each rule block starts with User-agent: followed by the bot name (\* for all). Then add Disallow: /path/ to block or Allow: /path/ to permit access. Rules are case-sensitive for paths. Add Sitemap: URL at the end to reference your sitemap. Lines starting with # are comments.

</details>

<details>

<summary>Why is Google still crawling pages I blocked?</summary>



A Disallow rule asks compliant crawlers not to fetch a path, but it does not prevent indexing from external links. Use authentication to prevent access. To keep a crawlable page out of search, use a noindex directive instead.

</details>

<details>

<summary>How do I test if a specific URL is blocked?</summary>



After validating your robots.txt, use the path tester feature to check any URL against your rules. Enter a path like /admin/ and select a user-agent to see if it would be blocked or allowed based on your current rules.

</details>

<details>

<summary>Is a robots.txt checker the same as a robots.txt validator?</summary>



The terms overlap. This checker parses common robots.txt syntax, reports issues it recognizes, and lets you test one crawler and path against the parsed rules.

</details>

Validated your robots.txt. Nuxt SEO Pro shows which pages are actually crawled and indexed, with Search Console built in.

[**Monitor in Pro **](https://nuxtseo.com/pro)

Part of [**Nuxt SEO **](https://nuxtseo.com/). One install covers sitemaps, robots.txt, meta tags, Schema.org, and OG images.

### **References**

[**Robots Exclusion Protocol** The original standard for robots.txt.](https://www.robotstxt.org/orig.html) [**Google: Robots.txt Specification** Google's implementation and extensions.](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) [**IETF: AI Preferences Vocab** New proposal for machine-readable AI signals.](https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/) [**Cloudflare: Content Signals** Cloudflare's AI bot control signals.](https://blog.cloudflare.com/content-signals-policy/)

## Learn More

[<h2>**Robots.txt Generator**</h2>Create a valid robots.txt file with AI bot blocking presets.](https://nuxtseo.com/tools/robots-txt-generator) [<h2>**Robots.txt Guide**</h2>Complete guide to robots.txt syntax, common patterns, and security.](https://nuxtseo.com/learn-seo/vue/controlling-crawlers/robots-txt) [<h2>**Robots.txt Module**</h2>Generate robots.txt automatically in your Nuxt application.](https://nuxtseo.com/docs/robots/getting-started/introduction) [<h2>**Meta Tag Checker**</h2>Check your page-level meta robots and Open Graph tags.](https://nuxtseo.com/tools/meta-tag-checker)