Free Robots.txt Validator

A robots.txt validator parses your file for syntax issues and common mistakes in User-agent, Allow, Disallow, and Sitemap directives. This checker also reports Content-Usage and Content-Signal preferences when present.

Enter your site URL, then review the matched robots.txt rule for each listed crawler. A blocked result reports what the file requests; each crawler decides how it handles that request. Content-use preferences are shown separately.

Each rule block starts with User-agent: followed by the bot name (* for all). Then add Disallow: /path/ to block or Allow: /path/ to permit access. Rules are case-sensitive for paths. Add Sitemap: URL at the end to reference your sitemap. Lines starting with # are comments.

Google generally caches robots.txt for up to 24 hours. Fetch errors can extend caching, so live rules may differ from its cached copy. A Disallow rule controls crawling, but does not prevent indexing from external links. Use authentication to prevent access. To keep a crawlable page out of search, use noindex.

After validating your robots.txt, use the path tester feature to check any URL against your rules. Enter a path like /admin/ and select a user-agent to see if it would be blocked or allowed based on your current rules.

Enter any URL on your site. The validator fetches /robots.txt from the root of that host, reports the syntax issues it recognizes, and lists the matched rule for common AI crawlers. To check a draft before you publish it, paste the file instead.

Google uses the most specific rule, which is the rule with the longest matching path. If an Allow rule and a Disallow rule have the same length, Google uses the Allow rule. The path tester on this page applies the same order.

Google treats a 4xx response other than 429 as if no robots.txt exists, so it crawls without restrictions. Google treats a 5xx or 429 response as if the whole site is disallowed, and later can fall back to a cached copy. Make sure /robots.txt returns 200.

Google reads the first 500 KiB of a robots.txt file and ignores the rest. Keep the file short and group rules by crawler.

The terms overlap. This checker parses common robots.txt syntax, reports issues it recognizes, and lets you test one crawler and path against the parsed rules.