Free Robots.txt Validator

A robots.txt validator parses your file for syntax issues and common mistakes in User-agent, Allow, Disallow, and Sitemap directives. This checker also reports Content-Usage and Content-Signal preferences when present.

Enter your site URL, then review the matched robots.txt rule for each listed crawler. A blocked result reports what the file requests; each crawler decides how it handles that request. Content-use preferences are shown separately.

Each rule block starts with User-agent: followed by the bot name (* for all). Then add Disallow: /path/ to block or Allow: /path/ to permit access. Rules are case-sensitive for paths. Add Sitemap: URL at the end to reference your sitemap. Lines starting with # are comments.

Google generally caches robots.txt for up to 24 hours. Fetch errors can extend caching, so live rules may differ from its cached copy. A Disallow rule controls crawling, but does not prevent indexing from external links. Use authentication to prevent access. To keep a crawlable page out of search, use noindex.

After validating your robots.txt, use the path tester feature to check any URL against your rules. Enter a path like /admin/ and select a user-agent to see if it would be blocked or allowed based on your current rules.

The terms overlap. This checker parses common robots.txt syntax, reports issues it recognizes, and lets you test one crawler and path against the parsed rules.