How to Set Up robots.txt in Nuxt

The robots.txt file controls which parts of your site crawlers can access. Officially adopted as RFC 9309 in September 2022, it's used to manage crawl budget on large sites and to tell AI crawlers whether they can use your content for training or real-time answers.

Quick Setup

Static robots.txt

For simple static rules, add the file in your public directory:

public/
  robots.txt

Add your rules:

# Allow all crawlers
User-agent: *
Disallow:

# Optionally point to your sitemap
Sitemap: https://mysite.com/sitemap.xml

Server Route

For custom dynamic generation, create a server route:

export default defineEventHandler((event) => {
  const isDev = process.env.NODE_ENV !== 'production'
  const robots = isDev
    ? 'User-agent: *\nDisallow: /'
    : 'User-agent: *\nDisallow:\nSitemap: https://mysite.com/sitemap.xml'

  setHeader(event, 'Content-Type', 'text/plain')
  return robots
})

Automatic Generation with Module

Use the Nuxt Robots module for generated rules. Set the intended site indexing state explicitly:

Install the module:

npx nuxt module add robots

The module generates robots.txt. This example explicitly excludes a staging site from indexing:

export default defineNuxtConfig({
  modules: ['@nuxtjs/robots'],
  site: {
    indexable: false
  }
})

Robots.txt Syntax

The robots.txt file consists of directives grouped by user agent. Google uses the most specific matching rule based on path length:

# Define which crawler these rules apply to
User-agent: *

# Block access to specific paths
Disallow: /admin

# Allow access to specific paths (optional, more specific than Disallow)
Allow: /admin/public

# Point to your sitemap
Sitemap: https://mysite.com/sitemap.xml

User-agent

The User-agent directive specifies which crawler the rules apply to:

# One group for two specific crawlers
User-agent: Googlebot
User-agent: Bingbot
Disallow: /private

Common crawler user agents:

• Googlebot: Google's search crawler
• Google-Extended: not a separate crawler, a control token for Gemini training and grounding uses; it does not affect Google Search inclusion or ranking
• GPTBot: OpenAI's training crawler
• ClaudeBot: Anthropic's training crawler
• CCBot: Common Crawl's dataset builder, widely blocked because its data feeds many third-party AI models
• Bingbot: Microsoft's search crawler
• FacebookExternalHit: Facebook's link preview crawler

Not every OpenAI bot needs blocking. ChatGPT-User only fires when a person asks ChatGPT about your page live; it doesn't crawl proactively for training the way GPTBot does.

Allow / Disallow

The Allow and Disallow directives control path access:

User-agent: *
# Block all paths starting with /admin
Disallow: /admin

# Block a specific file
Disallow: /private.html

# Block files with specific extensions
Disallow: /*.pdf$

# Block URL parameters
Disallow: /*?*

Wildcards supported (RFC 9309):

• *: matches zero or more characters
• $: matches the end of the URL
• Paths are case-sensitive and relative to domain root

Sitemap

The Sitemap directive tells crawlers where to find your sitemap.xml:

Sitemap: https://mysite.com/sitemap.xml

# Multiple sitemaps
Sitemap: https://mysite.com/products-sitemap.xml
Sitemap: https://mysite.com/blog-sitemap.xml

The Nuxt Sitemap module automatically adds the sitemap URL to your robots.txt.

Crawl-Delay (Non-Standard)

Crawl-Delay is not part of RFC 9309. Google ignores it. Bing and Yandex support it:

User-agent: Bingbot
Crawl-delay: 10  # seconds between requests

Google manages crawl rate automatically. See current crawl-budget guidance for server overload cases.

Security: Why robots.txt Fails

Robots.txt is not a security mechanism. Malicious crawlers ignore it, and listing paths in Disallow reveals their location to attackers.

A common mistake:

# ❌ Advertises your admin panel location
User-agent: *
Disallow: /admin
Disallow: /wp-admin
Disallow: /api/internal

Use proper authentication instead. See our security guide for details.

Crawling vs Indexing

Blocking a URL in robots.txt prevents crawling but doesn't prevent indexing. If other sites link to the URL, Google can still index it without crawling, showing the URL with no snippet. Use meta robots tags for page-level indexing control.

To prevent indexing:

• Use noindex meta tag (requires allowing crawl)
• Use password protection or authentication
• Return 404/410 status codes

Don't block pages with noindex in robots.txt. Google can't see the tag if it can't crawl.

Common Mistakes

1. Blocking JavaScript and CSS

Google needs JavaScript and CSS to render pages. Blocked resources cannot render. This can hide content that depends on them, while server-rendered content remains available:

# ❌ Prevents Google from rendering your Nuxt app
User-agent: *
Disallow: /assets/
Disallow: /*.js$
Disallow: /*.css$

Nuxt apps are JavaScript-heavy. Never block .js, .css, or /assets/ from Googlebot.

2. Blocking Dev Sites in Production

Copy-pasting a dev robots.txt to production blocks all crawlers:

# ❌ Accidentally left from staging
User-agent: *
Disallow: /

Inspect the generated response on both staging and production. NODE_ENV alone does not identify your deployment environment.

Also easy to get wrong: confusing robots.txt with noindex. Blocking a page in robots.txt doesn't remove it from search results, robots.txt controls crawling, not indexing. Use noindex meta tags to deindex a page instead.

To confirm your robots.txt is live and correct:

1. Visit https://yoursite.com/robots.txt to confirm it loads
2. Inspect the rules against the robots.txt specification. Check the exact URLs you intend to allow or block.
3. Check server logs for a 200 status on /robots.txt to confirm crawlers can access it

Google generally caches robots.txt for up to 24 hours. Fetch errors can extend caching.

A correct live file does not prove Google refreshed its cached rules. Compare the robots.txt report and response errors before assuming the fix failed. See Google Search timelines.

Common Patterns

Allow Everything (Default)

User-agent: *
Disallow:

Block Everything

Useful for staging or development environments.

User-agent: *
Disallow: /

See our security guide for more on environment protection.

Block AI Training Crawlers

Block only the crawler groups whose documented use you want to exclude. This example sets crawl rules, not a universal search-ranking guarantee:

# Crawl preferences for these provider tokens
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

Google-Extended doesn't have its own crawler: it's a control token that governs whether Googlebot-crawled content trains Gemini, and blocking it doesn't affect Search visibility.

AI Directives: Content-Usage & Content-Signal

These directives express usage preferences without necessarily disallowing crawling. Content-Usage is an IETF work in progress: the inspected Internet-Draft was published September 14, 2026. It is not a finalized standard. Services must implement these preferences for them to have an effect.

• Content-Usage (IETF AI Preferences): a robots.txt directive and HTTP header using y/n values for train-ai
• Content-Signal (Cloudflare): uses yes/no values for search, ai-input, and ai-train

User-agent: *
Allow: /

# Allow AI for real-time answers, block training
Content-Usage: train-ai=n
Content-Signal: search=yes, ai-input=yes, ai-train=no

The Nuxt Robots module configures these signals programmatically in nuxt.config.ts instead of you hand-writing the directives:

export default defineNuxtConfig({
  robots: {
    groups: [{
      userAgent: '*',
      allow: '/',
      contentUsage: {
        'train-ai': 'n'
      },
      contentSignal: {
        'ai-train': 'no',
        'ai-input': 'yes',
        'search': 'yes'
      }
    }]
  }
})

Block Search, Allow Social Sharing

For public, unlisted pages where you still want link previews:

# Block search engines
User-agent: Googlebot
User-agent: Bingbot
Disallow: /

# Allow social link preview crawlers
User-agent: facebookexternalhit
User-agent: Twitterbot
User-agent: Slackbot
Allow: /

Optimize Crawl Budget for Large Sites

For large sites, decide which URL patterns have useful distinct content before restricting crawlers. Keep the primary pagination sequence crawlable.

User-agent: *
# Block internal search results
Disallow: /search?
# Block filtered/sorted product pages
Disallow: /products?*sort=
Disallow: /products?*filter=
# Block print versions
Disallow: /*/print

Most small sites can focus on useful content, crawlable links, and reliable responses before tuning crawl budget.

Checklist

Building with Vue instead? Robots.txt in Vue →