How to Set Up robots.txt in Vue

The robots.txt file controls which parts of your site crawlers can access. Officially adopted as RFC 9309 in September 2022, it's used to manage crawl budget on large sites and to tell AI crawlers whether they can use your content for training or real-time answers.

Robots.txt is not a security mechanism: crawlers can ignore it entirely. For individual page indexing control, use meta robots tags instead.

Quick Setup

For a static robots.txt, add the file to your public directory:

public/
  robots.txt

Add your rules:

# Allow all crawlers
User-agent: *
Disallow:

# Optionally point to your sitemap
Sitemap: https://mysite.com/sitemap.xml

For environment-specific crawl rules, generate robots.txt server-side. These Express excerpts require pnpm add express and registration before your page handler. They do not restrict access to staging:

import express from 'express'

const app = express()

app.get('/robots.txt', (req, res) => {
  const isDev = process.env.NODE_ENV !== 'production'
  const robots = isDev
    ? 'User-agent: *\nDisallow: /'
    : 'User-agent: *\nDisallow:\nSitemap: https://mysite.com/sitemap.xml'
  res.type('text/plain').send(robots)
})

// server.ts for Vite SSR
import express from 'express'

const app = express()

app.use((req, res, next) => {
  if (req.path === '/robots.txt') {
    const isDev = process.env.NODE_ENV !== 'production'
    const robots = isDev
      ? 'User-agent: *\nDisallow: /'
      : 'User-agent: *\nDisallow:\nSitemap: https://mysite.com/sitemap.xml'
    return res.type('text/plain').send(robots)
  }
  next()
})

Robots.txt Syntax

The robots.txt file consists of directives grouped by user agent. Google uses the most specific matching rule based on path length:

# Define which crawler these rules apply to
User-agent: *

# Block access to specific paths
Disallow: /admin

# Allow access to specific paths (optional, more specific than Disallow)
Allow: /admin/public

# Point to your sitemap
Sitemap: https://mysite.com/sitemap.xml

User-agent

The User-agent directive specifies which crawler the rules apply to:

# All crawlers
User-agent: *

# Just Googlebot
User-agent: Googlebot

# Multiple specific crawlers
User-agent: Googlebot
User-agent: Bingbot
Disallow: /private

Common crawler user agents:

• Googlebot: Google's search crawler, roughly half of the AI-and-search-crawler traffic Cloudflare measured in May 2025
• Google-Extended: not a separate crawler, a control token that governs whether crawled content can train or ground specified Gemini products; blocking it does not affect Google Search
• GPTBot: OpenAI's training crawler, about 7.7% of that same May 2025 sample
• OAI-SearchBot: OpenAI's search crawler for ChatGPT. Search and training controls are independent
• ClaudeBot: Anthropic's training crawler
• Applebot: Apple's search crawler for Siri, Spotlight, and Safari
• Applebot-Extended: opts your content out of training Apple's foundation models, without affecting Applebot's crawling or Apple Search visibility
• CCBot: Common Crawl's dataset builder, widely blocked because its data feeds many third-party AI models
• Bingbot: Microsoft's search crawler

Not every OpenAI bot needs blocking. ChatGPT-User can fetch a page during user-initiated ChatGPT actions; it doesn't crawl proactively for training the way GPTBot does.

Allow / Disallow

The Allow and Disallow directives control path access:

User-agent: *
# Block all paths starting with /admin
Disallow: /admin

# Block a specific file
Disallow: /private.html

# Block files with specific extensions
Disallow: /*.pdf$

# Block URL parameters
Disallow: /*?*

Wildcards supported (RFC 9309):

• *: matches zero or more characters
• $: matches the end of the URL
• Paths are case-sensitive and relative to domain root

Sitemap

The Sitemap directive tells crawlers where to find your sitemap.xml:

Sitemap: https://mysite.com/sitemap.xml

# Multiple sitemaps
Sitemap: https://mysite.com/products-sitemap.xml
Sitemap: https://mysite.com/blog-sitemap.xml

Crawl-Delay (Non-Standard)

Crawl-Delay is not part of RFC 9309. Google ignores it. Bing supports it:

User-agent: Bingbot
Crawl-delay: 10  # seconds between requests

Google retired the Search Console crawl-rate limiter. Use Google's current crawl-rate guidance when crawling overloads your server.

Security: Why robots.txt Fails

Robots.txt is not a security mechanism. Malicious crawlers ignore it, and listing paths in Disallow reveals their location.

A common mistake:

# ❌ Advertises your admin panel location
User-agent: *
Disallow: /admin
Disallow: /wp-admin
Disallow: /api/internal

Use proper authentication instead. See our security guide for details.

Crawling vs Indexing

Blocking a URL in robots.txt prevents crawling but doesn't prevent indexing. If other sites link to the URL, Google can still index it without crawling, showing the URL with no snippet.

To prevent indexing:

• Use noindex meta tag (requires allowing crawl)
• Use password protection or authentication
• Return 404/410 status codes

Don't block pages with noindex in robots.txt. Google can't see the tag if it can't crawl.

Common Mistakes

1. Blocking JavaScript and CSS

Google needs JavaScript and CSS to render pages. Blocking required resources can prevent Google from rendering your content:

# ❌ Prevents Google from rendering your Vue app
User-agent: *
Disallow: /assets/
Disallow: /*.js$
Disallow: /*.css$

Vue apps are JavaScript-heavy. Never block .js, .css, or /assets/ from Googlebot.

2. Blocking Dev Sites in Production

Copy-pasting a dev robots.txt to production blocks all crawlers:

# ❌ Accidentally left from staging
User-agent: *
Disallow: /

Use environment checks in your server-rendered robots.txt (see Quick Setup) to avoid this.

Also easy to get wrong: confusing robots.txt with noindex. Blocking a page in robots.txt doesn't remove it from search results, robots.txt controls crawling, not indexing. Use noindex meta tags to deindex a page instead.

To confirm your robots.txt is live and correct:

1. Visit https://yoursite.com/robots.txt to confirm it loads
2. Open the Search Console robots.txt report to inspect fetch status and parsing errors. Use URL Inspection to test whether a specific page is blocked
3. Check server logs for a 200 status on /robots.txt to confirm crawlers can access it

Google generally caches robots.txt for up to 24 hours. Fetch errors can extend caching.

A correct live file does not prove Google refreshed its cached rules. Compare the robots.txt report and response errors before assuming the fix failed. See Google Search timelines.

Common Patterns

Allow Everything (Default)

User-agent: *
Disallow:

Block Everything

Requests crawl exclusion for staging or development. Protect private environments with authentication.

User-agent: *
Disallow: /

See our security guide for more on environment protection.

Block AI Training Crawlers

GPTBot is one of the most blocked AI crawlers: Cloudflare's 2025 analysis found it fully disallowed by 250 domains and partially disallowed by another 62. Training controls and search controls are provider-specific. Configure them separately.

# Block AI model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

AI Directives: Content-Usage & Content-Signal

Blocking user agents outright isn't always what you want. Content-Usage and Content-Signal let you express preferences about how AI systems use your content without blocking crawlers entirely:

• Content-Usage (IETF AI Preferences): a robots.txt directive and HTTP header using y/n values for train-ai
• Content-Signal (Cloudflare): uses yes/no values for search, ai-input, and ai-train

User-agent: *
Allow: /

# IETF AI Preferences
Content-Usage: train-ai=n

# Cloudflare Content Signals
Content-Signal: search=yes, ai-input=no, ai-train=no

These directives express usage preferences. Content-Usage remains a work-in-progress Internet-Draft dated September 14, 2026. Neither directive authenticates users or prevents a noncompliant client from fetching content. Check provider support before relying on them.

Block Search, Allow Social Sharing

For public pages excluded from search crawling where you still want link previews:

# Block search engines
User-agent: Googlebot
User-agent: Bingbot
Disallow: /

# Allow social link preview crawlers
User-agent: facebookexternalhit
User-agent: Twitterbot
User-agent: Slackbot
Allow: /

Optimize Crawl Budget for Large Sites

For large, rapidly changing sites, block low-value URLs to focus crawl budget on important content:

User-agent: *
# Block internal search results
Disallow: /search?
# Block filtered/sorted product pages
Disallow: /products?*sort=
Disallow: /products?*filter=
# Block print versions
Disallow: /*/print

Google's crawl-budget guidance gives rough size examples, not fixed thresholds. Small sites usually need an accurate sitemap and indexing checks.

Checklist

Building with Nuxt instead? The Nuxt SEO module handles most of this automatically. Robots.txt in Nuxt →