---
title: "How to Set Up robots.txt in Vue · Nuxt SEO"
canonical_url: "https://nuxtseo.com/learn-seo/vue/controlling-crawlers/robots-txt"
last_updated: "2026-07-16T12:00:00.000Z"
meta:
  author: "Harlan Wilton"
  description: "Write or generate robots.txt server-side in Vue, and avoid the mistakes that block Googlebot or leak your admin paths."
  "og:description": "Write or generate robots.txt server-side in Vue, and avoid the mistakes that block Googlebot or leak your admin paths."
  "og:title": "How to Set Up robots.txt in Vue · Nuxt SEO"
---

Nuxt SEO on GitHub

# **How to Set Up robots.txt in Vue**

Write or generate robots.txt server-side in Vue, and avoid the mistakes that block Googlebot or leak your admin paths.

[Harlan Wilton](https://x.com/harlan-zw)10 mins read Published **Nov 3, 2024** Updated **Jul 16, 2026**

**What you'll learn**

- robots.txt is advisory: crawlers can ignore it, so never use it for security
- Primary uses are crawl budget optimization and blocking AI training bots
- Content-Usage and Content-Signal directives let you allow search indexing while opting out of AI training

The `**robots.txt**` file controls which parts of your site crawlers can access. [**~~Officially adopted as RFC 9309~~**](https://datatracker.ietf.org/doc/html/rfc9309) in September 2022, it's used to [**~~manage crawl budget~~**](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget) on large sites and to tell AI crawlers whether they can use your content for training or real-time answers.

[**~~Robots.txt is not a security mechanism~~**](https://developer.mozilla.org/en-US/docs/Web/Security/Practical_implementation_guides/Robots_txt): crawlers can ignore it entirely. For individual page control, use [**~~meta robots tags~~**](https://nuxtseo.com/learn-seo/vue/controlling-crawlers/meta-tags) instead.

## Quick Setup

To get started quickly with a static `**robots.txt**`, add the file in your public directory:

```dir
public/
  robots.txt
```

Add your rules:

robots.txt

```robots-txt
# Allow all crawlers
User-agent: *
Disallow:

# Optionally point to your sitemap
Sitemap: https://mysite.com/sitemap.xml
```

For environment-specific rules (e.g. blocking all crawlers in staging), generate `**robots.txt**` server-side instead:

```ts
import express from 'express'

const app = express()

app.get('/robots.txt', (req, res) => {
  const isDev = process.env.NODE_ENV !== 'production'
  const robots = isDev
    ? 'User-agent: *\nDisallow: /'
    : 'User-agent: *\nDisallow:\nSitemap: https://mysite.com/sitemap.xml'
  res.type('text/plain').send(robots)
})
```

## Robots.txt Syntax

The `**robots.txt**` file consists of directives grouped by user agent. Google [**~~uses the most specific matching rule~~**](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) based on path length:

robots.txt

```robots-txt
# Define which crawler these rules apply to
User-agent: *

# Block access to specific paths
Disallow: /admin

# Allow access to specific paths (optional, more specific than Disallow)
Allow: /admin/public

# Point to your sitemap
Sitemap: https://mysite.com/sitemap.xml
```

### User-agent

The `**User-agent**` directive specifies which crawler the rules apply to:

robots.txt

```robots-txt
# All crawlers
User-agent: *

# Just Googlebot
User-agent: Googlebot

# Multiple specific crawlers
User-agent: Googlebot
User-agent: Bingbot
Disallow: /private
```

Common crawler user agents:

- [**~~Googlebot~~**](https://developers.google.com/search/docs/advanced/crawling/overview-google-crawlers): Google's search crawler, roughly half of the AI-and-search-crawler traffic [**~~Cloudflare measured in May 2025~~**](https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/)
- [**~~Google-Extended~~**](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers): not a separate crawler, a control token that governs whether Googlebot-crawled content can train [**~~Gemini~~**](https://gemini.google.com); blocking it doesn't affect Search
- [**~~GPTBot~~**](https://developers.openai.com/api/docs/bots): OpenAI's training crawler, about 7.7% of that same May 2025 sample
- [**~~OAI-SearchBot~~**](https://developers.openai.com/api/docs/bots): OpenAI's crawler for the [**~~ChatGPT~~**](https://chatgpt.com) search index, not used for training
- [**~~ClaudeBot~~**](https://support.claude.com/en/articles/9906653-claude-bot-and-crawling): Anthropic's training crawler
- [**~~Applebot~~**](https://support.apple.com/en-us/119829): Apple's search crawler for Siri, Spotlight, and Safari
- [**~~Applebot-Extended~~**](https://support.apple.com/en-us/119829): opts your content out of training Apple's foundation models, without affecting Applebot's crawling or Apple Search visibility
- [**~~CCBot~~**](https://commoncrawl.org/ccbot): Common Crawl's dataset builder, widely blocked because its data feeds many third-party AI models
- [**~~Bingbot~~**](https://ahrefs.com/seo/glossary/bingbot): Microsoft's search crawler

Not every [**~~OpenAI~~**](https://openai.com) bot needs blocking. [**~~ChatGPT-User~~**](https://developers.openai.com/api/docs/bots) only fires when a person asks ChatGPT about your page live; it doesn't crawl proactively for training the way GPTBot does.

### Allow / Disallow

The `**Allow**` and `**Disallow**` directives control path access:

robots.txt

```robots-txt
User-agent: *
# Block all paths starting with /admin
Disallow: /admin

# Block a specific file
Disallow: /private.html

# Block files with specific extensions
Disallow: /*.pdf$

# Block URL parameters
Disallow: /*?*
```

Wildcards supported ([**~~RFC 9309~~**](https://datatracker.ietf.org/doc/html/rfc9309)):

- `*****`: matches zero or more characters
- `**$**`: matches the end of the URL
- Paths are case-sensitive and relative to domain root

### Sitemap

The `**Sitemap**` directive tells crawlers where to find your [**~~sitemap.xml~~**](https://nuxtseo.com/learn-seo/vue/controlling-crawlers/sitemaps):

robots.txt

```robots-txt
Sitemap: https://mysite.com/sitemap.xml

# Multiple sitemaps
Sitemap: https://mysite.com/products-sitemap.xml
Sitemap: https://mysite.com/blog-sitemap.xml
```

### Crawl-Delay (Non-Standard)

`**Crawl-Delay**` is not part of [**~~RFC 9309~~**](https://datatracker.ietf.org/doc/html/rfc9309). Google ignores it. Bing and Yandex support it:

robots.txt

```robots-txt
User-agent: Bingbot
Crawl-delay: 10  # seconds between requests
```

For Google, you [**~~manage crawl rate in Search Console~~**](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget).

## Security: Why robots.txt Fails

[**~~Robots.txt is not a security mechanism~~**](https://developer.mozilla.org/en-US/docs/Web/Security/Practical_implementation_guides/Robots_txt). Malicious crawlers ignore it, and listing paths in `**Disallow**` [**~~reveals their location to attackers~~**](https://www.searchenginejournal.com/robots-txt-security-risks/289719/).

A common mistake:

```robots-txt
# ❌ Advertises your admin panel location
User-agent: *
Disallow: /admin
Disallow: /wp-admin
Disallow: /api/internal
```

Never use robots.txt to hide sensitive content. Listing paths in Disallow advertises their location to attackers, and malicious bots ignore robots.txt entirely. Use authentication and proper access controls instead.

Use [**~~proper authentication~~**](https://developers.google.com/search/docs/crawling-indexing/control-what-you-share) instead. See our [**~~security guide~~**](https://nuxtseo.com/learn-seo/vue/routes-and-rendering/security) for details.

## Crawling vs Indexing

Blocking a URL in `**robots.txt**` prevents crawling but [**~~doesn't prevent indexing~~**](https://developers.google.com/search/docs/crawling-indexing/robots/intro). If other sites link to the URL, Google can still index it without crawling, showing the URL with no snippet.

To prevent indexing:

- Use [`**noindex**`**~~ meta tag~~**](https://nuxtseo.com/learn-seo/vue/controlling-crawlers/meta-tags) (requires allowing crawl)
- Use password protection or authentication
- Return 404/410 status codes

Don't block pages with `**noindex**` in `**robots.txt**`. Google can't see the tag if it can't crawl.

## Common Mistakes

### 1. Blocking JavaScript and CSS

[**~~Google needs JavaScript and CSS to render pages~~**](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics). Blocking them breaks indexing:

robots.txt

```robots-txt
# ❌ Prevents Google from rendering your Vue app
User-agent: *
Disallow: /assets/
Disallow: /*.js$
Disallow: /*.css$
```

Vue apps are JavaScript-heavy. Never block `**.js**`, `**.css**`, or `**/assets/**` from Googlebot.

### 2. Blocking Dev Sites in Production

Copy-pasting a dev `**robots.txt**` to production blocks all crawlers:

robots.txt

```robots-txt
# ❌ Accidentally left from staging
User-agent: *
Disallow: /
```

Use environment checks in your server-rendered `**robots.txt**` (see [**~~Quick Setup~~**](#quick-setup)) to avoid this.

Also easy to get wrong: confusing `**robots.txt**` with `**noindex**`. Blocking a page in `**robots.txt**` doesn't remove it from search results, robots.txt controls crawling, not indexing. Use [`**noindex**`**~~ meta tags~~**](https://nuxtseo.com/learn-seo/vue/controlling-crawlers/meta-tags) to deindex a page instead.

To confirm your robots.txt is live and correct:

1. Visit `**https://yoursite.com/robots.txt**` to confirm it loads
2. Run it through the [**~~Google Search Console robots.txt tester~~**](https://search.google.com/search-console/robots-txt), which validates syntax and tests individual URLs
3. Check server logs for a 200 status on `**/robots.txt**` to confirm crawlers can access it

## Common Patterns

### Allow Everything (Default)

```robots-txt
User-agent: *
Disallow:
```

### Block Everything

Useful for staging or development environments.

```robots-txt
User-agent: *
Disallow: /
```

See our [**~~security guide~~**](https://nuxtseo.com/learn-seo/vue/routes-and-rendering/security) for more on environment protection.

### Block AI Training Crawlers

GPTBot is one of the most blocked AI crawlers: [**~~Cloudflare's 2025 analysis~~**](https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/) found it fully disallowed by 250 domains and partially disallowed by another 62. Blocking AI training bots doesn't affect your appearance in search results.

```robots-txt
# Block AI model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: Google-Extended
User-agent: CCBot
Disallow: /
```

Be careful not to block search bots like `**OAI-SearchBot**` or `**Applebot**` (unless you want to be invisible in their search products). Blocking `**GPTBot**` is safe for search visibility; blocking `**OAI-SearchBot**` removes you from ChatGPT Search.

### AI Directives: Content-Usage & Content-Signal

Blocking user agents outright isn't always what you want. Content-Usage and Content-Signal let you express preferences about how AI systems use your content without blocking crawlers entirely:

- **[**~~Content-Usage~~**](https://ietf-wg-aipref.github.io/drafts/draft-ietf-aipref-attach.html)** (IETF AI Preferences): a robots.txt directive and HTTP header using `**y**`/ `**n**` values for `**train-ai**`
- **[**~~Content-Signal~~**](https://contentsignals.org/)** (Cloudflare): uses `**yes**`/ `**no**` values for `**search**`, `**ai-input**`, and `**ai-train**`

```robots-txt
User-agent: *
Allow: /

# IETF AI Preferences
Content-Usage: train-ai=n

# Cloudflare Content Signals
Content-Signal: search=yes, ai-input=no, ai-train=no
```

This lets crawlers access your content for search indexing while blocking AI training and RAG/grounding uses. Use both together for broader coverage.

AI directives rely on voluntary compliance: crawlers can ignore them, so combine them with User-agent blocks for stronger protection.

### Block Search, Allow Social Sharing

For private sites where you still want [**~~link previews~~**](https://nuxtseo.com/learn-seo/vue/mastering-meta/social-sharing):

```robots-txt
# Block search engines
User-agent: Googlebot
User-agent: Bingbot
Disallow: /

# Allow social link preview crawlers
User-agent: facebookexternalhit
User-agent: Twitterbot
User-agent: Slackbot
Allow: /
```

### Optimize Crawl Budget for Large Sites

If you have 10,000+ pages, [**~~block low-value URLs~~**](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget) to focus crawl budget on important content:

```robots-txt
User-agent: *
# Block internal search results
Disallow: /search?
# Block infinite scroll pagination
Disallow: /*?page=
# Block filtered/sorted product pages
Disallow: /products?*sort=
Disallow: /products?*filter=
# Block print versions
Disallow: /*/print
```

Sites under 1,000 pages don't need crawl budget optimization.

## Checklist

**Checklist**

- robots.txt lives at your domain root and returns 200
- Sitemap directive points to your sitemap.xml
- JavaScript, CSS, and `**/assets/**` stay unblocked for Googlebot
- Staging and dev environments block all crawlers automatically
- Sensitive paths use authentication, not `**Disallow**`, to stay hidden
- AI training bots are blocked with User-agent rules or Content-Usage/Content-Signal directives

Building with Nuxt instead? The [**~~Nuxt SEO module~~**](https://nuxtseo.com/docs/nuxt-seo/getting-started/introduction) handles most of this automatically. [**~~Robots.txt in Nuxt →~~**](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/robots-txt)

[**The 2026 SEO Checklist for Nuxt & Vue ** Pre-launch setup, post-launch verification, and ongoing monitoring. Interactive checklist with links to every guide.](https://nuxtseo.com/learn-seo/checklist) [Haven't launched yet? Start with the **Pre-Launch Warmup**](https://nuxtseo.com/learn-seo/pre-launch-warmup)

---

### **Related **

[**Meta Robots Tags**](https://nuxtseo.com/learn-seo/vue/controlling-crawlers/meta-tags)

[**XML Sitemaps**](https://nuxtseo.com/learn-seo/vue/controlling-crawlers/sitemaps)

[**Social Sharing Tags**](https://nuxtseo.com/learn-seo/vue/mastering-meta/social-sharing)

[**Robots.txt in Nuxt**](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/robots-txt)

[**Controlling Crawlers** Configure robots.txt, sitemaps, canonicals, and redirects in Vue to control crawl budget and block unwanted AI bots.](https://nuxtseo.com/learn-seo/vue/controlling-crawlers) [**Sitemaps** Generate an XML sitemap for a Vue app with a Vite plugin, server-side route, or build script, then get it into Search Console.](https://nuxtseo.com/learn-seo/vue/controlling-crawlers/sitemaps)

**On this page**

- [Quick Setup](#quick-setup)
- [Robots.txt Syntax](#robotstxt-syntax)
- [Security: Why robots.txt Fails](#security-why-robotstxt-fails)
- [Crawling vs Indexing](#crawling-vs-indexing)
- [Common Mistakes](#common-mistakes)
- [Common Patterns](#common-patterns)
- [Checklist](#checklist)