---
title: "How to Set Up robots.txt in Nuxt"
description: "Generate robots.txt automatically with the Nuxt Robots module, and avoid the mistakes that block Googlebot or leak your admin paths."
canonical_url: "https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/robots-txt"
last_updated: "2026-10-08"
---

> Index of every page on this site: [llms.txt](https://nuxtseo.com/llms.txt)

::key-takeaways
- robots.txt is advisory: crawlers can ignore it, so never rely on it for security
- Configure staging indexing settings and inspect the generated robots.txt; a production-mode build can still run on staging
- Content-Usage and Content-Signal express AI-use preferences; support and compliance depend on the service
- Include a sitemap reference and keep your search-engine rules separate from your AI-crawler rules
::

The `robots.txt` file controls which parts of your site crawlers can access. [Officially adopted as RFC 9309](https://datatracker.ietf.org/doc/html/rfc9309) in September 2022, it's used to [manage crawl budget](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget) on large sites and to tell AI crawlers whether they can use your content for training or real-time answers.

## Quick Setup

### Static robots.txt

For simple static rules, add the file in your public directory:

```dir
public/
  robots.txt
```

Add your rules:

```robots-txt [robots.txt]
# Allow all crawlers
User-agent: *
Disallow:

# Optionally point to your sitemap
Sitemap: https://mysite.com/sitemap.xml
```

### Server Route

For custom dynamic generation, create a server route:

```ts [server/routes/robots.txt.ts]
export default defineEventHandler((event) => {
  const isDev = process.env.NODE_ENV !== 'production'
  const robots = isDev
    ? 'User-agent: *\nDisallow: /'
    : 'User-agent: *\nDisallow:\nSitemap: https://mysite.com/sitemap.xml'

  setHeader(event, 'Content-Type', 'text/plain')
  return robots
})
```

### Automatic Generation with Module

Use the Nuxt Robots module for generated rules. Set the intended site indexing state explicitly:

::module-card{slug="robots" .w-1/2}
::

Install the module:

```bash
npx nuxt module add robots
```

The module generates `robots.txt`. This example explicitly excludes a staging site from indexing:

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  modules: ['@nuxtjs/robots'],
  site: {
    indexable: false
  }
})
```

## Robots.txt Syntax

The `robots.txt` file consists of directives grouped by user agent. Google [uses the most specific matching rule](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) based on path length:

```robots-txt [robots.txt]
# Define which crawler these rules apply to
User-agent: *

# Block access to specific paths
Disallow: /admin

# Allow access to specific paths (optional, more specific than Disallow)
Allow: /admin/public

# Point to your sitemap
Sitemap: https://mysite.com/sitemap.xml
```

### User-agent

The `User-agent` directive specifies which crawler the rules apply to:

```robots-txt [robots.txt]
# One group for two specific crawlers
User-agent: Googlebot
User-agent: Bingbot
Disallow: /private
```

Common crawler user agents:

- [Googlebot](https://developers.google.com/search/docs/advanced/crawling/overview-google-crawlers): Google's search crawler
- [Google-Extended](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers): not a separate crawler, a control token for [Gemini](https://gemini.google.com) training and grounding uses; it does not affect Google Search inclusion or ranking
- [GPTBot](https://developers.openai.com/api/docs/bots): OpenAI's training crawler
- [ClaudeBot](https://support.claude.com/en/articles/9906653-claude-bot-and-crawling): Anthropic's training crawler
- [CCBot](https://commoncrawl.org/ccbot): Common Crawl's dataset builder, widely blocked because its data feeds many third-party AI models
- [Bingbot](https://ahrefs.com/seo/glossary/bingbot): Microsoft's search crawler
- [FacebookExternalHit](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/): Facebook's link preview crawler

Not every [OpenAI](https://openai.com) bot needs blocking. [ChatGPT-User](https://developers.openai.com/api/docs/bots) only fires when a person asks [ChatGPT](https://chatgpt.com) about your page live; it doesn't crawl proactively for training the way GPTBot does.

### Allow / Disallow

The `Allow` and `Disallow` directives control path access:

```robots-txt [robots.txt]
User-agent: *
# Block all paths starting with /admin
Disallow: /admin

# Block a specific file
Disallow: /private.html

# Block files with specific extensions
Disallow: /*.pdf$

# Block URL parameters
Disallow: /*?*
```

Wildcards supported ([RFC 9309](https://datatracker.ietf.org/doc/html/rfc9309)):

- `*`: matches zero or more characters
- `$`: matches the end of the URL
- Paths are case-sensitive and relative to domain root

### Sitemap

The `Sitemap` directive tells crawlers where to find your [sitemap.xml](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/sitemaps):

```robots-txt [robots.txt]
Sitemap: https://mysite.com/sitemap.xml

# Multiple sitemaps
Sitemap: https://mysite.com/products-sitemap.xml
Sitemap: https://mysite.com/blog-sitemap.xml
```

The [Nuxt Sitemap module](https://nuxtseo.com/docs/sitemap/getting-started/introduction) automatically adds the sitemap URL to your `robots.txt`.

### Crawl-Delay (Non-Standard)

`Crawl-Delay` is not part of [RFC 9309](https://datatracker.ietf.org/doc/html/rfc9309). Google ignores it. Bing and Yandex support it:

```robots-txt [robots.txt]
User-agent: Bingbot
Crawl-delay: 10  # seconds between requests
```

Google manages crawl rate automatically. See [current crawl-budget guidance](https://developers.google.com/crawling/docs/crawl-budget) for server overload cases.

## Security: Why robots.txt Fails

[Robots.txt is not a security mechanism](https://developer.mozilla.org/en-US/docs/Web/Security/Practical_implementation_guides/Robots_txt). Malicious crawlers ignore it, and listing paths in `Disallow` [reveals their location to attackers](https://www.searchenginejournal.com/robots-txt-security-risks/289719/).

A common mistake:

```robots-txt
# ❌ Advertises your admin panel location
User-agent: *
Disallow: /admin
Disallow: /wp-admin
Disallow: /api/internal
```

Use [proper authentication](https://developers.google.com/search/docs/crawling-indexing/control-what-you-share) instead. See our [security guide](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security) for details.

## Crawling vs Indexing

Blocking a URL in `robots.txt` prevents crawling but [doesn't prevent indexing](https://developers.google.com/search/docs/crawling-indexing/robots/intro). If other sites link to the URL, Google can still index it without crawling, showing the URL with no snippet. Use [meta robots tags](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/meta-tags) for page-level indexing control.

To prevent indexing:

- Use [`noindex` meta tag](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/meta-tags) (requires allowing crawl)
- Use password protection or authentication
- Return 404/410 status codes

Don't block pages with `noindex` in `robots.txt`. Google can't see the tag if it can't crawl.

## Common Mistakes

### 1. Blocking JavaScript and CSS

[Google needs JavaScript and CSS to render pages](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics). Blocked resources cannot render. This can hide content that depends on them, while server-rendered content remains available:

```robots-txt [robots.txt]
# ❌ Prevents Google from rendering your Nuxt app
User-agent: *
Disallow: /assets/
Disallow: /*.js$
Disallow: /*.css$
```

Nuxt apps are JavaScript-heavy. Never block `.js`, `.css`, or `/assets/` from Googlebot.

### 2. Blocking Dev Sites in Production

Copy-pasting a dev `robots.txt` to production blocks all crawlers:

```robots-txt [robots.txt]
# ❌ Accidentally left from staging
User-agent: *
Disallow: /
```

Inspect the generated response on both staging and production. NODE_ENV alone does not identify your deployment environment.

Also easy to get wrong: confusing `robots.txt` with `noindex`. Blocking a page in `robots.txt` doesn't remove it from search results, robots.txt controls crawling, not indexing. Use [`noindex` meta tags](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/meta-tags) to deindex a page instead.

To confirm your robots.txt is live and correct:

1. Visit `https://yoursite.com/robots.txt` to confirm it loads
2. Inspect the rules against the [robots.txt specification](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt). Check the exact URLs you intend to allow or block.
3. Check server logs for a 200 status on `/robots.txt` to confirm crawlers can access it

Google [generally caches robots.txt for up to 24 hours](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec). Fetch errors can extend caching.

A correct live file does not prove Google refreshed its cached rules. Compare the robots.txt report and response errors before assuming the fix failed. See [Google Search timelines](https://nuxtseo.com/learn-seo/google-search-timelines).

## Common Patterns

### Allow Everything (Default)

```robots-txt
User-agent: *
Disallow:
```

### Block Everything

Useful for staging or development environments.

```robots-txt
User-agent: *
Disallow: /
```

See our [security guide](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security) for more on environment protection.

### Block AI Training Crawlers

Block only the crawler groups whose documented use you want to exclude. This example sets crawl rules, not a universal search-ranking guarantee:

```robots-txt
# Crawl preferences for these provider tokens
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /
```

`Google-Extended` doesn't have its own crawler: it's a control token that governs whether Googlebot-crawled content trains Gemini, and blocking it doesn't affect Search visibility.

### AI Directives: Content-Usage & Content-Signal

These directives express usage preferences without necessarily disallowing crawling. Content-Usage is an IETF work in progress: the inspected Internet-Draft was published September 14, 2026. It is not a finalized standard. Services must implement these preferences for them to have an effect.

- **[Content-Usage](https://ietf-wg-aipref.github.io/drafts/draft-ietf-aipref-attach.html)** (IETF AI Preferences): a robots.txt directive and HTTP header using `y`/`n` values for `train-ai`
- **[Content-Signal](https://contentsignals.org/)** (Cloudflare): uses `yes`/`no` values for `search`, `ai-input`, and `ai-train`

```robots-txt [robots.txt]
User-agent: *
Allow: /

# Allow AI for real-time answers, block training
Content-Usage: train-ai=n
Content-Signal: search=yes, ai-input=yes, ai-train=no
```

The [Nuxt Robots module](https://nuxtseo.com/docs/robots/guides/ai-directives) configures these signals programmatically in `nuxt.config.ts` instead of you hand-writing the directives:

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  robots: {
    groups: [{
      userAgent: '*',
      allow: '/',
      contentUsage: {
        'train-ai': 'n'
      },
      contentSignal: {
        'ai-train': 'no',
        'ai-input': 'yes',
        'search': 'yes'
      }
    }]
  }
})
```

::tip
These preferences apply only to services that implement them. They do not guarantee citations or exclusion from AI answers. See [AI-optimized content](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen/ai-optimized-content) for the limits.
::

::warning
Usage preferences and User-agent crawl restrictions both rely on provider compliance. Neither authenticates visitors or stops malicious scraping.
::

### Block Search, Allow Social Sharing

For public, unlisted pages where you still want [link previews](https://nuxtseo.com/learn-seo/nuxt/mastering-meta/open-graph):

```robots-txt
# Block search engines
User-agent: Googlebot
User-agent: Bingbot
Disallow: /

# Allow social link preview crawlers
User-agent: facebookexternalhit
User-agent: Twitterbot
User-agent: Slackbot
Allow: /
```

### Optimize Crawl Budget for Large Sites

For large sites, decide which URL patterns have useful distinct content before restricting crawlers. Keep the primary pagination sequence crawlable.

```robots-txt
User-agent: *
# Block internal search results
Disallow: /search?
# Block filtered/sorted product pages
Disallow: /products?*sort=
Disallow: /products?*filter=
# Block print versions
Disallow: /*/print
```

Most small sites can focus on useful content, crawlable links, and reliable responses before tuning crawl budget.

## Checklist

::checklist{#nuxt-robots-txt}
- robots.txt lives at your domain root and returns 200
- Sitemap directive points to your sitemap.xml
- JavaScript, CSS, and `/assets/` stay unblocked for Googlebot
- Staging indexing settings are explicit and the generated robots.txt is checked
- Sensitive paths use authentication, not `Disallow`, to stay hidden
- Set provider-specific voluntary crawl or usage preferences; verify their documented scope
::

Building with Vue instead? [Robots.txt in Vue →](https://nuxtseo.com/learn-seo/vue/controlling-crawlers/robots-txt)

## Sitemap

See the full [sitemap](https://nuxtseo.com/sitemap.md) for all pages.