Robots.txt Recipes · Nuxt Robots · Nuxt SEO

[NuxtSEO](https://nuxtseo.com/ "Home")

- [Modules](https://nuxtseo.com/docs/nuxt-seo/getting-started/introduction)
- [Tools](https://nuxtseo.com/tools)
- [Pro](https://nuxtseo.com/pro)
- [Learn SEO](https://nuxtseo.com/learn-seo/nuxt) [Releases](https://nuxtseo.com/releases)

[1.4K](https://github.com/harlan-zw/nuxt-seo)

[Nuxt SEO on GitHub](https://github.com/harlan-zw/nuxt-seo)

[User Guides](https://nuxtseo.com/docs/robots/getting-started/introduction)

[API](https://nuxtseo.com/docs/robots/api/use-robots-rule)

[Releases](https://nuxtseo.com/docs/robots/releases/v6)

Robots

- [Switch to Robots](https://nuxtseo.com/docs/robots/getting-started/introduction)
- [Switch to Nuxt SEO](https://nuxtseo.com/docs/nuxt-seo/getting-started/introduction)
- [Switch to Sitemap](https://nuxtseo.com/docs/sitemap/getting-started/introduction)
- [Switch to OG Image](https://nuxtseo.com/docs/og-image/getting-started/introduction)
- [Switch to Schema.org](https://nuxtseo.com/docs/schema-org/getting-started/introduction)
- [Switch to Link Checker](https://nuxtseo.com/docs/link-checker/getting-started/introduction)
- [Switch to SEO Utils](https://nuxtseo.com/docs/seo-utils/getting-started/introduction)
- [Switch to Site Config](https://nuxtseo.com/docs/site-config/getting-started/introduction)
- [Switch to Skew Protection](https://nuxtseo.com/docs/skew-protection/getting-started/introduction)
- [Switch to AI Ready](https://nuxtseo.com/docs/ai-ready/getting-started/introduction)

Search…```k`` /`

v6.0.7

- Playgrounds
- [Discord Support](https://discord.com/invite/275MBUBvgP)

### Getting Started

- [Introduction](https://nuxtseo.com/docs/robots/getting-started/introduction)
- [Installation](https://nuxtseo.com/docs/robots/getting-started/installation)
- [Troubleshooting](https://nuxtseo.com/docs/robots/getting-started/troubleshooting)

### Core Concepts

- [Disabling Site Indexing](https://nuxtseo.com/docs/robots/guides/disable-indexing)
- [Disable Page Indexing](https://nuxtseo.com/docs/robots/guides/disable-page-indexing)
- [How Nuxt Robots Works](https://nuxtseo.com/docs/robots/guides/how-it-works)
- [Robots.txt Recipes](https://nuxtseo.com/docs/robots/guides/robot-recipes)
- [Config using Robots.txt](https://nuxtseo.com/docs/robots/guides/robots-txt)
- [Config using Nuxt Config](https://nuxtseo.com/docs/robots/guides/nuxt-config)
- [Config Using Route Rules](https://nuxtseo.com/docs/robots/guides/route-rules)
- [Bot Detection](https://nuxtseo.com/docs/robots/guides/bot-detection)
- [AI Directives](https://nuxtseo.com/docs/robots/guides/ai-directives)

### Advanced

- [Yandex: Clean-param](https://nuxtseo.com/docs/robots/advanced/yandex)
- [Nuxt Content](https://nuxtseo.com/docs/robots/advanced/content)
- [Nuxt I18n](https://nuxtseo.com/docs/robots/advanced/i18n)

Core Concepts

# Robots.txt Recipes

[Copy for LLMs](https://nuxtseo.com/docs/robots/guides/robot-recipes.md)

## [Introduction](#introduction)

As a minimum the only recommended configuration for robots is to [disable indexing for non-production environments](https://nuxtseo.com/docs/robots/guides/disable-indexing).

Many sites will never need to configure their [`robots.txt`](https://nuxtseo.com/learn/controlling-crawlers/robots-txt) or [`robots` meta tag](https://nuxtseo.com/learn/controlling-crawlers/meta-tags) beyond this, as the [controlling web crawlers](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers) is an advanced use case and topic.

However, if you're looking to get the best SEO and performance results, you may consider some of the recipes on this page for your site.

## [Robots.txt recipes](#robotstxt-recipes)

### [Blocking Bad Bots](#blocking-bad-bots)

If you're finding your site is getting hit with a lot of bots, you may consider enabling the `blockNonSeoBots` option.

nuxt.config.ts

```
export default defineNuxtConfig({
  robots: {
    blockNonSeoBots: true
  }
})
```

This will block mostly web scrapers, the full list is: `Nuclei`, `WikiDo`, `Riddler`, `PetalBot`, `Zoominfobot`, `Go-http-client`, `Node/simplecrawler`, `CazoodleBot`, `dotbot/1.0`, `Gigabot`, `Barkrowler`, `BLEXBot`, `magpie-crawler`.

### [Blocking AI Crawlers](#blocking-ai-crawlers)

AI crawlers can be beneficial as they can help users finding your site, but for some educational sites or those not interested in being indexed by AI crawlers, you can block them using the `blockAIBots` option.

nuxt.config.ts

```
export default defineNuxtConfig({
  robots: {
    blockAiBots: true
  }
})
```

This will block the following AI crawlers: `GPTBot`, `ChatGPT-User`, `Claude-Web`, `anthropic-ai`, `Applebot-Extended`, `Bytespider`, `CCBot`, `cohere-ai`, `Diffbot`, `FacebookBot`, `Google-Extended`, `ImagesiftBot`, `PerplexityBot`, `OmigiliBot`, `Omigili`

### [Blocking Privileged Pages](#blocking-privileged-pages)

If you have pages that require authentication or are only available to certain users, you should block these from being indexed.

public/_robots.txt

```
User-agent: *
Disallow: /admin
Disallow: /dashboard
```

See [Config using Robots.txt](https://nuxtseo.com/docs/robots/guides/robots-txt) for more information.

### [Whitelisting Open Graph Tags](#whitelisting-open-graph-tags)

If you have certain pages that you don't want indexed but you still want their [Open Graph Tags](https://nuxtseo.com/learn-seo/nuxt/mastering-meta/social-sharing) to be crawled, you can target the specific user-agents.

public/_robots.txt

```
# Block search engines
User-agent: Googlebot
User-agent: Bingbot
Disallow: /user-profiles

# Allow social crawlers
User-agent: facebookexternalhit
User-agent: Twitterbot
Allow: /user-profiles
```

See [Config using Robots.txt](https://nuxtseo.com/docs/robots/guides/robots-txt) for more information.

### [Blocking Search Results](#blocking-search-results)

You may consider blocking search results from being indexed, as they can be seen as duplicate content and can be a poor user experience.

public/_robots.txt

```
User-agent: *
# block search results
Disallow: /*?query=
# block pagination
Disallow: /*?page=
# block sorting
Disallow: /*?sort=
# block filtering
Disallow: /*?filter=
```

[Edit this page](https://github.com/nuxt-modules/robots/edit/main/docs/content/2.guides/1.robot-recipes.md)

[Markdown For LLMs](https://nuxtseo.com/docs/robots/guides/robot-recipes.md)

Did this page help you?

[How Nuxt Robots Works Learn more about how Nuxt Robots works.](https://nuxtseo.com/docs/robots/guides/how-it-works) [Config using Robots.txt Configure your generated robots.txt file with a robots.txt file.](https://nuxtseo.com/docs/robots/guides/robots-txt)

On this page

- [Introduction](#introduction)
- [Robots.txt recipes](#robotstxt-recipes)

[GitHub](https://github.com/harlan-zw/nuxt-seo) [ Discord](https://discord.com/invite/275MBUBvgP)

### [NuxtSEO](https://nuxtseo.com/ "Home")

- [Getting Started](https://nuxtseo.com/docs/nuxt-seo/getting-started/introduction)
- [MCP](https://nuxtseo.com/docs/nuxt-seo/guides/mcp)

Modules

- [Robots](https://nuxtseo.com/docs/robots/getting-started/introduction)
- [Sitemap](https://nuxtseo.com/docs/sitemap/getting-started/introduction)
- [OG Image](https://nuxtseo.com/docs/og-image/getting-started/introduction)
- [Schema.org](https://nuxtseo.com/docs/schema-org/getting-started/introduction)
- [Link Checker](https://nuxtseo.com/docs/link-checker/getting-started/introduction)
- [SEO Utils](https://nuxtseo.com/docs/seo-utils/getting-started/introduction)
- [Site Config](https://nuxtseo.com/docs/site-config/getting-started/introduction)
- [Skew Protection](https://nuxtseo.com/docs/skew-protection/getting-started/introduction)
- [AI Ready](https://nuxtseo.com/docs/ai-ready/getting-started/introduction)

### [NuxtSEO Pro](https://nuxtseo.com/pro "Home")

- [Getting Started](https://nuxtseo.com/pro)
- [Dashboard](https://nuxtseo.com/pro/dashboard)
- [Pro MCP](https://nuxtseo.com/docs/nuxt-seo-pro/mcp/installation)

### [Learn SEO](https://nuxtseo.com/learn-seo "Learn SEO")

Nuxt

- [Mastering Meta](https://nuxtseo.com/learn-seo/nuxt/mastering-meta)
- [Controlling Crawlers](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers)
- [Launch & Listen](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen)
- [Routes & Rendering](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering)
- [Staying Secure](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security)

Vue

- [Vue SEO Guide](https://nuxtseo.com/learn-seo/vue)
- [Mastering Meta](https://nuxtseo.com/learn-seo/vue/mastering-meta)
- [Controlling Crawlers](https://nuxtseo.com/learn-seo/vue/controlling-crawlers)
- [SPA SEO](https://nuxtseo.com/learn-seo/vue/spa)
- [SSR Frameworks](https://nuxtseo.com/learn-seo/vue/ssr-frameworks)
- [SEO Checklist](https://nuxtseo.com/learn-seo/checklist)
- [Pre-Launch Warmup](https://nuxtseo.com/learn-seo/pre-launch-warmup)
- [Backlinks & Authority](https://nuxtseo.com/learn-seo/backlinks)

### [Tools](https://nuxtseo.com/tools "SEO Tools")

- [Social Share Debugger](https://nuxtseo.com/tools/social-share-debugger)
- [Robots.txt Generator](https://nuxtseo.com/tools/robots-txt-generator)
- [Meta Tag Checker](https://nuxtseo.com/tools/meta-tag-checker)
- [HTML to Markdown](https://nuxtseo.com/tools/html-to-markdown)
- [XML Sitemap Validator](https://nuxtseo.com/tools/xml-sitemap-validator)
- [Schema.org Validator](https://nuxtseo.com/tools/schema-validator)
- [Keyword Idea Generator](https://nuxtseo.com/tools/keyword-generator)
- [Keyword Research](https://nuxtseo.com/tools/keyword-research)
- [SERP Analyzer](https://nuxtseo.com/tools/serp-analyzer)
- [Domain Rankings](https://nuxtseo.com/tools/domain-rankings)

Copyright © 2023-2026 Harlan Wilton - [MIT License](https://github.com/harlan-zw/nuxt-seo/blob/main/license) · [mdream](https://mdream.dev)