---
title: "Control Web Crawlers and Crawl Budget in Nuxt"
description: "Configure robots.txt, sitemaps, canonicals, and redirects in Nuxt to control crawl budget and block unwanted AI bots."
canonical_url: "https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers"
last_updated: "2026-10-05"
---

> Index of every page on this site: [llms.txt](https://nuxtseo.com/llms.txt)

Web crawlers determine what gets indexed and how often. Controlling them affects your [crawl budget](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget): the set of URLs Google can and wants to crawl on your site. It also lets you state crawl preferences for AI bots that honor robots.txt. Use authentication to protect private content.

Most sites do not need crawl-budget work. Google's [rough scope](https://developers.google.com/crawling/docs/crawl-budget) includes very large sites and sites with 10,000+ pages that change daily. AI crawl preferences are a separate decision.

## Types of Crawlers

**Search engines**: Index your pages for search results

- [Googlebot](https://developers.google.com/search/docs/advanced/crawling/overview-google-crawlers)
- [Bingbot](https://ahrefs.com/seo/glossary/bingbot)
- [Applebot](https://support.apple.com/en-us/119829) (Spotlight and Siri suggestions)

**Social platforms**: Generate link previews when shared

- [FacebookExternalHit](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)
- Twitterbot, Slackbot, Discordbot

**AI training**: Scrape content to train models

- [GPTBot](https://developers.openai.com/api/docs/bots) (OpenAI)
- [ClaudeBot](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (Anthropic)
- [Google-Extended](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers) is a robots.txt control token, not a separate crawler user agent
- [Applebot-Extended](https://support.apple.com/en-us/119829) is a robots.txt control token for Apple's training use, not a separate crawler

**AI search and user fetches**: providers distinguish automated search crawling from user-initiated requests. Allowing a bot does not guarantee a citation.

- [OAI-SearchBot](https://developers.openai.com/api/docs/bots) supports ChatGPT Search. `ChatGPT-User` performs user-initiated fetches; it does not control Search inclusion
- [Claude-SearchBot and Claude-User](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (Claude)
- [PerplexityBot and Perplexity-User](https://docs.perplexity.ai/guides/bots) (Perplexity)

**Malicious**: Ignore robots.txt, spoof user agents, scan for vulnerabilities. Block these at the [firewall level](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security), not with robots.txt.

::tip{title="Training bots vs. search bots"}
Training bots feed your content into model training; search bots fetch it live to answer a specific query. You can block one without the other: disallow `GPTBot` in robots.txt while leaving `OAI-SearchBot` untouched if you want your site cited in ChatGPT Search.
::

## Control Mechanisms

| Mechanism                                                                                              | Use When                                                              |
| ------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------- |
| [robots.txt](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/robots-txt)                                          | Block site sections, manage crawl budget, block AI crawlers           |
| [Sitemaps](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/sitemaps)                                              | Help crawlers discover pages, especially on large sites               |
| [Meta robots](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/meta-tags)                                          | Control indexing per page (noindex, nofollow)                         |
| [Canonical URLs](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/canonical-urls)                                  | Consolidate duplicate content, handle URL parameters                  |
| [Redirects](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/redirects)                                            | Preserve SEO when moving/deleting pages                               |
| [llms.txt](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/llms-txt)                                              | Publish an optional document index; see the guide for scope and setup |
| [X-Robots-Tag](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag#xrobotstag) | Control non-HTML files (PDFs, images)                                 |
| [Firewall](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security)                                              | Block malicious bots at network level                                 |

## Quick Recipes

**Block page from indexing**: [Full guide](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/meta-tags)

```vue [pages/admin.vue]
<script setup lang="ts">
useSeoMeta({ robots: 'noindex, follow' })
</script>
```

**Block AI training bots**: [Full guide](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/robots-txt)

```robots-txt [public/robots.txt]
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /
```

**Fix duplicate content**: [Full guide](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/canonical-urls)

```vue [pages/products/[id\\].vue]
<script setup lang="ts">
const route = useRoute()
useHead(() => ({
  link: [{ rel: 'canonical', href: `https://mysite.com/products/${route.params.id}` }]
}))
</script>
```

**Redirect moved page**: [Full guide](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/redirects)

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  routeRules: {
    '/old-url': { redirect: { to: '/new-url', statusCode: 301 } }
  }
})
```

## When Crawler Control Matters

Most small sites don't need to optimize crawler behavior, but it matters when:

**Crawl budget concerns**: use Google's rough scope and your crawl evidence. A 10,000-page count alone is not a reason to block URLs. Robots.txt cannot protect private pages or guarantee deindexing.

**Duplicate content**: URLs like `/about` and `/about/` compete against each other, and so do `?sort=price` variations. [Canonical tags](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/canonical-urls) consolidate these.

**Staging environments**: protect private staging with authentication. For public pages that should leave search results, allow crawling and use noindex. robots.txt alone does not prevent indexing.

**AI training opt-out**: check the provider's policy and disallow its training crawler if appropriate. A rule affects only crawlers that honor it; it does not guarantee search rankings or block malicious scraping.

**Server costs**: Bots consume CPU, and heavy pages (maps, infinite scroll, SSR) cost money per request. Blocking unnecessary crawlers reduces load.

## Nuxt SEO Modules

Nuxt handles crawler control through dedicated modules. Install once, configure in `nuxt.config.ts`, and forget about it.

::div{.grid.grid-cols-2.gap-4}
  :::module-card{slug="nuxt-seo"}
  :::

  :::module-card{slug="robots"}
  :::

  :::module-card{slug="sitemap"}
  :::

  :::module-card{slug="seo-utils"}
  :::
::

## Sitemap

See the full [sitemap](https://nuxtseo.com/sitemap.md) for all pages.