---
title: "Control Web Crawlers and Crawl Budget in Nuxt"
description: "Manage how search engines crawl and index your Nuxt app. Configure robots.txt, sitemaps, canonical URLs, and redirects for better SEO."
canonical_url: "https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers"
last_updated: "2026-01-29"
---

Web crawlers determine what gets indexed and how often. Controlling them affects your [crawl budget](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget): the number of pages Google will crawl on your site in a given timeframe.

In 2026, crawler control extends to **AI Bot Governance** and the **Agentic Web**. You need to guide LLMs to your best content while protecting your data from unauthorized training.

Most sites don't need to worry about crawl budget. But if you have 10,000+ pages, frequently updated content, or want to manage how AI agents consume your data, crawler control matters.

## Types of Crawlers

**Search engines**: Index your pages for search results

- [Googlebot](https://developers.google.com/search/docs/advanced/crawling/overview-google-crawlers) (28% of bot traffic)
- [Bingbot](https://ahrefs.com/seo/glossary/bingbot)

**Social platforms**: Generate link previews when shared

- [FacebookExternalHit](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)
- Twitterbot, Slackbot, Discordbot

**AI training & Inference**: Scrape content for model training or real-time answering

- [GPTBot](https://platform.openai.com/docs/bots/overview-of-openai-crawlers) (OpenAI training)
- [ClaudeBot](https://www.anthropic.com) (Anthropic training)
- [PerplexityBot](https://www.perplexity.ai) (Real-time AI search)
- [Google-Extended](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers) (Google's AI training)

**Agentic AI**: Bots that perform actions on behalf of users (e.g., booking, shopping)

- [OpenAI-GPT-4o](https://openai.com)
- [Claude-3.5-Sonnet](https://anthropic.com)

**Malicious**: Ignore robots.txt, spoof user agents, scan for vulnerabilities. Block these at the [firewall level](/learn-seo/nuxt/routes-and-rendering/security), not with robots.txt.

## Control Mechanisms

<table>
<thead>
  <tr>
    <th>
      Mechanism
    </th>
    
    <th>
      Use When
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <a href="/learn-seo/nuxt/controlling-crawlers/robots-txt">
        robots.txt
      </a>
    </td>
    
    <td>
      Block site sections, manage crawl budget, block AI crawlers
    </td>
  </tr>
  
  <tr>
    <td>
      <a href="/learn-seo/nuxt/controlling-crawlers/sitemaps">
        Sitemaps
      </a>
    </td>
    
    <td>
      Help crawlers discover pages, especially on large sites
    </td>
  </tr>
  
  <tr>
    <td>
      <a href="/learn-seo/nuxt/controlling-crawlers/meta-tags">
        Meta robots
      </a>
    </td>
    
    <td>
      Control indexing per page (noindex, nofollow)
    </td>
  </tr>
  
  <tr>
    <td>
      <a href="/learn-seo/nuxt/controlling-crawlers/canonical-urls">
        Canonical URLs
      </a>
    </td>
    
    <td>
      Consolidate duplicate content, handle URL parameters
    </td>
  </tr>
  
  <tr>
    <td>
      <a href="/learn-seo/nuxt/controlling-crawlers/redirects">
        Redirects
      </a>
    </td>
    
    <td>
      Preserve SEO when moving/deleting pages
    </td>
  </tr>
  
  <tr>
    <td>
      <a href="/learn-seo/nuxt/controlling-crawlers/llms-txt">
        llms.txt
      </a>
    </td>
    
    <td>
      Guide AI tools to your documentation (via nuxt-llms)
    </td>
  </tr>
  
  <tr>
    <td>
      <a href="https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag#xrobotstag" rel="nofollow">
        X-Robots-Tag
      </a>
    </td>
    
    <td>
      Control non-HTML files (PDFs, images)
    </td>
  </tr>
  
  <tr>
    <td>
      <a href="/learn-seo/nuxt/routes-and-rendering/security">
        Firewall
      </a>
    </td>
    
    <td>
      Block malicious bots at network level
    </td>
  </tr>
</tbody>
</table>

## Quick Recipes

**Block page from indexing**: [Full guide](/learn-seo/nuxt/controlling-crawlers/meta-tags)

```vue [pages/admin.vue]
<script setup lang="ts">
useSeoMeta({ robots: 'noindex, follow' })
</script>
```

**Block AI training bots**: [Full guide](/learn-seo/nuxt/controlling-crawlers/robots-txt)

```robots-txt [public/robots.txt]
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /
```

**Fix duplicate content**: [Full guide](/learn-seo/nuxt/controlling-crawlers/canonical-urls)

```vue [pages/products/[id].vue]
<script setup lang="ts">
const route = useRoute()
useHead({
  link: [{ rel: 'canonical', href: `https://mysite.com/products/${route.params.id}` }]
})
</script>
```

**Redirect moved page**: [Full guide](/learn-seo/nuxt/controlling-crawlers/redirects)

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  routeRules: {
    '/old-url': { redirect: { to: '/new-url', statusCode: 301 } }
  }
})
```

## When Crawler Control Matters

Most small sites don't need to optimize crawler behavior. But it matters when:

**Crawl budget concerns**: Sites with 10,000+ pages need Google to prioritize important content. Block low-value pages (search results, filtered products, admin areas) so crawlers focus on what matters.

**Duplicate content**: URLs like `/about` and `/about/` compete against each other. Same with `?sort=price` variations. [Canonical tags](/learn-seo/nuxt/controlling-crawlers/canonical-urls) consolidate these.

**Staging environments**: Search engines index any public site they find. Block staging/dev environments in [robots.txt](/learn-seo/nuxt/controlling-crawlers/robots-txt) to avoid duplicate content issues.

**AI training opt-out**: [GPTBot was the most-blocked crawler in 2024](https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/). Block AI training bots without affecting search rankings.

**Server costs**: Bots consume CPU. Heavy pages (maps, infinite scroll, SSR) cost money per request. Blocking unnecessary crawlers reduces load.

## Nuxt SEO Modules

Nuxt handles crawler control through dedicated modules. Install once, configure in `nuxt.config.ts`, and forget about it.

<div className="grid,grid-cols-2,gap-4">
<module-card slug="nuxt-seo">



</module-card>

<module-card slug="robots">



</module-card>

<module-card slug="sitemap">



</module-card>

<module-card slug="seo-utils">



</module-card>
</div>