---
title: "Control Web Crawlers and Crawl Budget in Vue"
description: "Configure robots.txt, sitemaps, canonicals, and redirects in Vue to control crawl budget and block unwanted AI bots."
canonical_url: "https://nuxtseo.com/learn-seo/vue/controlling-crawlers"
last_updated: "2026-07-16"
---

Web crawlers determine what gets indexed and how often. Controlling them affects your [crawl budget](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget): the set of URLs Google can and wants to crawl on your site. It also means guiding AI bots to your best content while blocking the ones that scrape it for training.

Most sites don't need to worry about crawl budget. But if you have 10,000+ pages, frequently updated content, or want to control how AI bots consume your data, crawler control matters.

## Types of Crawlers

**Search engines**: Index your pages for search results

- [Googlebot](https://developers.google.com/search/docs/advanced/crawling/overview-google-crawlers)
- [Bingbot](https://ahrefs.com/seo/glossary/bingbot)
- [Applebot](https://support.apple.com/en-us/119829) (Spotlight and Siri suggestions)

**Social platforms**: Generate link previews when shared

- [FacebookExternalHit](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)
- Twitterbot, Slackbot, Discordbot

**AI training**: Scrape content to train models

- [GPTBot](https://developers.openai.com/api/docs/bots) (OpenAI)
- [ClaudeBot](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (Anthropic)
- [Google-Extended](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers) (Gemini)
- [Applebot-Extended](https://support.apple.com/en-us/119829) (Apple Intelligence)

**AI search**: Fetch live content to answer a specific query. Allow these if you want your site cited

- [OAI-SearchBot and ChatGPT-User](https://developers.openai.com/api/docs/bots) (ChatGPT)
- [Claude-SearchBot and Claude-User](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (Claude)
- [PerplexityBot and Perplexity-User](https://docs.perplexity.ai/guides/bots) (Perplexity)

**Malicious**: Ignore robots.txt, spoof user agents, scan for vulnerabilities. Block these at the [firewall level](/learn-seo/vue/routes-and-rendering/security), not with robots.txt.

<tip title="Training bots vs. search bots">

Training bots feed your content into model training; search bots fetch it live to answer a specific query. You can block one without the other: disallow `GPTBot` in robots.txt while leaving `OAI-SearchBot` untouched if you want your site cited in ChatGPT Search.

</tip>

## Control Mechanisms

<table>
<thead>
  <tr>
    <th>
      Mechanism
    </th>
    
    <th>
      Use When
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <a href="/learn-seo/vue/controlling-crawlers/robots-txt">
        robots.txt
      </a>
    </td>
    
    <td>
      Block site sections, manage crawl budget, block AI bots
    </td>
  </tr>
  
  <tr>
    <td>
      <a href="/learn-seo/vue/controlling-crawlers/sitemaps">
        Sitemaps
      </a>
    </td>
    
    <td>
      Help crawlers discover pages, especially on large sites
    </td>
  </tr>
  
  <tr>
    <td>
      <a href="/learn-seo/vue/controlling-crawlers/meta-tags">
        Meta robots
      </a>
    </td>
    
    <td>
      Control indexing per page (noindex, nofollow)
    </td>
  </tr>
  
  <tr>
    <td>
      <a href="/learn-seo/vue/controlling-crawlers/canonical-urls">
        Canonical URLs
      </a>
    </td>
    
    <td>
      Consolidate duplicate content, handle URL parameters
    </td>
  </tr>
  
  <tr>
    <td>
      <a href="/learn-seo/vue/controlling-crawlers/redirects">
        Redirects
      </a>
    </td>
    
    <td>
      Preserve SEO when moving/deleting pages
    </td>
  </tr>
  
  <tr>
    <td>
      <a href="/learn-seo/vue/controlling-crawlers/llms-txt">
        llms.txt
      </a>
    </td>
    
    <td>
      Guide AI tools to your documentation (MCP servers, coding assistants)
    </td>
  </tr>
  
  <tr>
    <td>
      <a href="https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag#xrobotstag" rel="nofollow">
        X-Robots-Tag
      </a>
    </td>
    
    <td>
      Control non-HTML files (PDFs, images)
    </td>
  </tr>
  
  <tr>
    <td>
      <a href="/learn-seo/vue/routes-and-rendering/security">
        Firewall
      </a>
    </td>
    
    <td>
      Block malicious bots at network level
    </td>
  </tr>
</tbody>
</table>

## Quick Recipes

**Block page from indexing**: [Full guide](/learn-seo/vue/controlling-crawlers/meta-tags)

```vue [pages/admin.vue]
<script setup lang="ts">
import { useSeoMeta } from '@unhead/vue'

useSeoMeta({ robots: 'noindex, follow' })
</script>
```

**Block AI training bots**: [Full guide](/learn-seo/vue/controlling-crawlers/robots-txt)

```robots-txt [public/robots.txt]
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: Google-Extended
Disallow: /
```

**Fix duplicate content**: [Full guide](/learn-seo/vue/controlling-crawlers/canonical-urls)

```vue [pages/products/[id].vue]
<script setup lang="ts">
import { useHead } from '@unhead/vue'
import { useRoute } from 'vue-router'

const route = useRoute()

useHead({
  link: [{ rel: 'canonical', href: `https://mysite.com/products/${route.params.id}` }]
})
</script>
```

**Redirect moved page**: [Full guide](/learn-seo/vue/controlling-crawlers/redirects)

```ts [server.ts]
app.get('/old-url', (req, res) => res.redirect(301, '/new-url'))
```

## When Crawler Control Matters

Most small sites don't need to optimize crawler behavior, but it matters when:

**Crawl budget concerns**: Sites with 10,000+ pages need Google to prioritize important content. Block low-value pages (search results, filtered products, admin areas) so crawlers focus on what matters.

**Duplicate content**: URLs like `/about` and `/about/` compete against each other, and so do `?sort=price` variations. [Canonical tags](/learn-seo/vue/controlling-crawlers/canonical-urls) consolidate these.

**Staging environments**: Search engines index any public site they find. Block staging and dev environments in [robots.txt](/learn-seo/vue/controlling-crawlers/robots-txt) to avoid duplicate content issues.

**AI training opt-out**: GPTBot is [the most frequently blocked AI crawler](https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/), disallowed by more than 300 of the top 10,000 domains per Cloudflare's 2025 analysis. Block AI training bots without affecting search rankings.

**Server costs**: Bots consume CPU, and heavy pages (maps, infinite scroll, SSR) cost money per request. Blocking unnecessary crawlers reduces load.

Using Nuxt? [Nuxt SEO](/docs/nuxt-seo/getting-started/introduction) handles much of this automatically. See the [Nuxt controlling crawlers guide](/learn-seo/nuxt/controlling-crawlers) for details.