Control Web Crawlers and Crawl Budget in Vue

Web crawlers discover and fetch pages. Search engines decide which pages to index. Controlling them affects your crawl budget: the set of URLs Google can and wants to crawl on your site. It also means guiding AI bots to your best content while blocking the ones that scrape it for training.

Most sites don't need to worry about crawl budget. Large, rapidly changing sites may need crawl-budget controls. AI crawler preferences are a separate concern.

Types of Crawlers

Search engines: Index your pages for search results

• Googlebot
• Bingbot
• Applebot (Spotlight and Siri suggestions)

Social platforms: Generate link previews when shared

• FacebookExternalHit
• Twitterbot, Slackbot, Discordbot

AI training crawlers and usage controls: Providers use crawlers and robots.txt control tokens for different purposes

• GPTBot (OpenAI)
• ClaudeBot (Anthropic)
• Google-Extended (a control token for training and grounding in specified Gemini products, not a separate crawler)
• Applebot-Extended (a control token for Apple foundation-model training, not a separate crawler)

AI search and user fetches: Search crawlers build indexes. User agents can fetch pages after a user request. Provider rules differ

• OAI-SearchBot and ChatGPT-User (ChatGPT)
• Claude-SearchBot and Claude-User (Claude)
• PerplexityBot and Perplexity-User (Perplexity)

Malicious: Ignore robots.txt, spoof user agents, scan for vulnerabilities. Block these at the firewall level, not with robots.txt.

Control Mechanisms

Quick Recipes

Block page from indexing: Full guide

<script setup lang="ts">
import { useSeoMeta } from '@unhead/vue'

useSeoMeta({ robots: 'noindex, follow' })
</script>

Block AI training bots: Full guide

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: Google-Extended
Disallow: /

Fix duplicate content: Full guide

<script setup lang="ts">
import { useHead } from '@unhead/vue'
import { useRoute } from 'vue-router'

const route = useRoute()

useHead({
  link: [{ rel: 'canonical', href: () => new URL(route.path, 'https://mysite.com').href }]
})
</script>

Redirect moved page: Express middleware excerpt; install Express and register this before your page handler. Full guide

app.get('/old-url', (req, res) => res.redirect(301, '/new-url'))

When Crawler Control Matters

Most small sites don't need to optimize crawler behavior, but it matters when:

Crawl budget concerns: Large sites with rapidly changing pages may need crawl inventory controls. Google's size estimates are examples, not exact thresholds. Block low-value pages (search results, filtered products, admin areas) so crawlers focus on what matters.

Duplicate content: URLs like /about and /about/ can expose duplicate content, as can ?sort=price variations. Canonical tags consolidate these.

Staging environments: Protect private staging with authentication. Use crawlable noindex when a public staging page must stay out of search. robots.txt alone does not prevent indexing.

AI training opt-out: GPTBot is the most frequently blocked AI crawler, disallowed by more than 300 of the top 10,000 domains per Cloudflare's 2025 analysis. Configure each provider's training controls separately from its search controls.

Server costs: Bots consume CPU, and heavy pages (maps, infinite scroll, SSR) cost money per request. Blocking unnecessary crawlers reduces load.

Using Nuxt? Nuxt SEO handles much of this automatically. See the Nuxt controlling crawlers guide for details.