Control Web Crawlers and Crawl Budget in Nuxt
Web crawlers determine what gets indexed and how often. Controlling them affects your crawl budget: the set of URLs Google can and wants to crawl on your site. It also lets you state crawl preferences for AI bots that honor robots.txt. Use authentication to protect private content.
Most sites do not need crawl-budget work. Google's rough scope includes very large sites and sites with 10,000+ pages that change daily. AI crawl preferences are a separate decision.
Types of Crawlers
Search engines: Index your pages for search results
Social platforms: Generate link previews when shared
- FacebookExternalHit
- Twitterbot, Slackbot, Discordbot
AI training: Scrape content to train models
- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- Google-Extended is a robots.txt control token, not a separate crawler user agent
- Applebot-Extended is a robots.txt control token for Apple's training use, not a separate crawler
AI search and user fetches: providers distinguish automated search crawling from user-initiated requests. Allowing a bot does not guarantee a citation.
- OAI-SearchBot supports ChatGPT Search.
ChatGPT-Userperforms user-initiated fetches; it does not control Search inclusion - Claude-SearchBot and Claude-User (Claude)
- PerplexityBot and Perplexity-User (Perplexity)
Malicious: Ignore robots.txt, spoof user agents, scan for vulnerabilities. Block these at the firewall level, not with robots.txt.
GPTBot in robots.txt while leaving OAI-SearchBot untouched if you want your site cited in ChatGPT Search.Control Mechanisms
| Mechanism | Use When |
|---|---|
| robots.txt | Block site sections, manage crawl budget, block AI crawlers |
| Sitemaps | Help crawlers discover pages, especially on large sites |
| Meta robots | Control indexing per page (noindex, nofollow) |
| Canonical URLs | Consolidate duplicate content, handle URL parameters |
| Redirects | Preserve SEO when moving/deleting pages |
| llms.txt | Publish an optional document index; see the guide for scope and setup |
| X-Robots-Tag | Control non-HTML files (PDFs, images) |
| Firewall | Block malicious bots at network level |
Quick Recipes
Block page from indexing: Full guide
<script setup lang="ts">
useSeoMeta({ robots: 'noindex, follow' })
</script>Block AI training bots: Full guide
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /Fix duplicate content: Full guide
<script setup lang="ts">
const route = useRoute()
useHead(() => ({
link: [{ rel: 'canonical', href: `https://mysite.com/products/${route.params.id}` }]
}))
</script>Redirect moved page: Full guide
export default defineNuxtConfig({
routeRules: {
'/old-url': { redirect: { to: '/new-url', statusCode: 301 } }
}
})When Crawler Control Matters
Most small sites don't need to optimize crawler behavior, but it matters when:
Crawl budget concerns: use Google's rough scope and your crawl evidence. A 10,000-page count alone is not a reason to block URLs. Robots.txt cannot protect private pages or guarantee deindexing.
Duplicate content: URLs like /about and /about/ compete against each other, and so do ?sort=price variations. Canonical tags consolidate these.
Staging environments: protect private staging with authentication. For public pages that should leave search results, allow crawling and use noindex. robots.txt alone does not prevent indexing.
AI training opt-out: check the provider's policy and disallow its training crawler if appropriate. A rule affects only crawlers that honor it; it does not guarantee search rankings or block malicious scraping.
Server costs: Bots consume CPU, and heavy pages (maps, infinite scroll, SSR) cost money per request. Blocking unnecessary crawlers reduces load.
Nuxt SEO Modules
Nuxt handles crawler control through dedicated modules. Install once, configure in nuxt.config.ts, and forget about it.
Pre-launch setup, post-launch verification, and ongoing monitoring. Interactive checklist with links to every guide.