--- title: "Control Web Crawlers and Crawl Budget in Nuxt · Nuxt SEO" canonical_url: "https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers" last_updated: "2026-07-16T12:00:00.000Z" meta: author: "Harlan Wilton" description: "Configure robots.txt, sitemaps, canonicals, and redirects in Nuxt to control crawl budget and block unwanted AI bots." "og:description": "Configure robots.txt, sitemaps, canonicals, and redirects in Nuxt to control crawl budget and block unwanted AI bots." "og:title": "Control Web Crawlers and Crawl Budget in Nuxt · Nuxt SEO" --- Nuxt SEO on GitHub # **Control Web Crawlers and Crawl Budget in Nuxt** Configure robots.txt, sitemaps, canonicals, and redirects in Nuxt to control crawl budget and block unwanted AI bots. [Harlan Wilton](https://x.com/harlan-zw)8 mins read Published **Nov 3, 2024** Updated **Jul 16, 2026** Web crawlers determine what gets indexed and how often. Controlling them affects your [**~~crawl budget~~**](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget): the set of URLs Google can and wants to crawl on your site. It also means guiding AI bots to your best content while blocking the ones that scrape it for training. Most sites don't need to worry about crawl budget. But if you have 10,000+ pages, frequently updated content, or want to control how AI bots consume your data, crawler control matters. ## Types of Crawlers **Search engines**: Index your pages for search results - [**~~Googlebot~~**](https://developers.google.com/search/docs/advanced/crawling/overview-google-crawlers) - [**~~Bingbot~~**](https://ahrefs.com/seo/glossary/bingbot) - [**~~Applebot~~**](https://support.apple.com/en-us/119829) (Spotlight and Siri suggestions) **Social platforms**: Generate link previews when shared - [**~~FacebookExternalHit~~**](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) - Twitterbot, Slackbot, Discordbot **AI training**: Scrape content to train models - [**~~GPTBot~~**](https://developers.openai.com/api/docs/bots) (OpenAI) - [**~~ClaudeBot~~**](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (Anthropic) - [**~~Google-Extended~~**](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers) (Gemini) - [**~~Applebot-Extended~~**](https://support.apple.com/en-us/119829) (Apple Intelligence) **AI search**: Fetch live content to answer a specific query. Allow these if you want your site cited - [**~~OAI-SearchBot and ChatGPT-User~~**](https://developers.openai.com/api/docs/bots) (ChatGPT) - [**~~Claude-SearchBot and Claude-User~~**](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (Claude) - [**~~PerplexityBot and Perplexity-User~~**](https://docs.perplexity.ai/guides/bots) (Perplexity) **Malicious**: Ignore robots.txt, spoof user agents, scan for vulnerabilities. Block these at the [**~~firewall level~~**](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security), not with robots.txt. Training bots feed your content into model training; search bots fetch it live to answer a specific query. You can block one without the other: disallow `**GPTBot**` in robots.txt while leaving `**OAI-SearchBot**` untouched if you want your site cited in ChatGPT Search. ## Control Mechanisms | **Mechanism** | **Use When** | | --- | --- | | [**~~robots.txt~~**](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/robots-txt) | Block site sections, manage crawl budget, block AI crawlers | | [**~~Sitemaps~~**](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/sitemaps) | Help crawlers discover pages, especially on large sites | | [**~~Meta robots~~**](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/meta-tags) | Control indexing per page (noindex, nofollow) | | [**~~Canonical URLs~~**](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/canonical-urls) | Consolidate duplicate content, handle URL parameters | | [**~~Redirects~~**](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/redirects) | Preserve SEO when moving/deleting pages | | [**~~llms.txt~~**](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/llms-txt) | Guide AI tools to your documentation (via nuxt-llms) | | [**~~X-Robots-Tag~~**](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag#xrobotstag) | Control non-HTML files (PDFs, images) | | [**~~Firewall~~**](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security) | Block malicious bots at network level | ## Quick Recipes **Block page from indexing**: [**~~Full guide~~**](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/meta-tags) pages/admin.vue ```vue ``` **Block AI training bots**: [**~~Full guide~~**](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/robots-txt) public/robots.txt ```robots-txt User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot Disallow: / ``` **Fix duplicate content**: [**~~Full guide~~**](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/canonical-urls) pages/products/\[id].vue ```vue ``` **Redirect moved page**: [**~~Full guide~~**](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/redirects) nuxt.config.ts ```ts export default defineNuxtConfig({ routeRules: { '/old-url': { redirect: { to: '/new-url', statusCode: 301 } } } }) ``` ## When Crawler Control Matters Most small sites don't need to optimize crawler behavior, but it matters when: **Crawl budget concerns**: Sites with 10,000+ pages need Google to prioritize important content. Block low-value pages (search results, filtered products, admin areas) so crawlers focus on what matters. **Duplicate content**: URLs like `**/about**` and `**/about/**` compete against each other, and so do `**?sort=price**` variations. [**~~Canonical tags~~**](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/canonical-urls) consolidate these. **Staging environments**: Search engines index any public site they find. Block staging and dev environments in [**~~robots.txt~~**](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/robots-txt) to avoid duplicate content issues. **AI training opt-out**: GPTBot is [**~~the most frequently blocked AI crawler~~**](https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/), disallowed by more than 300 of the top 10,000 domains per Cloudflare's 2025 analysis. Block AI training bots without affecting search rankings. **Server costs**: Bots consume CPU, and heavy pages (maps, infinite scroll, SSR) cost money per request. Blocking unnecessary crawlers reduces load. ## Nuxt SEO Modules Nuxt handles crawler control through dedicated modules. Install once, configure in `**nuxt.config.ts**`, and forget about it. [Nuxt SEO ** ** The all-in-one module that brings it all together.](https://nuxtseo.com/docs/nuxt-seo/getting-started/introduction) [Robots ** ** Tame the robots crawling and indexing your site with ease.](https://nuxtseo.com/docs/robots/getting-started/introduction) [Sitemap ** ** Powerfully flexible XML Sitemaps that integrate seamlessly.](https://nuxtseo.com/docs/sitemap/getting-started/introduction) [SEO Utils ** ** SEO utilities to improve your Nuxt sites discoverability and shareability.](https://nuxtseo.com/docs/seo-utils/getting-started/introduction) [**The 2026 SEO Checklist for Nuxt & Vue ** Pre-launch setup, post-launch verification, and ongoing monitoring. Interactive checklist with links to every guide.](https://nuxtseo.com/learn-seo/checklist) [Haven't launched yet? Start with the **Pre-Launch Warmup**](https://nuxtseo.com/learn-seo/pre-launch-warmup) --- ### **Related ** [**Mastering Meta Tags**](https://nuxtseo.com/learn-seo/nuxt/mastering-meta) [**SSR and SEO**](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/rendering) [**Security Guide**](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security) [**Routes & Rendering**](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering) [**Nuxt SEO Module**](https://nuxtseo.com/docs/nuxt-seo/getting-started/introduction) [**Nuxt Robots Module**](https://nuxtseo.com/docs/robots/getting-started/introduction) [**Nuxt Sitemap Module**](https://nuxtseo.com/docs/sitemap/getting-started/introduction) [**Image Alt Text** Write alt text that passes accessibility audits, ranks in Google Images, and helps AI crawlers understand your Nuxt site's images.](https://nuxtseo.com/learn-seo/nuxt/mastering-meta/alt-text) [**Robots.txt** Generate robots.txt automatically with the Nuxt Robots module, and avoid the mistakes that block Googlebot or leak your admin paths.](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers/robots-txt) **On this page** - [Types of Crawlers](#types-of-crawlers) - [Control Mechanisms](#control-mechanisms) - [Quick Recipes](#quick-recipes) - [When Crawler Control Matters](#when-crawler-control-matters) - [Nuxt SEO Modules](#nuxt-seo-modules)