--- title: "How to Set Up robots.txt in Vue" description: "Write or generate robots.txt server-side in Vue, and avoid the mistakes that block Googlebot or leak your admin paths." canonical_url: "https://nuxtseo.com/learn-seo/vue/controlling-crawlers/robots-txt" last_updated: "2026-07-16" --- - robots.txt is advisory: crawlers can ignore it, so never use it for security - Primary uses are crawl budget optimization and blocking AI training bots - Content-Usage and Content-Signal directives let you allow search indexing while opting out of AI training The `robots.txt` file controls which parts of your site crawlers can access. [Officially adopted as RFC 9309](https://datatracker.ietf.org/doc/html/rfc9309) in September 2022, it's used to [manage crawl budget](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget) on large sites and to tell AI crawlers whether they can use your content for training or real-time answers. [Robots.txt is not a security mechanism](https://developer.mozilla.org/en-US/docs/Web/Security/Practical_implementation_guides/Robots_txt): crawlers can ignore it entirely. For individual page control, use [meta robots tags](/learn-seo/vue/controlling-crawlers/meta-tags) instead. ## Quick Setup To get started quickly with a static `robots.txt`, add the file in your public directory: ```dir public/ robots.txt ``` Add your rules: ```robots-txt [robots.txt] # Allow all crawlers User-agent: * Disallow: # Optionally point to your sitemap Sitemap: https://mysite.com/sitemap.xml ``` For environment-specific rules (e.g. blocking all crawlers in staging), generate `robots.txt` server-side instead: ```ts [Express] import express from 'express' const app = express() app.get('/robots.txt', (req, res) => { const isDev = process.env.NODE_ENV !== 'production' const robots = isDev ? 'User-agent: *\nDisallow: /' : 'User-agent: *\nDisallow:\nSitemap: https://mysite.com/sitemap.xml' res.type('text/plain').send(robots) }) ``` ```ts [Vite] // server.js for Vite SSR import express from 'express' const app = express() app.use((req, res, next) => { if (req.path === '/robots.txt') { const isDev = process.env.NODE_ENV !== 'production' const robots = isDev ? 'User-agent: *\nDisallow: /' : 'User-agent: *\nDisallow:\nSitemap: https://mysite.com/sitemap.xml' return res.type('text/plain').send(robots) } next() }) ``` ```ts [H3] import { defineEventHandler, setHeader } from 'h3' export default defineEventHandler((event) => { if (event.path === '/robots.txt') { const isDev = process.env.NODE_ENV !== 'production' const robots = isDev ? 'User-agent: *\nDisallow: /' : 'User-agent: *\nDisallow:\nSitemap: https://mysite.com/sitemap.xml' setHeader(event, 'Content-Type', 'text/plain') return robots } }) ``` ## Robots.txt Syntax The `robots.txt` file consists of directives grouped by user agent. Google [uses the most specific matching rule](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) based on path length: ```robots-txt [robots.txt] # Define which crawler these rules apply to User-agent: * # Block access to specific paths Disallow: /admin # Allow access to specific paths (optional, more specific than Disallow) Allow: /admin/public # Point to your sitemap Sitemap: https://mysite.com/sitemap.xml ``` ### User-agent The `User-agent` directive specifies which crawler the rules apply to: ```robots-txt [robots.txt] # All crawlers User-agent: * # Just Googlebot User-agent: Googlebot # Multiple specific crawlers User-agent: Googlebot User-agent: Bingbot Disallow: /private ``` Common crawler user agents: - [Googlebot](https://developers.google.com/search/docs/advanced/crawling/overview-google-crawlers): Google's search crawler, roughly half of the AI-and-search-crawler traffic [Cloudflare measured in May 2025](https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/) - [Google-Extended](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers): not a separate crawler, a control token that governs whether Googlebot-crawled content can train [Gemini](https://gemini.google.com); blocking it doesn't affect Search - [GPTBot](https://developers.openai.com/api/docs/bots): OpenAI's training crawler, about 7.7% of that same May 2025 sample - [OAI-SearchBot](https://developers.openai.com/api/docs/bots): OpenAI's crawler for the [ChatGPT](https://chatgpt.com) search index, not used for training - [ClaudeBot](https://support.claude.com/en/articles/9906653-claude-bot-and-crawling): Anthropic's training crawler - [Applebot](https://support.apple.com/en-us/119829): Apple's search crawler for Siri, Spotlight, and Safari - [Applebot-Extended](https://support.apple.com/en-us/119829): opts your content out of training Apple's foundation models, without affecting Applebot's crawling or Apple Search visibility - [CCBot](https://commoncrawl.org/ccbot): Common Crawl's dataset builder, widely blocked because its data feeds many third-party AI models - [Bingbot](https://ahrefs.com/seo/glossary/bingbot): Microsoft's search crawler Not every [OpenAI](https://openai.com) bot needs blocking. [ChatGPT-User](https://developers.openai.com/api/docs/bots) only fires when a person asks ChatGPT about your page live; it doesn't crawl proactively for training the way GPTBot does. ### Allow / Disallow The `Allow` and `Disallow` directives control path access: ```robots-txt [robots.txt] User-agent: * # Block all paths starting with /admin Disallow: /admin # Block a specific file Disallow: /private.html # Block files with specific extensions Disallow: /*.pdf$ # Block URL parameters Disallow: /*?* ``` Wildcards supported ([RFC 9309](https://datatracker.ietf.org/doc/html/rfc9309)): - `*`: matches zero or more characters - `$`: matches the end of the URL - Paths are case-sensitive and relative to domain root ### Sitemap The `Sitemap` directive tells crawlers where to find your [sitemap.xml](/learn-seo/vue/controlling-crawlers/sitemaps): ```robots-txt [robots.txt] Sitemap: https://mysite.com/sitemap.xml # Multiple sitemaps Sitemap: https://mysite.com/products-sitemap.xml Sitemap: https://mysite.com/blog-sitemap.xml ``` ### Crawl-Delay (Non-Standard) `Crawl-Delay` is not part of [RFC 9309](https://datatracker.ietf.org/doc/html/rfc9309). Google ignores it. Bing and Yandex support it: ```robots-txt [robots.txt] User-agent: Bingbot Crawl-delay: 10 # seconds between requests ``` For Google, you [manage crawl rate in Search Console](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget). ## Security: Why robots.txt Fails [Robots.txt is not a security mechanism](https://developer.mozilla.org/en-US/docs/Web/Security/Practical_implementation_guides/Robots_txt). Malicious crawlers ignore it, and listing paths in `Disallow` [reveals their location to attackers](https://www.searchenginejournal.com/robots-txt-security-risks/289719/). A common mistake: ```robots-txt # ❌ Advertises your admin panel location User-agent: * Disallow: /admin Disallow: /wp-admin Disallow: /api/internal ``` Never use robots.txt to hide sensitive content. Listing paths in Disallow advertises their location to attackers, and malicious bots ignore robots.txt entirely. Use authentication and proper access controls instead. Use [proper authentication](https://developers.google.com/search/docs/crawling-indexing/control-what-you-share) instead. See our [security guide](/learn-seo/vue/routes-and-rendering/security) for details. ## Crawling vs Indexing Blocking a URL in `robots.txt` prevents crawling but [doesn't prevent indexing](https://developers.google.com/search/docs/crawling-indexing/robots/intro). If other sites link to the URL, Google can still index it without crawling, showing the URL with no snippet. To prevent indexing: - Use [`noindex` meta tag](/learn-seo/vue/controlling-crawlers/meta-tags) (requires allowing crawl) - Use password protection or authentication - Return 404/410 status codes Don't block pages with `noindex` in `robots.txt`. Google can't see the tag if it can't crawl. ## Common Mistakes ### 1. Blocking JavaScript and CSS [Google needs JavaScript and CSS to render pages](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics). Blocking them breaks indexing: ```robots-txt [robots.txt] # ❌ Prevents Google from rendering your Vue app User-agent: * Disallow: /assets/ Disallow: /*.js$ Disallow: /*.css$ ``` Vue apps are JavaScript-heavy. Never block `.js`, `.css`, or `/assets/` from Googlebot. ### 2. Blocking Dev Sites in Production Copy-pasting a dev `robots.txt` to production blocks all crawlers: ```robots-txt [robots.txt] # ❌ Accidentally left from staging User-agent: * Disallow: / ``` Use environment checks in your server-rendered `robots.txt` (see [Quick Setup](#quick-setup)) to avoid this. Also easy to get wrong: confusing `robots.txt` with `noindex`. Blocking a page in `robots.txt` doesn't remove it from search results, robots.txt controls crawling, not indexing. Use [`noindex` meta tags](/learn-seo/vue/controlling-crawlers/meta-tags) to deindex a page instead. To confirm your robots.txt is live and correct: 1. Visit `https://yoursite.com/robots.txt` to confirm it loads 2. Run it through the [Google Search Console robots.txt tester](https://search.google.com/search-console/robots-txt), which validates syntax and tests individual URLs 3. Check server logs for a 200 status on `/robots.txt` to confirm crawlers can access it ## Common Patterns ### Allow Everything (Default) ```robots-txt User-agent: * Disallow: ``` ### Block Everything Useful for staging or development environments. ```robots-txt User-agent: * Disallow: / ``` See our [security guide](/learn-seo/vue/routes-and-rendering/security) for more on environment protection. ### Block AI Training Crawlers GPTBot is one of the most blocked AI crawlers: [Cloudflare's 2025 analysis](https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/) found it fully disallowed by 250 domains and partially disallowed by another 62. Blocking AI training bots doesn't affect your appearance in search results. ```robots-txt # Block AI model training User-agent: GPTBot User-agent: ClaudeBot User-agent: Applebot-Extended User-agent: Google-Extended User-agent: CCBot Disallow: / ``` Be careful not to block search bots like `OAI-SearchBot` or `Applebot` (unless you want to be invisible in their search products). Blocking `GPTBot` is safe for search visibility; blocking `OAI-SearchBot` removes you from ChatGPT Search. ### AI Directives: Content-Usage & Content-Signal Blocking user agents outright isn't always what you want. Content-Usage and Content-Signal let you express preferences about how AI systems use your content without blocking crawlers entirely: - **Content-Usage** (IETF AI Preferences): a robots.txt directive and HTTP header using `y`/`n` values for `train-ai` - **Content-Signal** (Cloudflare): uses `yes`/`no` values for `search`, `ai-input`, and `ai-train` ```robots-txt User-agent: * Allow: / # IETF AI Preferences Content-Usage: train-ai=n # Cloudflare Content Signals Content-Signal: search=yes, ai-input=no, ai-train=no ``` This lets crawlers access your content for search indexing while blocking AI training and RAG/grounding uses. Use both together for broader coverage. AI directives rely on voluntary compliance: crawlers can ignore them, so combine them with User-agent blocks for stronger protection. ### Block Search, Allow Social Sharing For private sites where you still want [link previews](/learn-seo/vue/mastering-meta/social-sharing): ```robots-txt # Block search engines User-agent: Googlebot User-agent: Bingbot Disallow: / # Allow social link preview crawlers User-agent: facebookexternalhit User-agent: Twitterbot User-agent: Slackbot Allow: / ``` ### Optimize Crawl Budget for Large Sites If you have 10,000+ pages, [block low-value URLs](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget) to focus crawl budget on important content: ```robots-txt User-agent: * # Block internal search results Disallow: /search? # Block infinite scroll pagination Disallow: /*?page= # Block filtered/sorted product pages Disallow: /products?*sort= Disallow: /products?*filter= # Block print versions Disallow: /*/print ``` Sites under 1,000 pages don't need crawl budget optimization. ## Checklist - robots.txt lives at your domain root and returns 200 - Sitemap directive points to your sitemap.xml - JavaScript, CSS, and `/assets/` stay unblocked for Googlebot - Staging and dev environments block all crawlers automatically - Sensitive paths use authentication, not `Disallow`, to stay hidden - AI training bots are blocked with User-agent rules or Content-Usage/Content-Signal directives Building with Nuxt instead? The [Nuxt SEO module](/docs/nuxt-seo/getting-started/introduction) handles most of this automatically. [Robots.txt in Nuxt →](/learn-seo/nuxt/controlling-crawlers/robots-txt)