How Nuxt Robots Works

Nuxt Robots tells robots (crawlers) how to behave by creating a robots.txt file for you, adding an X-Robots-Tag header and <meta name="robots"> tag to your site where appropriate.

One important behavior to control is blocking Google from indexing pages to:

• Prevent duplicate content issues
• Prevent wasting crawl budget

Robots.txt

For robots to understand how they can access your site, they will first check for a robots.txt file.

public
 └── robots.txt

This file is generated differently depending on the environment:

• When deploying using nuxt generate or the nitro.prerender.routes rule, this is a static file.
• Otherwise, it's handled by the server and generated at runtime when requested.

When indexing is disabled a robots.txt will be generated with the following content:

User-agent: *
Disallow: /

This blocks all bots from crawling your site. It does not keep every URL out of search results. If other sites link to a blocked URL, Google can still index it. Crawlers cannot fetch the page, so they never read its noindex directive. See Disabling Site Indexing for the trade-off.

X-Robots-Tag Header and <meta name="robots">

In some situations, the robots.txt becomes too restrictive to provide the level of control you need to manage your site's indexing.

For this reason, the module by default will provide an X-Robots-Tag header and <meta name="robots"> tag.

These are applied using the following logic:

• X-Robots-Tag header - Route Rules are implemented for all modes, otherwise SSR only. This header is added for all pages: set to the enabled value (robotsEnabledValue) for indexable pages and the disabled value (robotsDisabledValue) for non-indexable pages.
• <meta name="robots"> - SSR only, will always be added

If the site is not indexable, every response gets the robotsDisabledValue header. This includes API routes, prerendered JSON files, /robots.txt and build assets. For static hosts, the module writes one /** header rule. Cloudflare and Netlify read this rule from _headers. The module reads the site config at build time for this rule. If you set NUXT_SITE_ENV or NUXT_SITE_INDEXABLE only at runtime, static files and API routes keep the build value. Set these variables at build time too.

Robot Rules

Default values for the robots rule depending on the mode.

For indexable routes the following is used:

<meta name="robots" content="index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1">

Besides giving robots the go-ahead, this also requests that Google:

Choose the snippet length that it believes is most effective to help users discover your content and direct users to your site.

You can learn more on the Robots Meta Tag documentation, feel free to change this to suit your needs using robotsEnabledValue.

For non-indexable routes the following is used:

<meta name="robots" content="noindex, nofollow">

This will tell robots to not index the page.

Development Environment

The module by default will disable indexing in development environments. This is for safety, as you don't want your development environment to be indexed by search engines.

# Block all bots
User-agent: *
Disallow: /

Production Environments

For production environments, the module will generate a robots.txt file that allows all bots.

Out-of-the-box, this will be the following:

User-agent: *
Disallow:

This tells all bots that they can index your entire site.