Disabling Site Indexing

Introduction

Disabling certain environments of your site from being indexed is an important practice to avoid SEO issues.

For example, you don't want your staging or preview environments to be indexed by search engines as they will cause duplicate content issues as well as confuse end-users.

If you need to disable specific pages from being indexed, refer to the Disabling Page Indexing guide.

Disable Indexing Completely

In some cases, such as internal business tools, or sites that are not ready for the public, you may want to disable indexing completely.

You can achieve this by setting the indexable option to false in your site config.

export default defineNuxtConfig({
  site: { indexable: false }
})

The module then changes two outputs:

• The robots.txt blocks all crawlers with Disallow: /.
• Every page gets a noindex, nofollow robots meta tag and X-Robots-Tag header.

Blocking Crawling Versus Blocking Indexing

The robots.txt controls crawling. The noindex directive controls indexing. A crawler must fetch a page to read its noindex directive. Disallow: / stops the crawler before the fetch. A crawler that obeys robots.txt never sees the noindex directive.

Use Disallow: / for staging and preview environments that no other site links to. Search engines seldom find these URLs. The rule also saves crawl budget.

Do not use it for a site that search engines already know. If another site links to a blocked URL, Google can index the URL without its content. Search Console reports these URLs as "Indexed, though blocked by robots.txt". Google can also keep URLs that it indexed before. It cannot read the new noindex directive on them.

Allow Crawling and Send Noindex

To remove a public site from search results, keep the site indexable and add a catch-all route rule.

export default defineNuxtConfig({
  routeRules: {
    '/**': { robots: 'noindex, nofollow' },
  },
})

The robots.txt then allows crawling. Every page gets the noindex, nofollow robots meta tag and X-Robots-Tag header. Crawlers can fetch each page and read the directive. If a public/robots.txt file disallows /, delete it. If robots.disallowNonIndexableRoutes is true, set it to false. Otherwise the route rule adds Disallow: /*. If the robots config disallows /, remove that entry.

Handling Staging Environments

Staging environments are great for testing out code before it goes to production. However, we definitely don't want search engines to index them.

To control the indexing of these sites we will make use of the env Site Config, which defaults to production.

NUXT_SITE_ENV=staging

Nuxt Robots will disable indexing for any sites which don't have a production environment, so feel free to set this to whatever makes sense for your project.

Handling Multiple Domains

If you deploy the same app to multiple domains (e.g. mydomain.com and test.temp-domain.com), you can disable indexing for specific hosts at runtime using a Nitro plugin.

import { getNitroOrigin } from '#site-config/server'

export default defineNitroPlugin((nitroApp) => {
  nitroApp.hooks.hook('site-config:init', ({ event, siteConfig }) => {
    const host = new URL(getNitroOrigin(event)).hostname
    if (host.includes('staging') || host.includes('test')) {
      siteConfig.push({ indexable: false, _context: 'host-indexing' })
    }
  })
})

This applies to each server request, including page headers and meta tags. For static sites, set NUXT_SITE_ENV in your CI build instead (see Handling Staging Environments).

See Site Config hooks for the site-config:init hook.

Verifying Indexing

To verify that your site is not being indexed, you can check the generated robots.txt file. With site.indexable: false, it should look something like this.

User-agent: *
Disallow: /

With the catch-all route rule, the robots.txt allows crawling instead.

A robots meta tag should also be generated that looks like:

<meta name="robots" content="noindex, nofollow">

For full confidence you can inspect the URL within Google Search Console to see if it's being indexed.