v3.0.0

Features 🚀

Robots.txt Config

The robots.txt standard is important for search engines to understand which pages to crawl and index.

To match closer to the standard, Nuxt Simple Robots now allows you to configure the module by using a robots.txt file.

public/_robots.txt

This file will be parsed and used to configure the module.

If you need programmatic control, you can still configure the module using nuxt.config.ts, Route Rules and Nitro hooks.

Read more at Robots.txt Config.

New Config: groups

• Type: { userAgent: []; allow: []; disallow: []; comments: [] }[]
• Default: []
• Required: false

Define more granular rules for the robots.txt. Each group is a set of rules for specific user agent(s).

export default defineNuxtConfig({
  robots: {
    groups: [
      {
        userAgents: ['AdsBot-Google-Mobile', 'AdsBot-Google-Mobile-Apps'],
        disallow: ['/admin'],
        allow: ['/admin/login'],
        comments: 'Allow Google AdsBot to index the login page but no-admin pages'
      },
    ]
  }
})

New Config: blockNonSeoBots

• Type: boolean
• Default: false
• Required: false

Blocks some non-SEO bots from crawling your site. This is not a replacement for a full-blown bot management solution, but it can help to reduce the load on your server.

See const.ts for the list of bots that are blocked.

export default defineNuxtConfig({
  robots: {
    blockNonSeoBots: true
  }
})

Improved header / meta tag integration

Previously, only routes added to the routeRules would be used to display the X-Robots-Tag header and the <meta name="robots" content="..." /> tag.

This has been changed to include all disallow paths for the * user-agent by default.

New Config: credits

• Type: boolean
• Default: true
• Required: false

Control the module credit comment in the generated robots.txt file.

# START nuxt-simple-robots (indexable) <- credits
 ...
# END nuxt-simple-robots <- credits

export default defineNuxtConfig({
  robots: {
    credits: false
  }
})

New Config: debug

• Type: boolean
• Default: false
• Required: false

Enables debug logs.

export default defineNuxtConfig({
  robots: {
    debug: true
  }
})

Deprecations

Nuxt Site Config Integration

The module now integrates with the nuxt-site-config module.

The siteUrl and indexable config is now deprecated, but will still work.

For most sites, you won't need to provide any further configuration, everything will just work. If you need to modify the default config, the easiest way is to do so through the site config.

export default defineNuxtConfig({
  site: {
    url: 'https://example.com',
    indexable: true
  }
})