Config using Robots.txt

Introduction

The robots.txt standard is important for search engines to understand which pages to crawl and index.

To match closer to the standard, Nuxt Robots recommends configuring the module by using a robots.txt.

This file will be parsed, validated and used to configure the module.

If you need programmatic control, you can configure the module using nuxt.config.ts, Route Rules and Nitro hooks.

Creating a robots.txt file

You can place your file in any location; the easiest is to use: <rootDir>/public/_robots.txt.

Additionally, the following paths are supported by default:

# root directory
robots.txt
# asset folders
assets/
├── robots.txt
# pages folder
pages/
├── robots.txt
├── _dir/
│   └── robots.txt
# public folder
public/
├── _robots.txt
├── _dir/
│   └── robots.txt

Custom paths

If you find this too restrictive, you can use the mergeWithRobotsTxtPath config to load your robots.txt file from any path.

export default defineNuxtConfig({
  robots: {
    mergeWithRobotsTxtPath: 'assets/custom/robots.txt'
  }
})

Parsed robots.txt

The following rules are parsed from your robots.txt file:

• User-agent - The user-agent to apply the rules to.
• Disallow - An array of paths to disallow for the user-agent.
• Allow - An array of paths to allow for the user-agent.
• Sitemap - An array of sitemap URLs to include in the generated sitemap.

This parsed data will be shown for environments that are indexable.

Conflicting public/robots.txt

To ensure other modules can integrate with your generated robots file, you must not have a robots.txt file in your public folder.

If you do, it will be moved to <rootDir>/public/_robots.txt and merged with the generated file.