---
title: "Config using Robots.txt"
description: "Configure your generated robots.txt file with a robots.txt file."
canonical_url: "https://nuxtseo.com/docs/robots/guides/robots-txt"
last_updated: "2026-05-13T04:15:23.367Z"
---

## Introduction

The [robots.txt standard](https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt) is important for search engines
to understand which pages to crawl and index on your site.

New to robots.txt? Check out the [Robots.txt Guide](/learn-seo/nuxt/controlling-crawlers/robots-txt) to learn more.

To match closer to the robots standard, Nuxt Robots recommends configuring the module by using a `robots.txt`, which will be parsed, validated, configuring the module.

If you need programmatic control, you can configure the module using [nuxt.config.ts](/docs/robots/guides/nuxt-config),
[Route Rules](/docs/robots/guides/route-rules) and [Nitro hooks](/docs/robots/nitro-api/nitro-hooks).

## Creating a `robots.txt` file

You can place your file in any location; the easiest is to use: `<rootDir>/public/_robots.txt`.

Additionally, the following paths are supported by default:

```bash [Example File Structure]
# root directory
robots.txt
# asset folders
assets/
├── robots.txt
# pages folder
pages/
├── robots.txt
├── _dir/
│   └── robots.txt
# public folder
public/
├── _robots.txt
├── _dir/
│   └── robots.txt
```

### Custom paths

If you find this too restrictive,
you can use the `mergeWithRobotsTxtPath` config to load your `robots.txt` file from any path.

```ts
export default defineNuxtConfig({
  robots: {
    mergeWithRobotsTxtPath: 'assets/custom/robots.txt'
  }
})
```

## Parsed robots.txt

The following rules are parsed from your `robots.txt` file:

- `User-agent` - The user-agent to apply the rules to.
- `Disallow` - An array of paths to disallow for the user-agent.
- `Allow` - An array of paths to allow for the user-agent.
- `Sitemap` - An array of sitemap URLs to include in the generated [sitemap](/docs/sitemap/getting-started/introduction).
- `Content-Usage` / `Content-Signal` - Directives for expressing AI usage preferences (see [AI Directives](#ai-directives) below).

This parsed data will be shown for environments that are `indexable`.

## AI Directives

AI Directives allow you to control how AI systems, search engines, and automated tools interact with your content. Several methods are supported:

- **Vendor-Specific Tokens** - Official opt-outs for major providers (`Google-Extended`, `Applebot-Extended`)
- **Content-Usage** - IETF standard (`bots`, `train-ai`, `ai-output`, `search`) with `y`/`n` values
- **Content-Signal** - Cloudflare implementation (`search`, `ai-input`, `ai-train`) with `yes`/`no` values

### Quick Example

```txt [robots.txt]
User-agent: *
Allow: /
Content-Usage: bots=y, train-ai=n
Content-Signal: ai-train=no, search=yes
```

<alert type="info">

See the [**AI Directives Guide**](/docs/robots/guides/ai-directives) for complete documentation, examples, validation rules, and best practices.

</alert>

<callout icon="i-heroicons-check-circle" to="/tools/robots-txt-generator">

**Validate your output** - Test your robots.txt rules with our [Robots.txt Tester](/tools/robots-txt-generator).

</callout>

## Conflicting `public/robots.txt`

To ensure other modules can integrate with your generated robots file, you must not have a `robots.txt` file in your `public` folder.

If you do, it will be moved to `<rootDir>/public/_robots.txt` and merged with the generated file.