Yandex: Clean-param

Nuxt Robots is built around first-party robots.txt specifications from Google and Bing.

Some users may want to configure Yandex, a popular search engine in Russia, and find that rules aren't working. To use Yandex you will need to provide alternative directives.

Clean-param

The clean-param directive is used to remove query parameters from URLs. This is useful for SEO as it prevents duplicate content and consolidates page rank.

It can either be configured directly through robots.txt when targeting Yandex or through the module configuration.

Robots.txt

To configure the clean-param directive in your robots.txt file, you can use the following syntax:

User-agent: Yandex
Clean-param: param1 param2

This will remove the param1 and param2 query parameters from URLs.

Module Configuration

To configure the clean-param directive in your nuxt.config.ts file, you can use the following syntax:

export default defineNuxtConfig({
  robots: {
    groups: [
      {
        userAgent: ['Yandex'],
        cleanParam: ['param1', 'param2']
      }
    ]
  }
})

Host & Crawl-delay

These directives are deprecated and should not be used. All search engines will ignore them.