---
title: "Protecting Nuxt Apps from Malicious Crawlers"
description: "Protect private Nuxt routes with authentication, set crawler preferences separately, and choose rate limits for your deployment."
canonical_url: "https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security"
last_updated: "2026-10-05"
---

::key-takeaways
- Robots.txt sets voluntary crawl preferences; authentication protects private content
- Indexing settings express crawler preferences; authentication protects private staging and sensitive routes
- Verifying a crawler by IP needs both a reverse and a forward DNS lookup; a single lookup can be spoofed
::

[Robots.txt](/learn-seo/nuxt/controlling-crawlers/robots-txt) and meta robots tags are polite suggestions. Malicious crawlers ignore them.

Protect non-production environments and development assets. Rate limit aggressive crawlers, authenticate sensitive routes, and use HTTPS. Don't rely on robots.txt for sensitive data, IP blocking alone (easily bypassed), or user-agent detection (trivial to fake).

## Quick Setup

For crawlers that honor robots.txt, set crawl preferences with the Robots module. This does not block requests or authenticate visitors:

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  modules: ['@nuxtjs/robots'],
  robots: {
    // set the module’s non-SEO bot crawl preferences
    blockNonSeoBots: true,
    // block specific paths
    groups: [
      { userAgent: '*', disallow: ['/admin', '/dashboard'] }
    ]
  }
})
```

::module-card{slug="robots" .w-1/2}
::

For security beyond crawler blocking, set headers via route rules:

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  nitro: {
    routeRules: {
      '/**': {
        headers: {
          'X-Frame-Options': 'DENY',
          'X-Content-Type-Options': 'nosniff',
          'Referrer-Policy': 'strict-origin-when-cross-origin'
        }
      }
    }
  }
})
```

See [Rate Limiting](#rate-limiting) below for throttling aggressive traffic.

## Environment Protection

### Development & Staging

Set indexing preferences for a non-production environment. This is not access control:

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  site: {
    // request exclusion from indexing; does not authenticate visitors
    indexable: false
  }
})
```

```dotenv [.env]
# request non-production indexing preferences; inspect the generated response
NUXT_SITE_ENV=staging
```

Inspect the generated robots.txt on staging and production. A production-mode build can still run on staging; configure and inspect the actual deployment environment.

Protect staging with your hosting provider's access control or the app's authentication system. A check for the presence of an Authorization header does not validate credentials.

### Sensitive Routes

Use the [Robots module](/docs/robots/getting-started/introduction) to express crawl preferences for sensitive paths. These rules do not prevent indexing or protect access:

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  modules: ['@nuxtjs/robots'],
  robots: {
    disallow: ['/admin', '/dashboard', '/user']
  }
})
```

Require server-side authentication through your app's auth system. A property such as `event.context.auth` exists only if trusted authentication middleware creates it. Protect API responses as well as pages.

Route rules can add indexing headers after authentication protects access:

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  nitro: {
    routeRules: {
      '/admin/**': {
        headers: {
          'X-Robots-Tag': 'noindex, nofollow'
        }
      }
    }
  }
})
```

## Crawler Identification

Verify claimed crawler identity against the provider's policy. This DNS excerpt handles IPv4 Googlebot addresses with a [googlebot.com](http://googlebot.com) hostname; it is not a verifier for every Google crawler or IPv6 address. DNS lookup failures propagate.

```ts [server/utils/verify-crawler.ts]
import { resolve4, reverse } from 'node:dns/promises'

export async function isLegitCrawler(ip: string, userAgent: string) {
  if (!userAgent.includes('Googlebot'))
    return false

  // reverse lookup: which hostname claims this IP?
  const [hostname] = await reverse(ip)
  if (!hostname?.endsWith('.googlebot.com'))
    return false

  // forward-confirm: does that hostname resolve back to the same IP?
  const ips = await resolve4(hostname)
  return ips.includes(ip)
}
```

A single reverse lookup can be spoofed; [Google's own verification guide](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot) requires the forward-confirmation step above. For less code, skip DNS and compare the request IP against Google's [published crawler IP ranges](https://developers.google.com/static/search/apis/ipranges/googlebot.json) instead.

## Rate Limiting

Use a limiter appropriate to your deployment. In-memory counters apply only to one process and need expiry; they do not provide a shared limit across workers or replicas.

Use [nuxt-security](https://nuxt-security.vercel.app/middleware/rate-limiter) for built-in rate limiting:

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  modules: ['nuxt-security'],
  security: {
    rateLimiter: {
      tokensPerInterval: 100,
      interval: 60000, // 1 minute
      headers: true
    }
  }
})
```

The module's [rate-limiter documentation](https://nuxt-security.vercel.app/middleware/rate-limiter) warns that its built-in limiter is intended for simpler applications. Choose infrastructure limits and shared storage for complex deployments. Do not treat a process-local map as a shared limit.

## Infrastructure Security

### HTTPS Enforcement

Configure HTTPS enforcement at your trusted deployment boundary. Redirect to your configured public origin; do not build a destination from an untrusted Host header. Check how your proxy supplies scheme and client IP information.

### Security Headers

Configure security headers from your app’s resource and embedding requirements. A permissive CSP with unsafe-inline does not prevent inline script execution. Cross-origin restrictions can break intended resources or integrations. Follow the [module’s current header documentation](https://nuxt-security.vercel.app/headers/content-security-policy) and test the policy in your deployment.

Or configure manually via route rules:

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  nitro: {
    routeRules: {
      '/**': {
        headers: {
          'X-Frame-Options': 'DENY',
          'X-Content-Type-Options': 'nosniff',
          'Referrer-Policy': 'strict-origin-when-cross-origin',

        }
      }
    }
  }
})
```

## Monitoring & Detection

Record request rates, rejected requests, and response status in your application or infrastructure logs. Define detection rules for your traffic; undefined helpers such as `isSuspiciousPattern` are not an implementation.

## Common Attacks

### Content Scraping

Use rate limits appropriate to your deployment. Request counters need bounded storage and expiry. A user-agent string alone does not establish that a request is automated.

### Form Spam

Validate form input and apply request limits at the endpoint. A honeypot can supplement these controls, but it does not authenticate a visitor or provide a shared rate limit. See the current [XSS validator configuration](https://nuxt-security.vercel.app/middleware/xss-validator); input checks do not replace safe output handling.

## Checklist

::checklist{#nuxt-security}
- Private staging requires authentication; verify indexing preferences separately
- Sensitive routes (`/admin`, `/dashboard`) require authentication and send `noindex`
- Rate limits configured on `/api/*` and form endpoints
- Security headers (`X-Frame-Options`, CSP, `Referrer-Policy`) set via route rules or nuxt-security
- HTTPS enforced for every request
- Crawler identity verified with both reverse and forward DNS lookups, or Google's published IP ranges
::

Building the same protections in plain Vue? [Protecting Vue Apps from Malicious Crawlers →](/learn-seo/vue/routes-and-rendering/security)

## Sitemap

See the full [sitemap](/sitemap.md) for all pages.