---
title: "Protecting Nuxt Apps from Malicious Crawlers"
description: "Robots.txt is a suggestion; malicious crawlers ignore it. Real protection is middleware, route rules, and DNS verification."
canonical_url: "https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security"
last_updated: "2026-07-16"
---

<key-takeaways>

- Robots.txt is a suggestion, not a security control; use server middleware and route rules for actual protection
- Block non-production environments at the site-config level with `site.indexable` or `NUXT_SITE_ENV`, no middleware required
- Verifying a crawler by IP needs both a reverse and a forward DNS lookup; a single lookup can be spoofed

</key-takeaways>

[Robots.txt](/learn-seo/nuxt/controlling-crawlers/robots-txt) and meta robots tags are polite suggestions. Malicious crawlers ignore them.

You need actual security: block non-production environments, protect development assets, rate limit aggressive crawlers, authenticate sensitive routes, and use HTTPS everywhere. Don't rely on robots.txt for sensitive data, IP blocking alone (easily bypassed), or user-agent detection (trivial to fake).

## Quick Setup

The easiest way to handle crawler blocking in Nuxt is with the Robots module:

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  modules: ['@nuxtjs/robots'],
  robots: {
    // block bots that don't help your SEO (scrapers, aggregators)
    blockNonSeoBots: true,
    // block specific paths
    groups: [
      { userAgent: '*', disallow: ['/admin', '/dashboard'] }
    ]
  }
})
```

<module-card className="w-1/2" slug="robots">



</module-card>

For security beyond crawler blocking, set headers via route rules:

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  nitro: {
    routeRules: {
      '/**': {
        headers: {
          'X-Frame-Options': 'DENY',
          'X-Content-Type-Options': 'nosniff',
          'Referrer-Policy': 'strict-origin-when-cross-origin'
        }
      }
    }
  }
})
```

See [Rate Limiting](#rate-limiting) below for throttling aggressive traffic.

## Environment Protection

### Development & Staging

Block non-production environments from being indexed at the site-config level:

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  site: {
    // fully disable indexing, e.g. for an internal tool
    indexable: false
  }
})
```

```dotenv [.env]
# any value other than "production" blocks indexing
NUXT_SITE_ENV=staging
```

This works by default on most hosting providers, since they set the deploy environment automatically. Verify it by checking the generated `robots.txt` on your staging URL. If you deploy the same app to multiple domains and need per-host control, hook into `robots:config`:

```ts [server/plugins/robots-domain.ts]
export default defineNitroPlugin((nitroApp) => {
  nitroApp.hooks.hook('robots:config', (ctx) => {
    const host = ctx.event?.headers.get('host')
    if (host?.includes('staging') || host?.includes('test')) {
      ctx.groups[0].disallow = ['/']
    }
  })
})
```

For basic auth on top of that, add middleware:

```ts [server/middleware/staging-auth.ts]
export default defineEventHandler((event) => {
  if (process.env.NODE_ENV === 'production')
    return

  const auth = getRequestHeader(event, 'authorization')
  if (!auth) {
    setResponseStatus(event, 401)
    setHeader(event, 'WWW-Authenticate', 'Basic')
    return 'Authentication required'
  }
})
```

### Sensitive Routes

Use the [Robots module](/docs/robots/getting-started/introduction) to block indexing of sensitive paths:

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  modules: ['@nuxtjs/robots'],
  robots: {
    disallow: ['/admin', '/dashboard', '/user']
  }
})
```

For authentication and custom protection logic, add middleware:

```ts [server/middleware/protect-routes.ts]
export default defineEventHandler((event) => {
  const protectedPaths = ['/admin', '/dashboard', '/user']

  if (protectedPaths.some(path => event.path.startsWith(path))) {
    // ensure the user is authenticated
    if (!event.context.auth?.user) {
      return sendRedirect(event, '/login')
    }

    // block indexing of protected content
    setHeader(event, 'X-Robots-Tag', 'noindex, nofollow')
  }
})
```

Or use route rules for static protection:

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  nitro: {
    routeRules: {
      '/admin/**': {
        headers: {
          'X-Robots-Tag': 'noindex, nofollow'
        }
      }
    }
  }
})
```

## Crawler Identification

Good crawlers publish verifiable IPs. Bad ones fake a `Googlebot` user agent and hope you don't check.

```ts [server/utils/verify-crawler.ts]
import { resolve4, reverse } from 'node:dns/promises'

export async function isLegitCrawler(ip: string, userAgent: string) {
  if (!userAgent.includes('Googlebot'))
    return false

  // reverse lookup: which hostname claims this IP?
  const [hostname] = await reverse(ip)
  if (!hostname?.endsWith('.googlebot.com'))
    return false

  // forward-confirm: does that hostname resolve back to the same IP?
  const ips = await resolve4(hostname)
  return ips.includes(ip)
}
```

A single reverse lookup can be spoofed; [Google's own verification guide](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot) requires the forward-confirmation step above. For less code, skip DNS and compare the request IP against Google's [published crawler IP ranges](https://developers.google.com/static/search/apis/ipranges/googlebot.json) instead.

## Rate Limiting

Use [nuxt-security](https://nuxt-security.vercel.app/middleware/rate-limiter) for built-in rate limiting:

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  modules: ['nuxt-security'],
  security: {
    rateLimiter: {
      tokensPerInterval: 100,
      interval: 60000, // 1 minute
      headers: true
    }
  }
})
```

Or [nuxt-rate-limit](https://github.com/timb-103/nuxt-rate-limit) for simpler API-focused rate limiting:

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  modules: ['nuxt-rate-limit'],
  nuxtRateLimit: {
    routes: {
      '/api/*': { maxRequests: 100, intervalSeconds: 60 },
      '/api/auth/*': { maxRequests: 10, intervalSeconds: 60 }
    }
  }
})
```

For custom tiered logic, implement it manually:

```ts [server/middleware/rate-limit.ts]
const requestCounts = new Map<string, { count: number, resetAt: number }>()

export default defineEventHandler((event) => {
  const ip = getRequestIP(event)
  const now = Date.now()
  const windowMs = 15 * 60 * 1000 // 15 minutes

  const record = requestCounts.get(ip)

  if (!record || now > record.resetAt) {
    requestCounts.set(ip, { count: 1, resetAt: now + windowMs })
    return
  }

  // different limits for different paths
  const maxRequests = event.path.startsWith('/api') ? 100 : 1000

  if (record.count > maxRequests) {
    throw createError({
      statusCode: 429,
      message: 'Too Many Requests'
    })
  }

  record.count++
})
```

## Infrastructure Security

### HTTPS Enforcement

Nuxt handles HTTPS redirects via middleware:

```ts [server/middleware/https.ts]
export default defineEventHandler((event) => {
  const proto = getRequestHeader(event, 'x-forwarded-proto')

  if (proto === 'http') {
    return sendRedirect(
      event,
      `https://${getRequestHost(event)}${event.path}`,
      301
    )
  }
})
```

### Security Headers

Use [nuxt-security](https://nuxt-security.vercel.app/) to configure security headers following OWASP patterns:

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  modules: ['nuxt-security'],
  security: {
    headers: {
      crossOriginResourcePolicy: 'same-origin',
      crossOriginOpenerPolicy: 'same-origin',
      xContentTypeOptions: 'nosniff',
      referrerPolicy: 'strict-origin-when-cross-origin',
      contentSecurityPolicy: {
        'default-src': ['\'self\''],
        'script-src': ['\'self\'', '\'unsafe-inline\'']
      }
    }
  }
})
```

Or configure manually via route rules:

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  nitro: {
    routeRules: {
      '/**': {
        headers: {
          'X-Frame-Options': 'DENY',
          'X-Content-Type-Options': 'nosniff',
          'Referrer-Policy': 'strict-origin-when-cross-origin',
          ...(process.env.NODE_ENV === 'production' && {
            'Content-Security-Policy': 'default-src \'self\';'
          })
        }
      }
    }
  }
})
```

## Monitoring & Detection

Log suspicious patterns so you know what you're blocking, not just that you're blocking something:

```ts [server/middleware/crawler-monitor.ts]
export default defineEventHandler((event) => {
  const ua = getRequestHeader(event, 'user-agent')
  const ip = getRequestIP(event)

  if (isSuspiciousPattern(ua, ip)) {
    console.warn(`Suspicious crawler: ${ip} with UA: ${ua}`)
  }
})
```

<tip>

If you're running a small blog, a full WAF (Cloudflare, AWS WAF) is overkill. Add one once you're getting attacked, not before.

</tip>

## Common Attacks

### Content Scraping

Use [nuxt-security](https://nuxt-security.vercel.app/middleware/rate-limiter) rate limiting to slow automated scraping:

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  modules: ['nuxt-security'],
  security: {
    rateLimiter: {
      tokensPerInterval: 50,
      interval: 60000
    }
  }
})
```

For more control over bot detection and delays:

```ts [server/middleware/anti-scraping.ts]
const requestCounts = new Map<string, number>()

export default defineEventHandler(async (event) => {
  const ip = getRequestIP(event)
  const count = requestCounts.get(ip) || 0

  if (count > 100) {
    throw createError({
      statusCode: 429,
      message: 'Too Many Requests'
    })
  }

  requestCounts.set(ip, count + 1)

  // add slight delays to automated requests
  const ua = getRequestHeader(event, 'user-agent')
  if (isBot(ua)) {
    await new Promise(r => setTimeout(r, 500))
  }
})
```

### Form Spam

Use [nuxt-security](https://nuxt-security.vercel.app/) for XSS validation and request limiting:

```ts [nuxt.config.ts]
export default defineNuxtConfig({
  modules: ['nuxt-security'],
  security: {
    xssValidator: true,
    rateLimiter: {
      tokensPerInterval: 5,
      interval: 60000
    }
  }
})
```

For honeypot fields and custom validation:

```ts [server/api/contact.post.ts]
const submissionCounts = new Map<string, number>()

export default defineEventHandler(async (event) => {
  const body = await readBody(event)
  const ip = getRequestIP(event)

  // honeypot check
  if (body.website) { // hidden field
    return { success: false }
  }

  // rate limiting
  const count = submissionCounts.get(ip) || 0
  if (count > 5) {
    throw createError({
      statusCode: 429,
      message: 'Too many attempts'
    })
  }

  submissionCounts.set(ip, count + 1)

  // process legitimate submission
  // ...
})
```

## Checklist

<checklist id="nuxt-security">

- Non-production environments blocked from indexing via `site.indexable` or `NUXT_SITE_ENV`
- Sensitive routes (`/admin`, `/dashboard`) require authentication and send `noindex`
- Rate limits configured on `/api/*` and form endpoints
- Security headers (`X-Frame-Options`, CSP, `Referrer-Policy`) set via route rules or nuxt-security
- HTTPS enforced for every request
- Crawler identity verified with both reverse and forward DNS lookups, or Google's published IP ranges

</checklist>

Building the same protections in plain Vue? [Protecting Vue Apps from Malicious Crawlers →](/learn-seo/vue/routes-and-rendering/security)