Protecting Nuxt Apps from Malicious Crawlers

Robots.txt and meta robots tags are polite suggestions. Malicious crawlers ignore them.

Protect non-production environments and development assets. Rate limit aggressive crawlers, authenticate sensitive routes, and use HTTPS. Don't rely on robots.txt for sensitive data, IP blocking alone (easily bypassed), or user-agent detection (trivial to fake).

Quick Setup

For crawlers that honor robots.txt, set crawl preferences with the Robots module. This does not block requests or authenticate visitors:

export default defineNuxtConfig({
  modules: ['@nuxtjs/robots'],
  robots: {
    // set the module’s non-SEO bot crawl preferences
    blockNonSeoBots: true,
    // block specific paths
    groups: [
      { userAgent: '*', disallow: ['/admin', '/dashboard'] }
    ]
  }
})

For security beyond crawler blocking, set headers via route rules:

export default defineNuxtConfig({
  nitro: {
    routeRules: {
      '/**': {
        headers: {
          'X-Frame-Options': 'DENY',
          'X-Content-Type-Options': 'nosniff',
          'Referrer-Policy': 'strict-origin-when-cross-origin'
        }
      }
    }
  }
})

See Rate Limiting below for throttling aggressive traffic.

Environment Protection

Development & Staging

Set indexing preferences for a non-production environment. This is not access control:

export default defineNuxtConfig({
  site: {
    // request exclusion from indexing; does not authenticate visitors
    indexable: false
  }
})

# request non-production indexing preferences; inspect the generated response
NUXT_SITE_ENV=staging

Inspect the generated robots.txt on staging and production. A production-mode build can still run on staging; configure and inspect the actual deployment environment.

Protect staging with your hosting provider's access control or the app's authentication system. A check for the presence of an Authorization header does not validate credentials.

Sensitive Routes

Use the Robots module to express crawl preferences for sensitive paths. These rules do not prevent indexing or protect access:

export default defineNuxtConfig({
  modules: ['@nuxtjs/robots'],
  robots: {
    disallow: ['/admin', '/dashboard', '/user']
  }
})

Require server-side authentication through your app's auth system. A property such as event.context.auth exists only if trusted authentication middleware creates it. Protect API responses as well as pages.

Route rules can add indexing headers after authentication protects access:

export default defineNuxtConfig({
  nitro: {
    routeRules: {
      '/admin/**': {
        headers: {
          'X-Robots-Tag': 'noindex, nofollow'
        }
      }
    }
  }
})

Crawler Identification

Verify claimed crawler identity against the provider's policy. This DNS excerpt handles IPv4 Googlebot addresses with a googlebot.com hostname; it is not a verifier for every Google crawler or IPv6 address. DNS lookup failures propagate.

import { resolve4, reverse } from 'node:dns/promises'

export async function isLegitCrawler(ip: string, userAgent: string) {
  if (!userAgent.includes('Googlebot'))
    return false

  // reverse lookup: which hostname claims this IP?
  const [hostname] = await reverse(ip)
  if (!hostname?.endsWith('.googlebot.com'))
    return false

  // forward-confirm: does that hostname resolve back to the same IP?
  const ips = await resolve4(hostname)
  return ips.includes(ip)
}

A single reverse lookup can be spoofed; Google's own verification guide requires the forward-confirmation step above. For less code, skip DNS and compare the request IP against Google's published crawler IP ranges instead.

Rate Limiting

Use a limiter appropriate to your deployment. In-memory counters apply only to one process and need expiry; they do not provide a shared limit across workers or replicas.

Use nuxt-security for built-in rate limiting:

export default defineNuxtConfig({
  modules: ['nuxt-security'],
  security: {
    rateLimiter: {
      tokensPerInterval: 100,
      interval: 60000, // 1 minute
      headers: true
    }
  }
})

The module's rate-limiter documentation warns that its built-in limiter is intended for simpler applications. Choose infrastructure limits and shared storage for complex deployments. Do not treat a process-local map as a shared limit.

Infrastructure Security

HTTPS Enforcement

Configure HTTPS enforcement at your trusted deployment boundary. Redirect to your configured public origin; do not build a destination from an untrusted Host header. Check how your proxy supplies scheme and client IP information.

Security Headers

Configure security headers from your app’s resource and embedding requirements. A permissive CSP with unsafe-inline does not prevent inline script execution. Cross-origin restrictions can break intended resources or integrations. Follow the module’s current header documentation and test the policy in your deployment.

Or configure manually via route rules:

export default defineNuxtConfig({
  nitro: {
    routeRules: {
      '/**': {
        headers: {
          'X-Frame-Options': 'DENY',
          'X-Content-Type-Options': 'nosniff',
          'Referrer-Policy': 'strict-origin-when-cross-origin',

        }
      }
    }
  }
})

Monitoring & Detection

Record request rates, rejected requests, and response status in your application or infrastructure logs. Define detection rules for your traffic; undefined helpers such as isSuspiciousPattern are not an implementation.

Common Attacks

Content Scraping

Use rate limits appropriate to your deployment. Request counters need bounded storage and expiry. A user-agent string alone does not establish that a request is automated.

Form Spam

Validate form input and apply request limits at the endpoint. A honeypot can supplement these controls, but it does not authenticate a visitor or provide a shared rate limit. See the current XSS validator configuration; input checks do not replace safe output handling.

Checklist

Building the same protections in plain Vue? Protecting Vue Apps from Malicious Crawlers →