Protecting Vue Apps from Malicious Crawlers

Robots.txt and meta robots tags are polite suggestions. Malicious crawlers ignore them.

You need actual security: block non-production environments, protect development assets, rate limit aggressive crawlers, authenticate sensitive routes, and use HTTPS everywhere. Don't rely on robots.txt for sensitive data, IP blocking alone (easily bypassed), or user-agent detection (trivial to fake).

Vue itself doesn't own a server, so all of this lives in whatever backend renders your app: Express, Fastify, h3, or your host's edge functions. The examples below use Express since it's the most common pairing.

Quick Setup

Protect your Vue app from unwanted crawlers at the server level:

// server/middleware/security.ts
import express from 'express'

const app = express()

// Request search exclusion for non-production environments
app.use((req, res, next) => {
  if (process.env.NODE_ENV !== 'production') {
    res.setHeader('X-Robots-Tag', 'noindex, nofollow')
  }
  next()
})

// Configure HTTPS redirects at your host using a fixed public origin.
// Do not construct destinations from untrusted forwarded headers.

// add security headers to your server
import helmet from 'helmet'

app.use(helmet({
  frameguard: { action: 'deny' },
  contentSecurityPolicy: {
    directives: {
      defaultSrc: ['\'self\''],
      styleSrc: ['\'self\'', '\'unsafe-inline\''],
      scriptSrc: ['\'self\'']
    }
  },
  referrerPolicy: { policy: 'strict-origin-when-cross-origin' }
}))

See Rate Limiting below for throttling aggressive traffic.

Environment Protection

Development & Staging

If staging contains private data, require real authentication through your host or application. Sending X-Robots-Tag does not restrict access. Do not accept an Authorization header without validating its credentials.

For example, Cloudflare Access can protect a self-hosted staging application. Configure an explicit access policy. Add noindex as a separate indexing control.

Sensitive Routes

This excerpt assumes your server has configured and validated session middleware. The user field comes from that application integration. Install authentication before this route check:

// middleware/protect-routes.js
app.use((req, res, next) => {
  const protectedPaths = ['/admin', '/dashboard', '/user']

  if (protectedPaths.some(path => req.path.startsWith(path))) {
    // ensure the user is authenticated
    if (!req.session?.user) {
      return res.redirect('/login')
    }

    // block indexing of protected content
    res.setHeader('X-Robots-Tag', 'noindex, nofollow')
  }
  next()
})

Crawler Identification

Good crawlers publish verifiable IPs. This example covers common Googlebot requests over IPv4. Other Google crawler categories use different hostname suffixes. Use the appropriate published IP ranges for IPv6 support. Bad ones fake a Googlebot user agent and hope you don't check.

// utils/verify-crawler.js
import dns from 'node:dns'
import { promisify } from 'node:util'

const reverse = promisify(dns.reverse)
const resolve4 = promisify(dns.resolve4)

export async function isLegitCrawler(ip, userAgent) {
  if (!userAgent.includes('Googlebot'))
    return false

  // reverse lookup: which hostnames claim this IP?
  const hostnames = await reverse(ip)
  const hostname = hostnames.find(h => h.endsWith('.googlebot.com'))
  if (!hostname)
    return false

  // forward-confirm: does that hostname resolve back to the same IP?
  const ips = await resolve4(hostname)
  return ips.includes(ip)
}

A single reverse lookup can be spoofed; Google's own verification guide requires the forward-confirmation step above. For less code, skip DNS and compare the request IP against Google's published crawler IP ranges instead.

Rate Limiting

Install the middleware:

pnpm add express express-rate-limit

These excerpts use the Express app created in Quick Setup. The default memory store covers one process. Use a shared store for multiple instances. Configure proxy trust correctly before deriving client IPs. Limits below are examples, not universal safe values.

import rateLimit from 'express-rate-limit'

const apiLimiter = rateLimit({
  windowMs: 15 * 60 * 1000,
  max: 100
})

// tighter limit for requests carrying a bot-like user agent
const crawlerLimiter = rateLimit({
  windowMs: 60 * 1000,
  max: 10,
  skip: req => !req.headers['user-agent']?.includes('bot')
})

app.use('/api', apiLimiter)
app.use(crawlerLimiter)

Infrastructure Security

HTTPS Enforcement

Configure HTTPS redirects at your host. Use your configured public origin as the destination. If Express runs behind a proxy, configure trust proxy for that topology. Do not trust arbitrary X-Forwarded-Host or X-Forwarded-Proto values.

Security Headers

Add security headers using helmet:

import helmet from 'helmet'

app.use(helmet({
  // prevent clickjacking
  frameguard: { action: 'deny' },
  // prevent MIME type sniffing
  noSniff: true,
  // control referrer information
  referrerPolicy: { policy: 'strict-origin-when-cross-origin' },
  // enable strict CSP in production
  contentSecurityPolicy: process.env.NODE_ENV === 'production'
    ? {
        directives: {
          defaultSrc: ['\'self\'']
        }
      }
    : false
}))

Monitoring & Detection

Log suspicious patterns to see which requests your controls block:

// middleware/crawler-monitor.js
app.use((req, res, next) => {
  const ua = req.headers['user-agent']
  const ip = req.ip

  if (isSuspiciousPattern(ua, ip)) {
    console.warn(`Suspicious crawler: ${ip} with UA: ${ua}`)
  }
  next()
})

Common Attacks

Content Scraping

Rate limits can reduce automated requests. They do not prevent all content copying. Use a limiter with an expiring window and a bounded store, as shown in Rate Limiting. Avoid counters that accumulate forever.

Form Spam

This form-handler excerpt assumes JSON body parsing, input parsing, and an application-owned exceedsRateLimit check. A honeypot alone does not validate a submission:

// routes/contact.js
app.post('/api/contact', async (req, res) => {
  const { website, ...formData } = req.body

  // honeypot check
  if (website) { // hidden field
    return res.json({ success: false })
  }

  // rate limiting
  if (exceedsRateLimit(req.ip)) {
    return res.status(429).json({
      error: 'Too many attempts'
    })
  }

  // process legitimate submission
  // ...
})

Checklist

Using Nuxt instead of a custom Vue backend? Configure authentication, headers, and rate limits explicitly for your Nuxt host and application. Protecting Nuxt Apps from Malicious Crawlers →