---
title: "Protecting Vue Apps from Malicious Crawlers"
description: "Robots.txt is a suggestion; malicious crawlers ignore it. Real protection is middleware, headers, and DNS verification."
canonical_url: "https://nuxtseo.com/learn-seo/vue/routes-and-rendering/security"
last_updated: "2026-07-16"
---

<key-takeaways>

- Robots.txt is a suggestion, not a security control; use server middleware for actual protection
- Block non-production environments by setting `X-Robots-Tag: noindex` from middleware before requests reach your app
- Verifying a crawler by IP needs both a reverse and a forward DNS lookup; a single lookup can be spoofed

</key-takeaways>

[Robots.txt](/learn-seo/vue/controlling-crawlers/robots-txt) and meta robots tags are polite suggestions. Malicious crawlers ignore them.

You need actual security: block non-production environments, protect development assets, rate limit aggressive crawlers, authenticate sensitive routes, and use HTTPS everywhere. Don't rely on robots.txt for sensitive data, IP blocking alone (easily bypassed), or user-agent detection (trivial to fake).

Vue itself doesn't own a server, so all of this lives in whatever backend renders your app: Express, Fastify, h3, or your host's edge functions. The examples below use Express since it's the most common pairing.

## Quick Setup

Protect your Vue app from unwanted crawlers at the server level:

```ts [Express Middleware]
// server/middleware/security.js
import express from 'express'

const app = express()

// block non-production environments
app.use((req, res, next) => {
  if (process.env.NODE_ENV !== 'production') {
    res.setHeader('X-Robots-Tag', 'noindex, nofollow')
  }
  next()
})

// enforce HTTPS
app.use((req, res, next) => {
  if (req.headers['x-forwarded-proto'] !== 'https') {
    return res.redirect(301, `https://${req.headers.host}${req.url}`)
  }
  next()
})
```

```ts [Security Headers]
// add security headers to your server
import helmet from 'helmet'

app.use(helmet({
  frameguard: { action: 'deny' },
  contentSecurityPolicy: {
    directives: {
      defaultSrc: ['\'self\''],
      styleSrc: ['\'self\'', '\'unsafe-inline\''],
      scriptSrc: ['\'self\'']
    }
  },
  referrerPolicy: { policy: 'strict-origin-when-cross-origin' }
}))
```

See [Rate Limiting](#rate-limiting) below for throttling aggressive traffic.

## Environment Protection

### Development & Staging

Always block search engines in non-production environments:

```ts
// middleware/block-non-production.js
app.use((req, res, next) => {
  const isProd = process.env.NODE_ENV === 'production'
  const isMainDomain = req.headers.host === 'mysite.com'

  if (!isProd || !isMainDomain) {
    res.setHeader('X-Robots-Tag', 'noindex, nofollow')

    // also consider basic auth for staging
    const auth = req.headers.authorization

    if (!auth) {
      res.setHeader('WWW-Authenticate', 'Basic')
      return res.status(401).send('Authentication required')
    }
  }
  next()
})
```

### Sensitive Routes

Protect admin and user areas:

```ts
// middleware/protect-routes.js
app.use((req, res, next) => {
  const protectedPaths = ['/admin', '/dashboard', '/user']

  if (protectedPaths.some(path => req.path.startsWith(path))) {
    // ensure the user is authenticated
    if (!req.session?.user) {
      return res.redirect('/login')
    }

    // block indexing of protected content
    res.setHeader('X-Robots-Tag', 'noindex, nofollow')
  }
  next()
})
```

## Crawler Identification

Good crawlers publish verifiable IPs. Bad ones fake a `Googlebot` user agent and hope you don't check.

```ts
// utils/verify-crawler.js
import dns from 'node:dns'
import { promisify } from 'node:util'

const reverse = promisify(dns.reverse)
const resolve4 = promisify(dns.resolve4)

export async function isLegitCrawler(ip, userAgent) {
  if (!userAgent.includes('Googlebot'))
    return false

  // reverse lookup: which hostnames claim this IP?
  const hostnames = await reverse(ip)
  const hostname = hostnames.find(h => h.endsWith('.googlebot.com'))
  if (!hostname)
    return false

  // forward-confirm: does that hostname resolve back to the same IP?
  const ips = await resolve4(hostname)
  return ips.includes(ip)
}
```

A single reverse lookup can be spoofed; [Google's own verification guide](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot) requires the forward-confirmation step above. For less code, skip DNS and compare the request IP against Google's [published crawler IP ranges](https://developers.google.com/static/search/apis/ipranges/googlebot.json) instead.

## Rate Limiting

```ts
import rateLimit from 'express-rate-limit'

const apiLimiter = rateLimit({
  windowMs: 15 * 60 * 1000,
  max: 100
})

// tighter limit for requests carrying a bot-like user agent
const crawlerLimiter = rateLimit({
  windowMs: 60 * 1000,
  max: 10,
  skip: req => !req.headers['user-agent']?.includes('bot')
})

app.use('/api', apiLimiter)
app.use(crawlerLimiter)
```

## Infrastructure Security

### HTTPS Enforcement

Redirect HTTP to HTTPS before anything else runs:

```ts
app.use((req, res, next) => {
  const proto = req.headers['x-forwarded-proto']

  if (proto === 'http') {
    return res.redirect(301, `https://${req.headers.host}${req.url}`)
  }
  next()
})
```

### Security Headers

Add security headers using helmet:

```ts
import helmet from 'helmet'

app.use(helmet({
  // prevent clickjacking
  frameguard: { action: 'deny' },
  // prevent MIME type sniffing
  noSniff: true,
  // control referrer information
  referrerPolicy: { policy: 'strict-origin-when-cross-origin' },
  // enable strict CSP in production
  contentSecurityPolicy: process.env.NODE_ENV === 'production'
    ? {
        directives: {
          defaultSrc: ['\'self\'']
        }
      }
    : false
}))
```

## Monitoring & Detection

Log suspicious patterns so you know what you're blocking, not just that you're blocking something:

```ts
// middleware/crawler-monitor.js
app.use((req, res, next) => {
  const ua = req.headers['user-agent']
  const ip = req.ip

  if (isSuspiciousPattern(ua, ip)) {
    console.warn(`Suspicious crawler: ${ip} with UA: ${ua}`)
  }
  next()
})
```

<tip>

If you're running a small blog, a full WAF (Cloudflare, AWS WAF) is overkill. Add one once you're getting attacked, not before.

</tip>

## Common Attacks

### Content Scraping

Prevent automated content theft:

```ts
const requestCounts = new Map()

app.use((req, res, next) => {
  const ip = req.ip
  const count = requestCounts.get(ip) || 0

  if (count > 100) {
    return res.status(429).send('Too Many Requests')
  }

  requestCounts.set(ip, count + 1)

  // add slight delays to automated requests
  if (isBot(req.headers['user-agent'])) {
    setTimeout(next, 500)
  }
  else {
    next()
  }
})
```

### Form Spam

Protect forms from bot submissions:

```ts
// routes/contact.js
app.post('/api/contact', async (req, res) => {
  const { website, ...formData } = req.body

  // honeypot check
  if (website) { // hidden field
    return res.json({ success: false })
  }

  // rate limiting
  if (exceedsRateLimit(req.ip)) {
    return res.status(429).json({
      error: 'Too many attempts'
    })
  }

  // process legitimate submission
  // ...
})
```

## Checklist

<checklist id="vue-security">

- Non-production environments blocked from indexing before any route handler runs
- Sensitive routes (`/admin`, `/dashboard`) require authentication and send `noindex`
- Rate limits configured on `/api/*` and form endpoints
- Security headers (`X-Frame-Options`, CSP, `Referrer-Policy`) set via helmet or equivalent
- HTTPS enforced for every request
- Crawler identity verified with both reverse and forward DNS lookups, or Google's published IP ranges

</checklist>

Using Nuxt instead of a custom Vue backend? Nuxt handles the environment blocking, headers, and rate limiting above through modules instead of hand-rolled middleware. [Protecting Nuxt Apps from Malicious Crawlers →](/learn-seo/nuxt/routes-and-rendering/security)