---
title: "Protecting Vue Apps from Malicious Crawlers · Nuxt SEO"
canonical_url: "https://nuxtseo.com/learn-seo/vue/routes-and-rendering/security"
last_updated: "2026-07-16T12:00:00.000Z"
meta:
  author: "Harlan Wilton"
  description: "Robots.txt is a suggestion; malicious crawlers ignore it. Real protection is middleware, headers, and DNS verification."
  "og:description": "Robots.txt is a suggestion; malicious crawlers ignore it. Real protection is middleware, headers, and DNS verification."
  "og:title": "Protecting Vue Apps from Malicious Crawlers · Nuxt SEO"
---

Nuxt SEO on GitHub

# **Protecting Vue Apps from Malicious Crawlers**

Robots.txt is a suggestion; malicious crawlers ignore it. Real protection is middleware, headers, and DNS verification.

[Harlan Wilton](https://x.com/harlan-zw)8 mins read Published **Nov 3, 2024** Updated **Jul 16, 2026**

**What you'll learn**

- Robots.txt is a suggestion, not a security control; use server middleware for actual protection
- Block non-production environments by setting `**X-Robots-Tag: noindex**` from middleware before requests reach your app
- Verifying a crawler by IP needs both a reverse and a forward DNS lookup; a single lookup can be spoofed

[**~~Robots.txt~~**](https://nuxtseo.com/learn-seo/vue/controlling-crawlers/robots-txt) and meta robots tags are polite suggestions. Malicious crawlers ignore them.

You need actual security: block non-production environments, protect development assets, rate limit aggressive crawlers, authenticate sensitive routes, and use HTTPS everywhere. Don't rely on robots.txt for sensitive data, IP blocking alone (easily bypassed), or user-agent detection (trivial to fake).

Vue itself doesn't own a server, so all of this lives in whatever backend renders your app: Express, Fastify, h3, or your host's edge functions. The examples below use Express since it's the most common pairing.

## Quick Setup

Protect your Vue app from unwanted crawlers at the server level:

Express Middleware

```ts
// server/middleware/security.js
import express from 'express'

const app = express()

// block non-production environments
app.use((req, res, next) => {
  if (process.env.NODE_ENV !== 'production') {
    res.setHeader('X-Robots-Tag', 'noindex, nofollow')
  }
  next()
})

// enforce HTTPS
app.use((req, res, next) => {
  if (req.headers['x-forwarded-proto'] !== 'https') {
    return res.redirect(301, `https://${req.headers.host}${req.url}`)
  }
  next()
})
```

Security Headers

```ts
// add security headers to your server
import helmet from 'helmet'

app.use(helmet({
  frameguard: { action: 'deny' },
  contentSecurityPolicy: {
    directives: {
      defaultSrc: ['\'self\''],
      styleSrc: ['\'self\'', '\'unsafe-inline\''],
      scriptSrc: ['\'self\'']
    }
  },
  referrerPolicy: { policy: 'strict-origin-when-cross-origin' }
}))
```

See [**~~Rate Limiting~~**](#rate-limiting) below for throttling aggressive traffic.

## Environment Protection

### Development & Staging

Always block search engines in non-production environments:

```ts
// middleware/block-non-production.js
app.use((req, res, next) => {
  const isProd = process.env.NODE_ENV === 'production'
  const isMainDomain = req.headers.host === 'mysite.com'

  if (!isProd || !isMainDomain) {
    res.setHeader('X-Robots-Tag', 'noindex, nofollow')

    // also consider basic auth for staging
    const auth = req.headers.authorization

    if (!auth) {
      res.setHeader('WWW-Authenticate', 'Basic')
      return res.status(401).send('Authentication required')
    }
  }
  next()
})
```

### Sensitive Routes

Protect admin and user areas:

```ts
// middleware/protect-routes.js
app.use((req, res, next) => {
  const protectedPaths = ['/admin', '/dashboard', '/user']

  if (protectedPaths.some(path => req.path.startsWith(path))) {
    // ensure the user is authenticated
    if (!req.session?.user) {
      return res.redirect('/login')
    }

    // block indexing of protected content
    res.setHeader('X-Robots-Tag', 'noindex, nofollow')
  }
  next()
})
```

## Crawler Identification

Good crawlers publish verifiable IPs. Bad ones fake a `**Googlebot**` user agent and hope you don't check.

```ts
// utils/verify-crawler.js
import dns from 'node:dns'
import { promisify } from 'node:util'

const reverse = promisify(dns.reverse)
const resolve4 = promisify(dns.resolve4)

export async function isLegitCrawler(ip, userAgent) {
  if (!userAgent.includes('Googlebot'))
    return false

  // reverse lookup: which hostnames claim this IP?
  const hostnames = await reverse(ip)
  const hostname = hostnames.find(h => h.endsWith('.googlebot.com'))
  if (!hostname)
    return false

  // forward-confirm: does that hostname resolve back to the same IP?
  const ips = await resolve4(hostname)
  return ips.includes(ip)
}
```

A single reverse lookup can be spoofed; [**~~Google's own verification guide~~**](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot) requires the forward-confirmation step above. For less code, skip DNS and compare the request IP against Google's [**~~published crawler IP ranges~~**](https://developers.google.com/static/search/apis/ipranges/googlebot.json) instead.

## Rate Limiting

```ts
import rateLimit from 'express-rate-limit'

const apiLimiter = rateLimit({
  windowMs: 15 * 60 * 1000,
  max: 100
})

// tighter limit for requests carrying a bot-like user agent
const crawlerLimiter = rateLimit({
  windowMs: 60 * 1000,
  max: 10,
  skip: req => !req.headers['user-agent']?.includes('bot')
})

app.use('/api', apiLimiter)
app.use(crawlerLimiter)
```

## Infrastructure Security

### HTTPS Enforcement

Redirect HTTP to HTTPS before anything else runs:

```ts
app.use((req, res, next) => {
  const proto = req.headers['x-forwarded-proto']

  if (proto === 'http') {
    return res.redirect(301, `https://${req.headers.host}${req.url}`)
  }
  next()
})
```

### Security Headers

Add security headers using helmet:

```ts
import helmet from 'helmet'

app.use(helmet({
  // prevent clickjacking
  frameguard: { action: 'deny' },
  // prevent MIME type sniffing
  noSniff: true,
  // control referrer information
  referrerPolicy: { policy: 'strict-origin-when-cross-origin' },
  // enable strict CSP in production
  contentSecurityPolicy: process.env.NODE_ENV === 'production'
    ? {
        directives: {
          defaultSrc: ['\'self\'']
        }
      }
    : false
}))
```

## Monitoring & Detection

Log suspicious patterns so you know what you're blocking, not just that you're blocking something:

```ts
// middleware/crawler-monitor.js
app.use((req, res, next) => {
  const ua = req.headers['user-agent']
  const ip = req.ip

  if (isSuspiciousPattern(ua, ip)) {
    console.warn(`Suspicious crawler: ${ip} with UA: ${ua}`)
  }
  next()
})
```

If you're running a small blog, a full WAF (Cloudflare, AWS WAF) is overkill. Add one once you're getting attacked, not before.

## Common Attacks

### Content Scraping

Prevent automated content theft:

```ts
const requestCounts = new Map()

app.use((req, res, next) => {
  const ip = req.ip
  const count = requestCounts.get(ip) || 0

  if (count > 100) {
    return res.status(429).send('Too Many Requests')
  }

  requestCounts.set(ip, count + 1)

  // add slight delays to automated requests
  if (isBot(req.headers['user-agent'])) {
    setTimeout(next, 500)
  }
  else {
    next()
  }
})
```

### Form Spam

Protect forms from bot submissions:

```ts
// routes/contact.js
app.post('/api/contact', async (req, res) => {
  const { website, ...formData } = req.body

  // honeypot check
  if (website) { // hidden field
    return res.json({ success: false })
  }

  // rate limiting
  if (exceedsRateLimit(req.ip)) {
    return res.status(429).json({
      error: 'Too many attempts'
    })
  }

  // process legitimate submission
  // ...
})
```

## Checklist

**Checklist**

- Non-production environments blocked from indexing before any route handler runs
- Sensitive routes ( `**/admin**`, `**/dashboard**`) require authentication and send `**noindex**`
- Rate limits configured on `**/api/***` and form endpoints
- Security headers ( `**X-Frame-Options**`, CSP, `**Referrer-Policy**`) set via helmet or equivalent
- HTTPS enforced for every request
- Crawler identity verified with both reverse and forward DNS lookups, or Google's published IP ranges

Using Nuxt instead of a custom Vue backend? Nuxt handles the environment blocking, headers, and rate limiting above through modules instead of hand-rolled middleware. [**~~Protecting Nuxt Apps from Malicious Crawlers →~~**](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security)

[**The 2026 SEO Checklist for Nuxt & Vue ** Pre-launch setup, post-launch verification, and ongoing monitoring. Interactive checklist with links to every guide.](https://nuxtseo.com/learn-seo/checklist) [Haven't launched yet? Start with the **Pre-Launch Warmup**](https://nuxtseo.com/learn-seo/pre-launch-warmup)

---

### **Related **

[**Web Crawler Basics**](https://nuxtseo.com/learn-seo/vue/controlling-crawlers)

[**Robots.txt Guide**](https://nuxtseo.com/learn-seo/vue/controlling-crawlers/robots-txt)

[**Meta Robots**](https://nuxtseo.com/learn-seo/vue/controlling-crawlers/meta-tags)

[**Programmatic SEO** Generate SEO-ready feature, comparison, and alternative pages in Vue without triggering Google's scaled content penalties.](https://nuxtseo.com/learn-seo/vue/routes-and-rendering/programmatic-seo) [**SSR Frameworks** Compare Vue server-side rendering frameworks. Choose between Nuxt, Quasar, Vite SSR, and VitePress for better SEO and performance.](https://nuxtseo.com/learn-seo/vue/ssr-frameworks)

**On this page**

- [Quick Setup](#quick-setup)
- [Environment Protection](#environment-protection)
- [Crawler Identification](#crawler-identification)
- [Rate Limiting](#rate-limiting)
- [Infrastructure Security](#infrastructure-security)
- [Monitoring & Detection](#monitoring-detection)
- [Common Attacks](#common-attacks)
- [Checklist](#checklist)