Bot Detection

Introduction

The modern web is full of bots. Start detecting them to better understand your traffic and optimize your Nuxt app for both human users and automated agents.

Nuxt Robots provides bot detection that works on both server and client side, from simple heuristics using HTTP header User-Agent checks to advanced client-side detection using BotD.

Bot Categories

The module classifies bots into categories based on their intended purpose and trustworthiness. Trusted bots are legitimate services that respect robots.txt and provide value to websites, while untrusted bots include automation tools, scrapers, and potentially malicious crawlers.

Search Engine Bots (trusted)

Search engines that index content for public search results and respect standard web protocols.

• Google: googlebot, google.com/bot.html
• Bing: bingbot, msnbot

Social Media Bots (trusted)

Social platforms that crawl content for link previews, cards, and social sharing features.

• Facebook: facebookexternalhit, facebook.com
• Twitter: twitterbot, twitter

SEO & Analytics Bots (trusted)

Professional SEO and analytics services that provide legitimate website analysis and insights.

• Ahrefs: ahrefsbot, ahrefs.com
• Majestic: mj12bot, majestic12.co.uk/bot

AI & ML Bots (trusted)

AI companies and research organizations training models or providing AI-powered services.

• OpenAI: gptbot, openai.com
• Anthropic: anthropic

Automation Tools (untrusted)

Browser automation and testing frameworks that may be used for legitimate testing or malicious scraping.

• Selenium: selenium, webdriver
• Playwright: playwright

HTTP Tools (untrusted)

Command-line HTTP clients and programmatic request libraries often used for automated data extraction.

• cURL: curl
• Python Requests: python-requests, python

Security Scanners (untrusted)

Network scanning and vulnerability assessment tools that may indicate malicious reconnaissance.

• Nmap: nmap, insecure.org
• Nikto: nikto

Scraping Tools (untrusted)

Dedicated web scraping frameworks designed for automated data collection.

• Scrapy: scrapy, scrapy.org
• Generic Scraper: scraper

Missing a bot? Submit a quick PR :) View and contribute to bot definitions →

Nitro Bot Detection

Since server-side detection only uses HTTP headers, detection can only work for bots that correctly identify themselves in the User-Agent header.

You can detect bots inside a Nitro route, middleware, or API handler.

import { getBotDetection } from '#robots/server/composables/getBotDetection'

export default defineEventHandler((e) => {
  const detection = getBotDetection(e)

  if (detection.isBot) {
    return { message: `${detection.botName} bot detected`, category: detection.botCategory }
  }

  return { message: 'Human user' }
})

For full behavior, please consult the getBotDetection API docs.

Nuxt Bot Detection

When using bot detection in Nuxt, it will use the User-Agent header by default. You can optionally use the BotD fingerprinting library to detect advanced automation tools by setting fingerprint: true.

<script setup lang="ts">
import { useBotDetection } from '#robots/app/composables/useBotDetection'

const { isBot, botName, botCategory, trusted } = useBotDetection({
  fingerprint: true, // detects using botd
})
</script>

<template>
  <div v-if="isBot">
    Bot detected: {{ botName }} ({{ botCategory }})
  </div>
</template>

See the useBotDetection() API docs for full usage details.

Fingerprinting with BotD

When using fingerprint: true, the composable will load the BotD library when the window is idle and perform client-side fingerprinting to detect advanced bots and automation tools.

Performance Considerations

This fingerprinting is computationally expensive for end users' CPUs, so you should be mindful of when you enable it. For example, you may consider only enabling it for sensitive pages where bot detection is critical.

That said, the composable aims to be performant and will cache the bot result in the user's local storage under the '__nuxt_robots:botd' key so it will only run once.

localStorage.getItem('__nuxt_robots:botd') // returns the cached bot detection result - used internally already

Watching For Fingerprinting

The properties returned from the composable are all refs. It's important to watch these for changes if you're using fingerprinting, as the results will not be immediately available when the composable is called.

import { useBotDetection } from '#robots/app/composables/useBotDetection'
import { watch } from 'vue'

const { isBot } = useBotDetection({
  fingerprint: true,
})

watch(isBot, (detected) => {
  if (detected) {
    console.log(`Bot detected!`)
  }
})

Alternatively you can use the onFingerprintResult callback to handle the result when fingerprinting completes.

import { useBotDetection } from '#robots/app/composables/useBotDetection'

const botd = useBotDetection({
  fingerprint: true,
  onFingerprintResult(result) {
    // Fingerprinting completed
    console.log('Detection result:', result)
  },
})