NuxtSEOBot · Nuxt SEO [NuxtSEO](https://nuxtseo.com/ "Home") - [Modules](https://nuxtseo.com/docs/nuxt-seo/getting-started/introduction) - [Tools](https://nuxtseo.com/tools) - [Pro](https://nuxtseo.com/pro) - [Learn SEO](https://nuxtseo.com/learn-seo/nuxt) [Releases](https://nuxtseo.com/releases) [Sign in](https://nuxtseo.com/pro/dashboard/login) [Nuxt SEO on GitHub](https://github.com/harlan-zw/nuxt-seo) Crawler information # NuxtSEOBot Nuxt SEO Pro audits sites their owners have added and verified. If NuxtSEOBot is on your site, someone with access to it asked us to look. ## How to identify us We send this User-Agent on every request: ``` NuxtSEOBot/1.0 (+https://nuxtseo.com/bot) ``` Treat that as a label, not as proof. Anything can send any User-Agent, so matching on it tells you where to look in a log and nothing more. There is one exception worth knowing about: when we render a page in a headless browser, Cloudflare's Browser Rendering service sends its own fixed `CloudflareBrowserRenderingCrawler/1.0` and will not let us override it. Those requests still carry the signature below. ## How to verify us Every request we make to your origin is signed with [Web Bot Auth](https://datatracker.ietf.org/doc/draft-meunier-webbotauth-httpsig-protocol/), an IETF draft built on [RFC 9421 HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421.html). Three headers arrive with each one: ``` Signature-Agent: "https://nuxtseo.com" Signature-Input: sig1=("@authority" "signature-agent");created=;keyid="";alg="ed25519";expires=;tag="web-bot-auth" Signature: sig1=:: ``` The signature is an Ed25519 signature over your own hostname and the `Signature-Agent` value. Our public key lives at as a JWKS, and the `keyid` is that key's [RFC 7638 thumbprint](https://www.rfc-editor.org/rfc/rfc7638.html). Because the signature covers the hostname, a signature captured from your site cannot be replayed against anyone else's. It expires 15 minutes after it is issued. ### Verifying it yourself Fetch the JWKS above, find the key whose `kid` matches the request's `keyid`, and verify the signature per RFC 9421. Cloudflare publishes an open-source [web-bot-auth](https://github.com/cloudflare/web-bot-auth) library that does this in about ten lines, and it is the same library our own test suite verifies against. ### If you are behind Cloudflare Cloudflare validates Web Bot Auth signatures itself and surfaces the result to WAF rules as `cf.bot_management.signed_agent`, with no verification code on your side. That only fires for crawlers registered in Cloudflare's own bot directory, and NuxtSEOBot's registration is not yet approved, so today the field will be false for us. Until it lands, verify directly or match on the `Signature-Agent` header. ## What we do not offer - **A fixed IP range.** We run on Cloudflare Workers, which egress from a shared pool with no addresses reserved to us. Any IP list we published would be wrong, and would also cover every other Workers customer. This is the reason we sign. - **A secret in the URL.** A `?token=` lands in your access logs, your CDN logs, your analytics, and every `Referer` header your pages send. If you want a shared secret, put it in a header (see below), never in the query string. - **Lighthouse traffic.** Performance audits are run by Google's PageSpeed Insights API. Those requests come from Google's servers under Google's User-Agent, not ours, and we cannot sign them or make any promise about them. - **DataForSEO's crawler.** We buy backlink, keyword and rank data from DataForSEO, and their crawler builds that index from their own machines. If you see `136.243.220.208/29`, `136.243.228.176/29` or `136.243.228.192/29` (Hetzner, AS24940) in your logs, that is `DataForSeoBot`, not us. It crawls whether or not you use Nuxt SEO, we never ask it to fetch your site on demand, and it carries none of our headers. Verify it by reverse DNS ( `crawling-gateway-*.dataforseo.com`), or turn it away with `User-agent: DataForSeoBot` in your robots.txt, which it obeys. Blocking it does not affect the crawl or the audit we run for you. ## If you'd rather not verify signatures Verifying a signature means writing code. If you'd rather add one line to a firewall rule, a Nuxt SEO Pro user can set custom request headers per site, under Settings → Crawl. We send them with every crawl request to that site, and you match on them however you already match on things. Be clear-eyed about the trade-off. A shared secret travels in cleartext to your origin on every request and lands in your access logs and any proxy in between; whoever can read those can replay it. A signature cannot be replayed against anyone else and expires in fifteen minutes. We support the header because a real rule you actually deploy beats a perfect one you don't, and we bound the damage as far as we can: - Headers go to _your site's exact hostname_ and nowhere else. A redirect to a different host, **including a subdomain of your own site**, drops them. - They are encrypted at rest, and our API never returns a value once saved, not even to you. Rotate one by typing a new value. - They are **not sent on JavaScript-rendered crawls**, which run in a browser that would replay them to every third-party host your pages load resources from. Those requests still carry the signature. - You cannot set `User-Agent`, `Signature`, or the other headers that carry our identity. ## How we crawl - We crawl a site only after someone has added it to Nuxt SEO Pro and proved they control it, either through Google Search Console or by serving a verification token. We do not crawl the open web. - We rate-limit per hostname and back off when a host answers `429` or `503`. A host that keeps refusing us gets progressively less traffic on every subsequent crawl, and we honour `Retry-After`. - We fetch HTML pages on the verified host, and we send one request to each image and outbound link we find so we can report which of them are broken. That second kind of request lands on third-party hosts, which is why you may see us on a site you never registered. - We never submit forms, send `POST` requests, or attempt to reach anything behind a login. - A Nuxt SEO Pro user can exclude paths from their own site's crawl (Settings → Crawl). We do not fetch an excluded URL at all, so if you see us somewhere you asked us not to go, that is a bug worth an email. ## robots.txt Our JavaScript-rendering crawl runs on Cloudflare Browser Rendering, which enforces `robots.txt` and will not fetch a disallowed path. Our plain HTML crawl does not currently consult `robots.txt` before fetching, because it only ever runs against a host whose owner added it and asked for the audit, and an audit that silently skipped the pages a site disallows would report a health score for a site that is not the one being served. If that reasoning does not hold for your site, tell us and we will fix it. Either way, `robots.txt` is a crawling directive and not an access control. RFC 9309 says so explicitly. To actually stop us, block at the network layer. ## How to block us Block on the verified signature, not on the User-Agent. Matching the User-Agent only blocks requests honest enough to announce themselves, and it will not match our browser-rendered crawl at all. If we are causing you trouble, the fastest fix is usually to email us rather than to write a rule. We would rather turn the crawl down. ## Contact If NuxtSEOBot is causing you trouble, email [harlan@nuxtseo.com](mailto:harlan@nuxtseo.com) and it will be a person who replies. In short One product token, `NuxtSEOBot`. Every request signed with Ed25519 over your hostname. Public key at `/.well-known/http-message-signatures-directory`. No IP list, and no shared secret required. Lighthouse and PageSpeed checks reach you from Google, not from us. [GitHub](https://github.com/harlan-zw/nuxt-seo) [Discord](https://discord.com/invite/275MBUBvgP) ### [NuxtSEO](https://nuxtseo.com/ "Home") - [Getting Started](https://nuxtseo.com/docs/nuxt-seo/getting-started/introduction) - [MCP](https://nuxtseo.com/docs/nuxt-seo/guides/mcp) Modules - [Robots](https://nuxtseo.com/docs/robots/getting-started/introduction) - [Sitemap](https://nuxtseo.com/docs/sitemap/getting-started/introduction) - [OG Image](https://nuxtseo.com/docs/og-image/getting-started/introduction) - [Schema.org](https://nuxtseo.com/docs/schema-org/getting-started/introduction) - [Link Checker](https://nuxtseo.com/docs/link-checker/getting-started/introduction) - [SEO Utils](https://nuxtseo.com/docs/seo-utils/getting-started/introduction) - [Site Config](https://nuxtseo.com/docs/site-config/getting-started/introduction) - [Skew Protection](https://nuxtseo.com/docs/skew-protection/getting-started/introduction) - [AI Ready](https://nuxtseo.com/docs/ai-ready/getting-started/introduction) ### [NuxtSEO Pro](https://nuxtseo.com/pro "Nuxt SEO Pro") - [Getting Started](https://nuxtseo.com/pro) - [Dashboard](https://nuxtseo.com/pro/dashboard) - [Pro MCP](https://nuxtseo.com/pro/dashboard/mcp) ### [Learn SEO](https://nuxtseo.com/learn-seo "Learn SEO") Nuxt - [Mastering Meta](https://nuxtseo.com/learn-seo/nuxt/mastering-meta) - [Controlling Crawlers](https://nuxtseo.com/learn-seo/nuxt/controlling-crawlers) - [Launch & Listen](https://nuxtseo.com/learn-seo/nuxt/launch-and-listen) - [Routes & Rendering](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering) - [Staying Secure](https://nuxtseo.com/learn-seo/nuxt/routes-and-rendering/security) Vue - [Vue SEO Guide](https://nuxtseo.com/learn-seo/vue) - [Mastering Meta](https://nuxtseo.com/learn-seo/vue/mastering-meta) - [Controlling Crawlers](https://nuxtseo.com/learn-seo/vue/controlling-crawlers) - [SPA SEO](https://nuxtseo.com/learn-seo/vue/spa) - [SSR Frameworks](https://nuxtseo.com/learn-seo/vue/ssr-frameworks) - [SEO Checklist](https://nuxtseo.com/learn-seo/checklist) - [Pre-Launch Warmup](https://nuxtseo.com/learn-seo/pre-launch-warmup) - [Backlinks & Authority](https://nuxtseo.com/learn-seo/backlinks) ### [Tools](https://nuxtseo.com/tools "SEO Tools") - [Social Share Debugger](https://nuxtseo.com/tools/social-share-debugger) - [Robots.txt Generator](https://nuxtseo.com/tools/robots-txt-generator) - [Meta Tag Checker](https://nuxtseo.com/tools/meta-tag-checker) - [HTML to Markdown](https://nuxtseo.com/tools/html-to-markdown) - [XML Sitemap Validator](https://nuxtseo.com/tools/xml-sitemap-validator) - [Schema.org Validator](https://nuxtseo.com/tools/schema-validator) - [Keyword Idea Generator](https://nuxtseo.com/tools/keyword-generator) - [Keyword Research](https://nuxtseo.com/tools/keyword-research) - [SERP Analyzer](https://nuxtseo.com/tools/serp-analyzer) - [Domain Rankings](https://nuxtseo.com/tools/domain-rankings) Copyright © 2023-2026 Harlan Wilton - [MIT License](https://github.com/harlan-zw/nuxt-seo/blob/main/license) · [Privacy](https://nuxtseo.com/privacy) · [mdream](https://mdream.dev)