RAG Setup

Last updated Sep 15, 2026Harlan Wilton

Use llms-full.txt to build a small vector-search example from your published pages. Nuxt AI Ready exports the Markdown. Your script creates embeddings and stores them separately.

The example uses paid OpenAI API calls and an in-memory sqlite-vec database. Run it on a small set of short pages first. It retrieves context for RAG; it does not generate an answer or provide a production ingestion service.

Install the script dependencies and set OPENAI_API_KEY in your environment:

pnpm add -D openai better-sqlite3 sqlite-vec tsx @types/better-sqlite3

Put the Fetch, Generate, sqlite-vec, and Query examples below in one scripts/vectorize.ts file. Use Upstash instead of the SQLite sections only if you already have an Upstash Vector index.

Fetch Markdown Content

Each page starts with a --- boundary and metadata. The exporter removes leading YAML frontmatter and trims the Markdown body. This parser targets Nuxt AI Ready’s export format, not arbitrary Markdown. It checks the response and required metadata:

const RE_PAGE_BOUNDARY = /^---\n\n(?=- \*\*Page:\*\* )/gm
const RE_METADATA = /^- \*\*(Page|Source|Description):\*\* (.*)$/

export function parseLlmsFullTxt(content: string, llmsFullUrl: string) {
  const [, ...pageBlocks] = content.replace(/\r\n/g, '\n').split(RE_PAGE_BOUNDARY)
  return pageBlocks.map((block) => {
    const [metadataBlock = '', ...markdownParts] = block.trim().split('\n\n')
    const metadata = Object.fromEntries(
      metadataBlock.split('\n').flatMap((line) => {
        const match = RE_METADATA.exec(line)
        return match ? [[match[1], match[2]]] : []
      }),
    )
    if (!metadata.Page || !metadata.Source)
      throw new Error('A page needs Page and Source metadata.')

    const source = new URL(metadata.Source, llmsFullUrl)
    if (!['http:', 'https:'].includes(source.protocol))
      throw new Error('Page sources must use HTTP or HTTPS.')

    return {
      title: metadata.Page,
      description: metadata.Description || '',
      source: source.href,
      route: source.pathname,
      markdown: markdownParts.join('\n\n').trim(),
    }
  }).filter(page => page.markdown.length > 0)
}

const llmsFullUrl = 'https://yoursite.com/llms-full.txt'
const response = await fetch(llmsFullUrl)
if (!response.ok)
  throw new Error(`Markdown export request failed: ${response.status}`)

const pages = parseLlmsFullTxt(await response.text(), llmsFullUrl)
if (pages.length === 0)
  throw new Error('The export contains no non-empty pages.')

Generate Embeddings

This example uses text-embedding-3-small, whose default vectors have 1,536 dimensions. Keep that dimension consistent with your vector table. See OpenAI’s embedding guide.

Each input must fit the model’s token limit. Split long pages before this step; a heading split alone does not guarantee that limit. Calls run sequentially here. A provider error stops the script; it does not retry or resume automatically.

import OpenAI from 'openai'

const openai = new OpenAI()

async function embed(text: string) {
  const response = await openai.embeddings.create({
    model: 'text-embedding-3-small',
    input: text,
  })
  const embedding = response.data[0]?.embedding
  if (!embedding)
    throw new Error('The embedding response contains no vector.')
  return embedding
}

const vectors = []
for (const page of pages) {
  vectors.push({
    id: page.source,
    embedding: await embed(page.markdown),
    metadata: { title: page.title, source: page.source },
  })
}

Store in Vector DB

sqlite-vec (Local)

This table exists only while the script runs. Use it to test retrieval before choosing persistent storage. The sqlite-vec JavaScript guide covers loading the extension.

import Database from 'better-sqlite3'
import * as sqliteVec from 'sqlite-vec'

const db = new Database(':memory:')
sqliteVec.load(db)

db.exec(`
  CREATE VIRTUAL TABLE pages USING vec0(
    id TEXT PRIMARY KEY,
    embedding FLOAT[1536]
  )
`)

const insert = db.prepare('INSERT INTO pages VALUES (?, ?)')
for (const v of vectors) {
  insert.run(v.id, new Float32Array(v.embedding))
}

Upstash Vector (Serverless)

This is an alternative to SQLite, not another step in the local example. Install @upstash/vector and set UPSTASH_VECTOR_REST_URL and UPSTASH_VECTOR_REST_TOKEN for your index. Its configured dimensions must match the embedding model. See Upstash’s upsert reference.

import { Index } from '@upstash/vector'

const index = new Index()

await index.upsert(vectors.map(v => ({
  id: v.id,
  vector: v.embedding,
  metadata: v.metadata
})))

Query

For the local SQLite example, retrieve the closest vectors and recover their source Markdown. The k constraint uses sqlite-vec’s KNN query syntax.

async function search(query: string, topK = 5) {
  const queryEmbedding = await embed(query)

  const results = db.prepare(`
    SELECT id, distance
    FROM pages
    WHERE embedding MATCH ? AND k = ?
    ORDER BY distance
  `).all(new Float32Array(queryEmbedding), topK) as { id: string, distance: number }[]

  return results
}

const relevant = await search('how do I configure meta tags?')
const context = relevant.map((result) => {
  const page = pages.find(page => page.source === result.id)
  return page ? `Source: ${page.source}\n\n${page.markdown}` : ''
}).filter(Boolean).join('\n\n')
console.log(context)
db.close()

Chunking Strategy

The example embeds one short page at a time. For larger pages, create smaller inputs before generating embeddings. This simple heading splitter preserves heading text. Use a Markdown parser if headings can appear inside code fences:

const RE_HEADING_SPLIT = /(?=^##\s)/m

function chunkByHeading(markdown: string, route: string) {
  const sections = markdown.split(RE_HEADING_SPLIT)
  return sections.filter(section => section.trim()).map((section, i) => ({
    id: `${route}#${i}`,
    content: section.trim(),
    route
  }))
}

Strategy	When to use
Page-level	Short pages that fit the embedding model’s input limit
Heading-level	Long docs, precise retrieval needed
Sliding window	Dense technical content, overlap matters

Build Script

After prerendering, you can read the local export instead of fetching the deployed URL:

// scripts/vectorize.ts
import { readFileSync } from 'node:fs'

const llmsFull = readFileSync('.output/public/llms-full.txt', 'utf-8')
const pages = parseLlmsFullTxt(llmsFull, 'https://yoursite.com/llms-full.txt')

Replace the fetch block with that local read. Keep parseLlmsFullTxt and the remaining example steps. Run it after the static files exist:

{
  "scripts": {
    "generate": "nuxt generate && tsx scripts/vectorize.ts"
  }
}

Was this page helpful?

Edit page Report an issue

llms.txt Configuration

Markdown Output

On this page