Building Resilient RAG Pipelines with Next.js and pgvector

“How to design a fault-tolerant multi-provider Retrieval-Augmented Generation system with fallback models, semantic vector caching, and streaming responses in Next.js.”

Building production-ready Retrieval-Augmented Generation (RAG) applications requires significantly more than simply connecting an embedding model to a vector database. When dealing with variable LLM latencies, rate limits, and contextual drift, resilience must be baked directly into every layer of the architecture.

In this deep dive, we explore how to architect a fault-tolerant RAG pipeline using Next.js Server-Sent Events (SSE), PostgreSQL with the pgvector extension, and an automatic provider fallback strategy.

1. The Core Architecture Challenge#

Standard RAG architectures often suffer from three major vulnerabilities:

01Single-point-of-failure LLM routing: If OpenAI or Anthropic suffers an outage or triggers a rate limit, the user request fails completely.
02High vector query latency: Inefficient indexing or unindexed embedding tables cause slow cosine similarity scans under high load.
03Inefficient context hydration: Injecting excessive or irrelevant chunk tokens degrades LLM reasoning and inflates token cost.

code

[ User Query ] ──► [ Semantic Cache ] ──► (Hit? Return Cached Stream)
                         │ (Miss)
                         ▼
             [ Embedding Generator ]
                         │
                         ▼
             [ pgvector HNSW Search ] ──► [ Top-K Filter & Re-rank ]
                         │
                         ▼
             [ Multi-Provider LLM Fallback Chain ]
                 ├─► Primary: Gemini 2.5 Flash
                 ├─► Secondary: Groq (Llama 3.3)
                 └─► Fallback: OpenRouter / Anthropic
                         │
                         ▼
             [ Streaming SSE Response to Next.js ]

2. Setting Up pgvector with HNSW Indexing#

When storing vector embeddings in Postgres, choosing the right index type is critical for maintaining sub-15ms search latencies as your document count scales into millions of chunks.

While IVFFlat is common, HNSW (Hierarchical Navigable Small World) provides superior query recall and throughput without requiring periodic index retraining:

sql

-- Enable vector extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create knowledge base document chunks table
CREATE TABLE document_chunks (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id UUID NOT NULL,
    content TEXT NOT NULL,
    token_count INT NOT NULL,
    embedding vector(1536), -- Dimension matching text-embedding-3-small or equivalent
    metadata JSONB DEFAULT '{}'::jsonb,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

-- Build an HNSW index using Cosine Distance
CREATE INDEX ON document_chunks 
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

With this index in place, similarity searches can be executed using SQL functions with cosine operator (<=>):

sql

CREATE OR REPLACE FUNCTION match_document_chunks (
    query_embedding vector(1536),
    match_threshold float,
    match_count int
)
RETURNS TABLE (
    id UUID,
    content TEXT,
    similarity float
)
LANGUAGE plpgsql
AS $$
BEGIN
    RETURN QUERY
    SELECT
        document_chunks.id,
        document_chunks.content,
        1 - (document_chunks.embedding <=> query_embedding) AS similarity
    FROM document_chunks
    WHERE 1 - (document_chunks.embedding <=> query_embedding) > match_threshold
    ORDER BY document_chunks.embedding <=> query_embedding
    LIMIT match_count;
END;
$$;

3. Implementing the Multi-Provider Fallback Chain#

A resilient system does not rely on a single AI provider. We can implement a cascading fallback handler that tracks provider health and cooldown periods:

typescript

interface ProviderConfig {
  name: string;
  execute: (prompt: string, context: string) => Promise<ReadableStream>;
  cooldownUntil?: number;
}

export class ResilientLLMService {
  private providers: ProviderConfig[];

  constructor(providers: ProviderConfig[]) {
    this.providers = providers;
  }

  async generateStream(prompt: string, context: string): Promise<ReadableStream> {
    const now = Date.now();

    for (const provider of this.providers) {
      if (provider.cooldownUntil && provider.cooldownUntil > now) {
        console.warn(`[LLM] Skipping ${provider.name} (in cooldown)`);
        continue;
      }

      try {
        console.info(`[LLM] Attempting stream with ${provider.name}...`);
        return await provider.execute(prompt, context);
      } catch (error) {
        console.error(`[LLM] Provider ${provider.name} failed: `, error);
        // Put failing provider on a 60-second cooldown
        provider.cooldownUntil = Date.now() + 60_000;
      }
    }

    throw new Error("All AI providers in fallback chain exhausted.");
  }
}

4. Streaming via Next.js Route Handlers#

Using Next.js API Routes, we can stream the response tokens directly back to the client using Server-Sent Events (SSE):

typescript

// pages/api/chat.ts
import type { NextApiRequest, NextApiResponse } from "next";

export default async function handler(req: NextApiRequest, res: NextApiResponse) {
  if (req.method !== "POST") return res.status(405).end();
  const { query } = req.body;

  // 1. Generate query embedding & retrieve pgvector matches
  const queryEmbedding = await getEmbedding(query);
  const chunks = await searchPgVector(queryEmbedding, { threshold: 0.72, limit: 5 });
  const context = chunks.map((c) => c.content).join("\n---\n");

  // 2. Stream tokens via SSE
  res.writeHead(200, {
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache, no-transform",
    Connection: "keep-alive",
  });

  const tokenStream = await llmService.generateStream(query, context);
  for await (const chunk of tokenStream) {
    res.write(`data: ${JSON.stringify({ token: chunk })}\n\n`);
  }
  res.end();
}

5. Key Takeaways & Recommendations#

01Semantic Caching: Cache query embeddings in Redis with exact and fuzzy cosine matching (> 0.96) to serve identical questions instantaneously without paying LLM costs.
02Chunking Strategy: Overlapping chunks of 400–600 tokens with 50-token overlaps provide optimal context retention for technical documentation without dilution.
03Provider Cooldowns: Implementing short dynamic cooldown periods prevents recurring request timeouts during localized API degraded states.