Building a Production RAG Chatbot with Claude, pgvector, and FastAPI

Retrieval-Augmented Generation (RAG) has moved from a research curiosity to a standard architectural pattern for enterprise AI. While many developers reach for complex frameworks like LangChain or LlamaIndex immediately, you can actually build a genuinely robust, production-ready RAG system in a single weekend using three reliable components: PostgreSQL (with the pgvector extension) as your vector store, a FastAPI service as the orchestration layer, and Claude (accessed via high-speed providers like n1n.ai) for the reasoning step.

The beauty of this stack lies in its simplicity. RAG is essentially a database lookup wrapped in a sophisticated prompt. By using tools you likely already know—Postgres and FastAPI—you eliminate the operational overhead of learning specialized vector databases while maintaining the performance required for production traffic.

The Core Architecture

A RAG system has a simple loop: at query time, you transform a user's question into a numerical vector (embedding), find the most relevant document chunks from your database, and feed those chunks into a Large Language Model (LLM) as context.

1. Vector Store: pgvector allows you to store and query embeddings directly within Postgres. If you are already running Postgres, you don't need a separate infrastructure piece.
2. Orchestration: FastAPI handles the API requests, input validation with Pydantic, and coordinates the flow between the embedding model, the database, and the LLM.
3. Reasoning: Claude 3.5 Sonnet or Claude 3 Haiku handles the final generation. For production environments, using an aggregator like n1n.ai ensures that you have the lowest latency and highest availability for these models.

The Embedding Gap: Bringing Your Own Vectors

One critical detail often overlooked is that the Anthropic API does not currently offer an embeddings endpoint. Claude is a reasoning engine, not an embedding model. To bridge this gap, you must use a dedicated embedding provider.

While OpenAI's text-embedding-3-small is popular, many developers are moving toward Voyage AI (specifically voyage-3) because it is optimized for RAG and pairs exceptionally well with Claude's reasoning style. Regardless of your choice, the embedding model defines your database schema. If your model outputs 1024-dimension vectors, your Postgres column must match that exactly.

import voyageai

vo = voyageai.Client(api_key="YOUR_VOYAGE_API_KEY")

def get_embeddings(texts: list[str], input_type: str) -> list[list[float]]:
    # Use "document" for indexing and "query" for searching
    result = vo.embed(texts, model="voyage-3", input_type=input_type)
    return result.embeddings

Setting Up the Vector Database

With pgvector, your vector search is just another SQL query. First, enable the extension and create a table with an HNSW (Hierarchical Navigable Small World) index. HNSW is superior to IVFFlat for production because it offers a better balance between search speed and recall accuracy without requiring a training step.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE document_chunks (
    id        bigserial PRIMARY KEY,
    metadata  jsonb,
    content   text NOT NULL,
    embedding vector(1024) NOT NULL
);

-- Create an HNSW index for fast cosine similarity search
CREATE INDEX ON document_chunks USING hnsw (embedding vector_cosine_ops);

The Ingestion Pipeline

Ingestion involves splitting documents into manageable pieces. A naive approach is splitting by character count, but for production, you should aim for "semantic chunks" or at least use a recursive character splitter to avoid cutting off sentences mid-thought.

import psycopg
from pgvector.psycopg import register_vector

def ingest_document(dsn: str, text: str, metadata: dict):
    # Simple chunking logic
    chunks = [text[i:i+1000] for i in range(0, len(text), 900)]
    vectors = get_embeddings(chunks, input_type="document")

    with psycopg.connect(dsn) as conn:
        register_vector(conn)
        with conn.cursor() as cur:
            cur.executemany(
                "INSERT INTO document_chunks (content, embedding, metadata) VALUES (%s, %s, %s)",
                [(c, v, psycopg.types.json.Json(metadata)) for c, v in zip(chunks, vectors)],
            )
        conn.commit()

Retrieval and Generation with Claude

Once your data is indexed, the retrieval step uses the cosine distance operator (<=>) to find the most similar chunks. We then pass these to Claude via n1n.ai to generate the final response.

import anthropic

# Pro Tip: Use n1n.ai for unified access to multiple LLM providers
client = anthropic.Anthropic(api_key="YOUR_API_KEY")

def generate_answer(dsn: str, question: str):
    # 1. Embed the question
    [query_vec] = get_embeddings([question], input_type="query")

    # 2. Retrieve top 5 matches
    with psycopg.connect(dsn) as conn:
        register_vector(conn)
        with conn.cursor() as cur:
            cur.execute(
                "SELECT content FROM document_chunks ORDER BY embedding <=> %s LIMIT 5",
                (query_vec,)
            )
            context = "\n\n".join([row[0] for row in cur.fetchall()])

    # 3. Generate with Claude
    system_prompt = "Answer only using the provided context. If unknown, say so."
    message = client.messages.create(
        model="claude-3-5-sonnet-20240620",
        max_tokens=1024,
        system=system_prompt,
        messages=[{"role": "user", "content": f"Context: {context}\n\nQuestion: {question}"}]
    )
    return message.content[0].text

Production Hardening: Beyond the Weekend

While the code above works for a demo, a production system requires more rigour:

1. Connection Pooling: Use psycopg_pool to handle concurrent database connections efficiently. Opening a new connection for every API request will crash your database under load.
2. Evaluation (RAGAS): You cannot optimize what you don't measure. Use a framework like RAGAS to evaluate "faithfulness" (did Claude hallucinate?) and "relevance" (was the retrieved context actually useful?).
3. Hybrid Search: Sometimes keyword search (BM25) outperforms vector search for specific terms like product IDs or names. Postgres allows you to combine both using Reciprocal Rank Fusion (RRF).
4. Model Selection: For high-volume, low-cost tasks, Claude 3 Haiku is unbeatable. For complex reasoning over large contexts, Claude 3.5 Sonnet is the gold standard. Accessing both through n1n.ai allows you to switch models with a single line of code change as your needs evolve.

Summary

Shipping a RAG bot doesn't require a massive infrastructure overhaul. By leveraging the stability of Postgres and the intelligence of Claude, you can build a system that scales from a weekend project to an enterprise tool. Focus your energy on the quality of your chunks and the precision of your system prompt—that is where the real value of RAG is created.

Get a free API key at n1n.ai