Why Your RAG Pipeline Fails in Production and How to Build a Continuous Evaluation System

Imagine this scenario: your engineering team builds a retrieval-augmented generation (RAG) chatbot over the company's internal policy documents. During the initial demo, a stakeholder asks, "How many days of parental leave do we get?" The bot answers correctly, citing the exact page of the benefits PDF. Another asks about travel expense limits—correct again. After ten questions and ten flawless answers, the team applauds and ships the system to production.

Three months later, an employee asks whether their contractor status qualifies for the health stipend. The bot returns a confident "Yes," assembling its answer from an outdated policy document that was superseded last year. The employee files a claim, only to have it rejected by HR.

Nobody on the development team can pinpoint when the pipeline started producing these incorrect answers. The system did not throw an exception, latency did not spike, and no database errors occurred. The pipeline failed silently because there was no apparatus in place to measure its accuracy over time. The demo was treated as the evaluation, which is the one test a RAG system essentially cannot fail.

The Silent Failure Modes of Production RAG

Traditional software architectures fail loudly. A broken database query throws a 500 error; a bad deployment triggers health check failures; a failing unit test turns the build pipeline red. A RAG regression does none of this. If you swap your embedding model, adjust your chunk size, or update the document index, the system continues to return fluent, well-formatted, and confidently cited answers. Whether those answers are actually grounded in the current truth is invisible to standard application monitoring.

This is not an isolated edge case. A CAIN 2024 experience report across three RAG case studies (spanning research, education, and biomedical domains) cataloged seven recurring failure points:

1. Missing Content: The required information is not present in the source corpus.
2. Missed Top-Ranked Documents: The information is present, but the retrieval step fails to rank it in the top-k results.
3. Lost in Consolidation: The information is retrieved but gets lost during context assembly due to model context window limitations.
4. Failure to Extract: The information is present in the context, but the LLM fails to extract it.
5. Wrong Format: The model ignores formatting instructions.
6. Wrong Specificity: The answer is either too generic or too detailed.
7. Incomplete Answers: The model addresses only part of a multi-hop query.

The authors of the study concluded with a stark realization: validation of a RAG system is only feasible during operation, and robustness must evolve over time rather than being fully designed in at the start. You cannot fully validate a RAG system before shipping it. Therefore, an evaluation harness is a core architectural requirement.

The Illusion of "Hallucination-Free" Retrieval

Many teams skip evaluation because they assume retrieval inherently solves the hallucination problem. The strongest counterexample comes from the legal technology domain, where accuracy is paramount. Legal research vendors marketed their RAG products as "eliminating" or "avoiding" hallucinations, with some even guaranteeing "hallucination-free" citations.

However, when Stanford's RegLab conducted the first preregistered empirical evaluation of these tools, they found that flagship products from major vendors hallucinated between 17% and 33% of the time. While retrieval reduced hallucinations compared to a bare LLM, one in six to one in three answers still contained fabricated information. If professional-grade systems built with massive resources over highly curated legal databases fail at this rate, your internal corporate chatbot is highly vulnerable.

Deconstructing RAG: Retrieval vs. Generation

To build an effective RAG evaluation pipeline, you must first understand that a RAG system can fail in two independent places. Lumping them together into a single "accuracy" score prevents effective optimization.

1. Retrieval Failures: The chunks passed to the LLM do not contain the correct answer. This happens because the document was not indexed, the embedding model failed to capture the semantic meaning, or the retrieval query was poorly formulated. Prompt engineering will not fix this; you must change your chunking strategy, embedding model, or retrieval algorithm (e.g., introducing hybrid search or re-ranking).
2. Generation Failures: The correct answer is present in the retrieved context, but the LLM ignores it, contradicts it, or hallucinates external information. Adjusting your vector database search parameters will not fix this; you must modify the system prompt, change the generator model, or implement output guardrails.

The RAGAS (Retrieval Augmented Generation Assessment) framework formalizes this separation by introducing distinct metrics for both steps:

Step-by-Step Implementation of an Evaluation Harness

Building a minimum viable evaluation harness does not require a dedicated research team. You can implement it using three core components: a golden dataset, an LLM-as-a-judge scoring script, and a regression gate.

Step 1: Define the Golden Dataset

Create a curated dataset of 50 to 100 real-world questions. Do not generate these solely using LLMs; collect them from user search logs, support tickets, and subject-matter experts. Each entry in the dataset should follow this schema:

[
  {
    "id": "policy_031",
    "question": "Does the health stipend apply to contractors?",
    "expected_answer": "No - eligibility requires full-time employment status.",
    "must_cite": ["benefits-eligibility-2026.pdf"],
    "trap": "superseded 2024 policy still in corpus says yes"
  }
]

The trap field is critical. It explicitly tests the pipeline's ability to handle edge cases, such as outdated documents, negative constraints, and queries where the correct response is "I do not know."

Step 2: Implement the LLM-as-a-Judge Script

To evaluate generation quality at scale, you can use an LLM as a judge. While human evaluation is the gold standard, it is too slow for continuous integration. By utilizing a high-performance LLM API aggregator like n1n.ai, you can access advanced models such as Claude 3.5 Sonnet or DeepSeek-V3 to score your pipeline's outputs.

Here is a complete Python implementation demonstrating how to evaluate the Faithfulness of a generated response against the retrieved context using the n1n.ai API interface:

import json
import requests

def evaluate_faithfulness(question, context, generated_answer, api_key):
    """
    Evaluates the faithfulness of a RAG response using Claude 3.5 Sonnet via n1n.ai.
    """
    url = "https://api.n1n.ai/v1/chat/completions"
    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json"
    }

    prompt = f"""
    You are an expert AI evaluator. Your task is to judge the faithfulness of a generated answer based on the provided context.

    Context:
    {context}

    Question:
    {question}

    Generated Answer:
    {generated_answer}

    Analyze the generated answer statement by statement. Determine if each statement is directly supported by the context.
    Provide your final output in JSON format with two keys:
    1. "reasoning": A brief explanation of your analysis.
    2. "score": A float value between 0.0 (entirely hallucinated) and 1.0 (completely faithful to the context).
    """

    payload = {
        "model": "anthropic/claude-3.5-sonnet",
        "messages": [
            {"role": "system", "content": "You evaluate RAG system outputs objectively based only on the provided context."},
            {"role": "user", "content": prompt}
        ],
        "temperature": 0.0,
        "response_format": {"type": "json_object"}
    }

    response = requests.post(url, headers=headers, json=payload)
    response.raise_for_status()
    result = response.json()

    return json.loads(result["choices"][0]["message"]["content"])

# Example usage:
mock_context = "Section 4.2: Full-time employees are eligible for the $500 annual health stipend. Contractors and part-time workers are excluded from this benefit."
mock_question = "Does the health stipend apply to contractors?"
mock_answer = "Yes, contractors can apply for the health stipend under the updated guidelines."

# Retrieve API key from your n1n.ai dashboard
API_KEY = "your_n1n_api_key_here"

eval_result = evaluate_faithfulness(mock_question, mock_context, mock_answer, API_KEY)
print(json.dumps(eval_result, indent=2))

Step 3: Set Up a Regression Gate

Run this evaluation script as a mandatory step in your deployment pipeline. If a change to your code—such as an updated prompt or a new chunking strategy—causes the evaluation score to drop below a predefined threshold (e.g., Faithfulness < 0.90), the build must fail. This prevents regressions from reaching production.

Evaluating the Evaluator: LLM Judge Trade-Offs

When using LLM-as-a-judge, you must account for known biases in model evaluations. The MT-Bench study highlighted several systematic biases:

• Verbosity Bias: Models tend to favor longer, more detailed responses, even if they contain irrelevant information.
• Self-Enhancement Bias: Some models score their own outputs higher than those generated by competing models.
• Position Bias: The order in which options are presented to the evaluator can influence the final score.

To mitigate these issues, you can leverage n1n.ai to route evaluations to different models (e.g., DeepSeek-V3 for cost-effective bulk evaluation, or Claude 3.5 Sonnet for complex reasoning tasks) and run periodic correlation tests against human-labeled samples.

Pro Tips for RAG Evaluation

• Isolate Vector DB Updates: When updating your vector database index, run your evaluation pipeline against a staging index first. This ensures that changes in document formatting or chunk parsing do not silently degrade retrieval recall.
• Track Latency alongside Accuracy: A high-accuracy RAG pipeline is unusable if it takes 15 seconds to respond. Measure retrieval latency and LLM generation latency separately during your evaluation runs.
• Log "No Answer" Triggers: Ensure your golden dataset includes questions that cannot be answered by the corpus. Your evaluation should verify that the system responds with a predefined fallback message (e.g., "I cannot find this information in the provided documents") rather than attempting to synthesize an answer.

Conclusion

A successful RAG deployment requires moving beyond the initial demo phase. By establishing a structured evaluation pipeline, you can detect regressions before they impact your users and systematically improve your system's retrieval and generation quality.

Get a free API key at n1n.ai