Evaluating LLM Performance with Real Production Traffic Replays

Many development teams fall into the trap of relying solely on public benchmarks like MMLU, HumanEval, or LMSYS Chatbot Arena when deciding to swap their Large Language Model (LLM) provider. For instance, you might see that a newly released model like DeepSeek-V3 or Claude 3.5 Sonnet outperforms your current model on paper, prompting you to plan an immediate migration. However, a demo that passes local tests often fails in production under real-world usage patterns.

To bridge this gap, you need a testing methodology that reflects your actual workload. The most reliable benchmark is not synthetic; it is the real traffic your users generate every day. By building a traffic replay harness, you can capture live requests, replay them against a candidate model, and diff the outputs to make an informed, data-driven migration decision. In this guide, we will build a complete, five-stage traffic replay harness using Node.js. We will also show how to leverage n1n.ai to seamlessly access and test different candidate models (such as OpenAI o3, DeepSeek-V3, and Claude 3.5 Sonnet) using a single, unified API key.

Why Public Benchmarks Fail in Production

Public benchmarks evaluate models on generic datasets. They do not account for the unique characteristics of your production system. Here is why synthetic benchmarks fail to tell the whole story:

1. System Prompt Sensitivity: A prompt optimized for GPT-4o might cause Claude 3.5 Sonnet to output unexpected formatting or refuse the request entirely.
2. RAG Context Windows: In Retrieval-Augmented Generation (RAG) setups, the way a model processes long, noisy context documents varies wildly. A model with high MMLU scores might struggle with "needle in a haystack" retrieval in your specific domain.
3. Schema Adherence: If your application relies on structured JSON outputs, a candidate model might introduce subtle schema violations that break downstream parsers.
4. Latency Profiles: High-concurrency environments expose latency spikes that average benchmark scores hide. You need to know how the model performs under your specific traffic load.

Using a unified API aggregator like n1n.ai allows you to test multiple model endpoints without changing your codebase's core integration logic. This makes it the ideal companion for a traffic replay harness.

Prerequisites

To follow this tutorial, you will need:

• A server with Node.js 18 or newer installed.
• A current production endpoint (the LLM model you run today).
• A candidate endpoint (such as a model accessed via n1n.ai).
• One hour of time and a small JSONL file to store the captured traffic.

Stage 1: Running a Logging Proxy

The first step is to record live production traffic. We will run a lightweight, non-blocking reverse proxy in front of your existing production LLM endpoint. This proxy intercepts incoming chat completion requests, forwards them to the production provider, and logs the request-response pairs to a local JSON Lines (.jsonl) file.

Create a file named capture-proxy.mjs. Node.js treats .mjs files as ES modules, which allows us to use top-level await syntax.

// capture-proxy.mjs
import { createServer } from 'node:http'
import { appendFile } from 'node:fs/promises'

const TARGET = process.env.TARGET_URL
const LOG = process.env.LOG_FILE || './traffic.jsonl'
const PORT = process.env.PORT || 3000

const server = createServer(async (req, res) => {
  const chunks = []
  for await (const chunk of req) chunks.push(chunk)
  const raw = Buffer.concat(chunks).toString('utf8')

  const started = Date.now()
  const upstream = await fetch(TARGET, {
    method: req.method,
    headers: { 'content-type': 'application/json' },
    body: raw || undefined,
  })
  const upstreamBody = await upstream.text()
  const latency = Date.now() - started

  if (req.method === 'POST' && req.url.includes('/chat/completions')) {
    const record = {
      ts: new Date().toISOString(),
      path: req.url,
      status: upstream.status,
      latency_ms: latency,
      request: JSON.parse(raw),
      response: JSON.parse(upstreamBody),
    }
    await appendFile(LOG, JSON.stringify(record) + '\n', 'utf8')
  }

  res.writeHead(upstream.status, { 'content-type': 'application/json' })
  res.end(upstreamBody)
})

server.listen(PORT, () => console.log(`Capture proxy running on port ${PORT}`))

Verify Stage 1

Run the proxy in the background and send a test request using curl:

node capture-proxy.mjs &
curl -s -X POST localhost:3000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"hello"}]}'
wc -l traffic.jsonl   # Expect 1 line in the log file

Pro Tip on Security: Request bodies can contain sensitive user data or API keys. Before running this proxy in production, implement a redaction step in the code to strip out Personally Identifiable Information (PII) or sensitive tokens before writing to the log file.

Stage 2: Replaying Traffic Against the Candidate Model

Once you have collected a representative sample of traffic (e.g., 24 hours of logs), you can replay these requests against the candidate model. The replay script reads traffic.jsonl line by line, sends each request payload to the candidate endpoint, and saves the results.

We will buffer the full response before parsing it. Note that streaming chunks should be handled differently; for this harness, we evaluate the complete payload to ensure structural comparison.

Create replay.mjs:

// replay.mjs
import { readFile, appendFile } from 'node:fs/promises'

const [logFile, target, apiKey] = process.argv.slice(2)
const lines = (await readFile(logFile, 'utf8')).trim().split()

for (const line of lines) {
  if (!line) continue
  const record = JSON.parse(line)
  const started = Date.now()
  const res = await fetch(target, {
    method: 'POST',
    headers: {
      'content-type': 'application/json',
      authorization: `Bearer ${apiKey}`,
    },
    body: JSON.stringify(record.request),
  })
  const latency = Date.now() - started
  const text = await res.text()

  let response = null
  try {
    response = JSON.parse(text)
  } catch {
    /* keep null on failure */
  }

  await appendFile(
    'results.jsonl',
    JSON.stringify({
      ts: record.ts,
      status: res.status,
      latency_ms: latency,
      response,
    }) + '\n',
    'utf8'
  )
}

Verify Stage 2

Run the replay script against your candidate endpoint. You can use the unified API gateway at n1n.ai to easily target different models:

node replay.mjs traffic.jsonl "https://api.n1n.ai/v1/chat/completions" "$N1N_API_KEY"
wc -l results.jsonl   # Must match the line count of traffic.jsonl

Keeping the order stable is critical. The diff script joins both files by line number. A failed parse stays null in the results, which acts as a signal rather than crashing the harness.

Stage 3: Diffing the Outputs

Now we compare the production outputs against the candidate outputs. We evaluate status codes, latency, content parity, and token usage. Create diff.mjs:

// diff.mjs
import { readFile } from 'node:fs/promises'

const traffic = (await readFile('traffic.jsonl', 'utf8')).trim().split('\n').map(JSON.parse)
const results = (await readFile('results.jsonl', 'utf8')).trim().split('\n').map(JSON.parse)

const rows = traffic.map((t, i) => {
  const r = results[i] ?? {}
  const prod = t.response?.choices?.[0]?.message?.content ?? null
  const cand = r.response?.choices?.[0]?.message?.content ?? null
  return {
    request: i,
    prod_status: t.status,
    cand_status: r.status,
    prod_ms: t.latency_ms,
    cand_ms: r.latency_ms,
    exact_match: prod === cand,
    both_valid: prod !== null && cand !== null,
    prod_tokens: t.response?.usage?.total_tokens ?? null,
    cand_tokens: r.response?.usage?.total_tokens ?? null,
  }
})

console.table(rows)

const valid = rows.filter((x) => x.cand_status === 200 && x.both_valid).length
const exact = rows.filter((x) => x.exact_match).length
console.log(`valid: ${valid}/${rows.length}`)
console.log(`exact match: ${exact}/${rows.length}`)

Verify Stage 3

Run the diff script:

node diff.mjs
# Expected output format:
# valid: 47/50
# exact match: 41/50

A count mismatch means the replay failed partway. Investigate the logs before trusting the diff. Exact match is a strong signal for extraction, classification, and formatting jobs, but it is a weak signal for creative tasks. Always manually inspect at least three mismatches to understand the qualitative differences.

Stage 4: Automating the Process

Because models drift and user behavior changes, a one-off test is not enough. You should run this replay harness periodically. You can schedule it as a cron job to run every six hours on a lightweight server:

0 */6 * * * cd /opt/replay && node replay.mjs traffic.jsonl "https://api.n1n.ai/v1/chat/completions" "$N1N_API_KEY" >> replay.log 2>&1

Run the diff after every replay and archive the last seven reports. This acts as an early warning system before you execute the final model swap in your production environment.

Stage 5: Establishing the Decision Matrix

To determine if a model swap is safe, you must establish clear, quantitative thresholds. Do not rely on average values; one slow outlier can skew the mean. Use medians to keep your metrics honest.

Rules for Swapping

1. The Single-Failure Rule: A single failing signal in the reject threshold must block the swap.
2. The All-Pass Rule: All four signals must pass the switch threshold to proceed. A model that is extremely fast but produces invalid content is still a failure.

Advanced Considerations for Enterprise LLM Swaps

1. Tokenizer Differences and Cost Implications

Different models use different tokenizers. For example, OpenAI's GPT-4o uses the o200k_base tokenizer, while other models might use different compression ratios. This means the exact same prompt and response could consume 20% more tokens on a candidate model, directly impacting your API costs. Always calculate the token delta during the diff stage to project the financial impact of the migration.

2. Handling Streaming Outputs (SSE)

If your production application relies on streaming (Server-Sent Events), capturing and replaying requests becomes more complex. In production, you must buffer the stream chunks in your logging proxy to reconstruct the full JSON response before writing it to traffic.jsonl. This ensures your diff script can compare the complete text output.

3. Evaluating Structured Outputs

If your application uses LLMs to generate structured data (e.g., JSON mode or function calling), a simple string comparison (exact_match) will fail. Instead, parse the outputs and validate them against your JSON Schema. A candidate model that returns the correct data but with different key ordering is still a successful match.

Summary

Replay testing is not a generic benchmark; it is a custom regression test built specifically for your application. By capturing real production traffic and replaying it against candidate models via n1n.ai, you can make model swap decisions based on hard data rather than marketing claims.

Get a free API key at n1n.ai