Benchmarking LLM Cost Routing: Lessons from 900 API Requests Across 9 Providers

You ship a feature, integrate a flagship model like GPT-4, and watch it work seamlessly. Six months later, a routine audit reveals a painful truth: half your incoming support traffic—simple requests like "where is my invoice?" or "reset my password"—is being processed by an expensive frontier model capable of passing medical board exams.

This is the default outcome of unrouted LLM architecture. Production support queues consist mostly of factual, straightforward queries. However, because developers pick the model that successfully solves their hardest edge cases, all traffic ends up routed through that top-tier model. The model isn't at fault—the missing routing layer is.

To solve this, developer Emil Igidov engineered CARDIAC-PURR, an automated LLM cost router designed to evaluate query complexity before hitting provider endpoints. By categorizing queries into Small, Medium, and Large model tiers across nine leading LLM providers, the system optimizes spend without compromising accuracy.

In this deep dive, we examine the production results of running 900 standardized queries across 9 API providers, analyze latency and failure modes, and demonstrate how platforms like n1n.ai provide the unified infrastructure required to deploy multi-provider routing strategies effectively.

The Core Concept: Pre-Inference Complexity Scoring

Unlike meta-model routers (such as OpenRouter's Auto Router) that send your prompt to an intermediate LLM to pick a destination—adding 200–500ms of latency—CARDIAC-PURR operates strictly pre-inference.

The router calculates a deterministic complexity score for each incoming prompt against calibrated internal thresholds (e.g., self.c_target set at 0.600 for the LARGE-tier gate). If a query scores well below the threshold, it is routed to a SMALL model tier; if it requires deeper context or complex logic, it escalates to MEDIUM or LARGE.

Incoming Query 
     │
     ▼
[ Complexity Evaluator ] ──( deterministic score < threshold )──► [ SMALL Tier Model ]
     │                                                                   │
     └──( score >= threshold )──► [ LARGE Tier Model ]                  ▼
                                                                  [ Weak/Truncated? ]
                                                                   ├── Yes ──► Retries via LARGE
                                                                   └── No  ──► Return Response

Safety Nets & Cascade Recovery

Routing algorithms must fail gracefully. CARDIAC-PURR implements a single-tier escalation fallback:

• If a SMALL tier model returns an incomplete or truncated answer (for instance, hitting output token limits), the router catches the condition and retries once against the LARGE tier model.
• If the upstream provider experiences a network timeout or 5xx error during retry, the system avoids propagating a hard 500 error to the client. It gracefully degrades by returning the pre-escalation response paired with diagnostic metadata (escalation_failed: true).

9-Provider Benchmark Matrix: Methodology & Setup

To establish an unskewed evaluation, 100 identical queries were executed against 9 major API providers. The dataset was structured to mimic real-world enterprise support and application workloads:

• 75% Factual/Definitions: "What does NDA stand for?"
• 17% Explanatory: "Describe the key operational steps in..."
• 8% Complex Reasoning: Legal compliance, financial disclosures, IT architecture.

Provider Model Tier Mapping

† Note on DeepSeek: DeepSeek routes both small and medium requests to deepseek-v4-flash, leaving deepseek-v4-pro strictly for high-complexity prompts.
§ Note on Grok: xAI routes all tiers to grok-4.3, varying only the reasoning_effort parameter (none/low/high).

To execute an architectural setup like this without managing nine distinct billing accounts, rate limits, and client SDKs, developers frequently leverage unified aggregators. Using n1n.ai, developers can route calls across DeepSeek, Claude, OpenAI, and Qwen endpoints via a single OpenAI-compatible API format, drastically simplifying multi-model routing code.

Key Benchmark Results & Analysis

The evaluation measured Routing Accuracy (matching expected complexity labels), Cost Savings (against an Always-LARGE baseline), API Error Rates, and Vertical Performance across Legal, Healthcare, Finance, and IT queries.

Baseline Accuracy: A naive router that hardcodes every request to "SMALL" achieves 75% accuracy by default based on this prompt distribution. 8 out of 9 providers achieved 100% routing accuracy.

Architectural Deep Dives: DeepSeek vs. Grok vs. Anthropic

1. Why DeepSeek Achieved 88.3% Savings

DeepSeek posted the highest cost savings in the benchmark. This efficiency stems directly from its pricing structure. Because deepseek-v4-flash costs a fraction of deepseek-v4-pro, and 92% of queries never required the PRO tier, real query costs averaged $0.0059 compared to the $0.0505 baseline of running pure PRO.

2. The Grok Pricing Ceiling (30.9% Savings)

Grok yielded the lowest financial savings despite hitting 100% routing accuracy. Because xAI maps all three tiers to the same underlying grok-4.3 model—adjusting only reasoning_effort—the small-tier cost ratio relative to large sits at 0.605 (compared to Anthropic's Haiku-to-Opus ratio of 0.052).

Without a distinct, scaled-down model architecture for light tasks, reasoning-knob adjustments alone cannot deliver deep structural savings.

Latency Profiles & Infrastructure Overhead

A critical question for enterprise developers: What is the computational overhead of adding a router layer?

The data reveals that pure router evaluation overhead is minimal (14.9ms to 16.7ms), whereas total end-to-end latency is overwhelmingly dictated by provider response times and model generation length.

† Cohere's inflated overhead figure was driven by two network read timeouts requiring upstream retries rather than routing execution delay.

Practical Implementation: Multi-Provider Routing with Python

To build a custom routing setup using n1n.ai, developers can leverage a unified client interface. Here is a production-grade Python snippet demonstrating complexity-based model dispatch with instant fallback:

import os
from openai import OpenAI

# Initialize client pointing to n1n.ai aggregator
client = OpenAI(
    api_key=os.environ.get("N1N_API_KEY"),
    base_url="https://api.n1n.ai/v1"
)

def evaluate_complexity(prompt: str) -> float: