Benchmarking Standard RAG, GraphRAG, and Agentic Pipelines on a Knowledge Graph

Ask a standard Retrieval-Augmented Generation (RAG) system a question like "How many biathlon events at the 2018 Winter Olympics had more than 73 competitors?" and it will provide a confident answer. However, that answer will almost certainly be incorrect. Resolving this query requires aggregating data scattered across 8 to 43 distinct documents. A top-8 vector similarity search simply cannot capture the full evidentiary context.

To analyze where knowledge graphs assist retrieval, where autonomous agents are necessary, and where simple pipelines suffice, we benchmarked three distinct architectures on identical hardware, the same underlying dataset, and the same LLM foundation (Claude 3.5 Sonnet).

Across a public benchmark of 100 structured questions, the Agentic GraphRAG pipeline achieved a 99% accuracy rate, whereas Standard Naive RAG achieved only 48%. Below is an in-depth analysis of the benchmark results, architectural breakdowns, code implementations, and economic trade-offs.

Comprehensive Benchmark Results

All three evaluation pipelines were tested on a corpus of 2,951 English Wikipedia articles covering Olympic events from 1987 to 2023, augmented with distractor documents. The underlying graph was constructed on TigerGraph Savanna using a deterministic parser on infoboxes, establishing an exact topology without LLM extraction errors.

Evaluation was managed by an isolated LLM judge grading against gold-standard labels after a deterministic syntax pre-check. LLM judge token usage was strictly excluded from pipeline metrics.

Cost Efficiency and Economic Analysis

While Agentic GraphRAG consumes 1.24x the total tokens of Naive RAG, it yields 2.06x higher accuracy. When measuring Tokens per Correct Answer (the true metric of token efficiency in production RAG systems), Agentic GraphRAG is 40% cheaper than Naive RAG.

Static GraphRAG boasts the lowest cost per correct answer (5,529 tokens), but its accuracy ceiling caps at 69%, making it inadequate for mission-critical enterprise workflows.

Benchmark Performance by Question Category

The evaluation benchmark splits 100 questions into five distinct archetypes to highlight pipeline edge cases:

1. Lookup (19 questions): Single-fact extraction from one document.
2. Aggregation (21 questions): Counting or calculating items across a full sport/event set based on a threshold.
3. Superlative (10 questions): Identifying maximum or minimum values across a global dataset.
4. Temporal (22 questions): Tracking entities backward or forward across chronological events (e.g., historical winners).
5. Multi-Hop (28 questions): Step-by-step entity resolution (e.g., matching venues, dates, and winners).

Key Architectural Takeaways

• Standard RAG Fails Hard at Aggregation: Naive vector top-k retrieval scored 0 out of 21 on aggregation questions. Vector embeddings rank semantic similarity, not relational metrics. Fetching 8 chunks cannot summarize 40 distinct documents.
• Static GraphRAG Degrades on Multi-Hop Queries: On multi-hop questions, static GraphRAG scored only 21%, falling far behind Naive RAG's 57%. Static traversal expands from initial seed entities using fixed rules. If vector search seeds the wrong initial entity, fixed expansion aggressively fetches irrelevant facts with high confidence.
• Agentic GraphRAG Handles Dynamic Multi-Hop Routing: The agent pipeline links entities step-by-step. For instance, on multi-hop questions, it first queries the venue vertex, filters by date, and finally inspects the matching event graph node.

Graph Schema and Topology

The graph topology combines structured graph relations with embedded chunk vectors within TigerGraph. The schema consists of the following vertices and edges:

• Vertices: Event, Sport, Games, Venue, Athlete, Country, Document, Chunk.
• Edges: IN_SPORT, AT_GAMES, HELD_AT, WON, PREV_EDITION, HAS_CHUNK.

Each Chunk vertex stores vector embeddings generated from text chunks. Vector similarity search executes natively inside the graph database, allowing joint vector-graph operations in a single engine.

// Sample GSQL: Aggregating competitors for a given sport & games threshold
CREATE QUERY count_events_by_competitors(STRING sport_name, STRING games_name, INT min_competitors) FOR GRAPH OlympicGraph {
  INT event_count = 0;
  
  Events = { Event.* };
  MatchedEvents = SELECT e FROM Events:e - (IN_SPORT) - Sport:s,
                         Events:e - (AT_GAMES) - Games:g
                  WHERE s.name == sport_name AND g.name == games_name AND e.num_competitors > min_competitors;
                  
  PRINT MatchedEvents.size() AS count;
}

Pipeline Architectures Deep Dive

[User Query]
     │
     ├──> 1. Standard RAG ───────> [ Top-8 Vector Chunks ] ──────> [ Single LLM Call ] ──> Output
     │
     ├──> 2. Static GraphRAG ────> [ Top-6 Vector Seed ] ────────> [ Fixed Graph Expansion ] ──> [ LLM Call ] ──> Output
     │
     └──> 3. Agentic GraphRAG ───> [ ReAct Loop / Tool Selection ]
                                         ├── GSQL Aggregation
                                         ├── Entity Linking & Vector Search
                                         └── Evidence Evaluator (Circuit Breaker)
                                         └── [ Final Evaluated Output ]

1. Standard Naive RAG

Executes vector search to retrieve the top-8 semantic matches (k=8). The retrieved chunks are concatenated directly into the prompt context of a single LLM invocation.

Failure Mode: When handling complex aggregation, the prompt context filled the 4,096-token output budget with irrelevant text context, causing context overflow and producing zero valid responses.

2. Static GraphRAG

Executes vector search for top-6 chunks, selects up to 3 seed event vertices, and performs a single fixed depth expansion across PREV_EDITION, WON, and sibling edges before passing the payload to the LLM.

3. Agentic GraphRAG Architecture

The agentic pipeline uses an LLM orchestrator running a step-by-step ReAct pattern. It is equipped with specific tool abstractions:

• entity_linking: Maps raw string entities to graph node IDs.
• gsql_aggregation: Runs parametrized GSQL queries for direct relational math.
• graph_traversal: Explores adjacent edges step-by-step.
• vector_search: Fallback semantic search on unstructured Chunk nodes.
• evidence_evaluator: Validates response correctness before output.

class AgenticGraphRAG:
    def __init__(self, llm_client, tigergraph_driver):
        self.llm = llm_client
        self.tg = tigergraph_driver
        self.max_steps = 8
        self.error_circuit_breaker = 2

    def run(self, query: str) -> str:
        consecutive_errors = 0
        trace = []
        
        for step in range(self.max_steps):
            action = self.llm.plan_next_step(query, trace)
            
            if action.type == "FINAL_ANSWER":
                # Evidence Evaluator Check
                if self.evaluate_evidence(action.payload, trace):
                    return action.payload
                else:
                    trace.append(\{"system_feedback": "Rejected: Missing valid document citations or unresolved gaps.