Improving RAG Recall with LLM Distillation and Comment Bursting

In the evolution of any Retrieval-Augmented Generation (RAG) system, there comes a point where simple vector search hits a wall. You might find that your Mean Reciprocal Rank (MRR) is respectable, but the system fails to surface the 'correct' answer when it is buried deep within a long, noisy conversation thread. This post explores a fundamental rebuild of the Cerebras Knowledge Base, focusing on two advanced techniques: LLM Distillation and Comment Bursting.

The Diagnosis: The Drowning Answer Problem

Previously, our vector search implementation achieved an MRR of 0.77. However, we identified a recurring failure mode: the 'drowning answer.' In long GitHub issues or Slack threads, the actual resolution—the high-signal comment that solves the problem—is often just one of eighty comments. When we embed the entire thread as a single vector, the specific semantic signal of the solution is averaged into oblivion by the surrounding noise of 'me too' comments and irrelevant logs.

To solve this, we implemented a two-pronged strategy to increase our recall ceiling, even at the temporary cost of precision.

Strategy 1: LLM Distillation

Distillation involves using a high-performance LLM to rewrite messy, unstructured threads into clean, structured Question-and-Answer documents. For every issue thread in our corpus, we now trigger an LLM call to generate a summary that includes the core problem, the eventual resolution, and any relevant technical symbols (function names, error codes).

We utilize a strict JSON schema for this process. If the model fails to return a usable summary, we fall back to the raw content. Out of 3,002 threads, 2,542 distilled cleanly (~85%), while 15% required a fallback.

When implementing this at scale, performance is critical. Using a high-speed aggregator like n1n.ai allows developers to handle these thousands of distillation calls in parallel without hitting traditional rate limits. By leveraging n1n.ai, you can ensure that the ingestion pipeline remains cost-effective and fast.

Strategy 2: Comment Bursting

While distillation cleans the text, 'Bursting' changes the architecture of the vector space. Instead of one document per thread, we 'burst' high-signal comments into their own vector-only rows.

• The Mechanism: Each resolving comment gets its own embedding (e.g., issue_N#burst_i).
• The Constraint: Bursts are excluded from keyword search and IDF statistics to avoid skewing the global frequency counts.
• Canonicalization: At scoring time, these rows canonicalize back to their parent issue. This ensures a single thread never occupies two slots in the top-K results.

This increased our document count from ~3,700 to 16,315, providing the retriever with many more 'entry points' into the same data.

The Results: A Counter-Intuitive Decline

After rebuilding with these techniques, we re-ran our evaluation against a fixed set of 31 questions. The results were surprising:

At first glance, the rebuild made things worse. MRR dropped from 0.77 to 0.63. Why? Because distillation trades verbatim matching for semantic tidiness. By summarizing the thread, we threw away the specific 'tracebacks' and 'literal error strings' that keyword search and dense vectors previously relied on.

For example, a query like TypeError: Object of type int64 is not JSON serializable was previously a top hit because the literal string existed in the raw thread. In the distilled version, the LLM might have summarized this as 'integer serialization error,' losing the exact match signal.

The Silver Lining: The Recall Ceiling

However, one number moved in the right direction: Hybrid Recall@10 climbed from 0.90 to 0.94.

This is the 'Useful Failure.' While our precision (ordering) suffered, our recall ceiling (the ability to find the answer at all) improved. We now have the correct document in the top 10 for 29 out of 31 questions. The problem shifted from 'the answer isn't there' to 'the answer is there, but it is not at the top.'

Implementation Pro-Tip: Handling the Rerank

This shift in metrics perfectly prepares the system for a Reranker. A rank-based fusion (like RRF) is confidence-blind; it doesn't know the difference between a cosine similarity of 0.95 and 0.70 if they are both ranked #1. By using a reranker in the next phase, we can look at the top 20 candidates retrieved by this high-recall system and re-order them.

When building these pipelines, developers often struggle with the latency of multiple LLM calls. Using n1n.ai provides access to the fastest models available, ensuring that adding a reranking step doesn't destroy the user experience.

Conclusion

Rebuilding for recall often feels like a step backward because precision metrics drop. But in RAG, recall is the ceiling of your system's potential. By using LLM distillation to clean the data and bursting to expose hidden signals, we have built a pile of candidates that are finally ready to be sorted accurately.

Get a free API key at n1n.ai