Why LLM Temperature 0 Is Not Reproducible: The Hidden Impact of Dynamic Batching

Many AI engineers assume that setting temperature=0 (or temperature=0.0) on a Large Language Model (LLM) guarantees 100% deterministic output. Under this belief, sending the exact same prompt, document, and hyperparameters to models like DeepSeek-V3, Claude 3.5 Sonnet, or OpenAI o3 should yield byte-identical results every single time.

However, empirical testing reveals a unsettling reality: identical requests with temperature=0 can experience up to a 30% difference in output between runs when executed under concurrent conditions.

The root cause of this variance is not sampling randomness or greedy decoding failure. It is GPU-level dynamic batching and the non-associative nature of floating-point arithmetic in parallel CUDA kernels.

If you build data extraction pipelines, evaluation benchmarks, or automated agentic workflows via unified LLM routing platforms like n1n.ai, understanding this non-determinism is vital. In this guide, we break down the empirical data, explain the underlying GPU mechanics, and detail actionable techniques to achieve true reproducibility.

The Empirical Discovery: Batching vs. Sampling

To isolate the root cause of temperature 0 variance, an experiment was designed around a structured data extraction task. The pipeline extracted key information records from a test corpus of documents. The model, prompt, document text, and temperature setting (0.0) were kept strictly constant.

The benchmark measured reproducibility using the Jaccard Similarity Index at the extracted record level:

$\\text\{Jaccard\}(A, B) = \\frac\{|A \\cap B|\}\{|A \\cup B|\}$

Where $A$ and $B$ represent the sets of extracted records from two identical runs. A score of 1.0000 indicates absolute byte-identical agreement, while lower values reflect variance.

Experimental Results

Key Takeaways from the Data:

1. Concurrency 1 is 100% Deterministic: Running requests strictly sequentially (concurrency=1) resulted in 137 out of 137 identical records across runs ($1.0000$ Jaccard agreement).
2. Concurrency 4 Introduces ~30% Drift: Running the exact same code with 4 concurrent requests dropped the Jaccard similarity to 0.7050—meaning nearly 30% of the extracted outputs shifted or degraded.
3. Cross-Concurrency Disagreement is Severe: Comparing the outputs of concurrency=1 against concurrency=4 dropped agreement further to 0.5490.

This proves conclusively that sampling is not the culprit. If greedy sampling were non-deterministic, concurrency=1 would also fail to achieve 1.0000 Jaccard agreement. The variance is directly driven by batch size and dynamic scheduling.

The Technical Root Cause: CUDA FP16/BF16 Non-Associativity

Why does changing the number of concurrent requests alter the mathematical output of a neural network?

To optimize GPU throughput, modern LLM serving engines (such as vLLM, TensorRT-LLM, and cloud API endpoints behind providers available on n1n.ai) utilize Continuous Batching and PagedAttention. Instead of processing one request end-to-end, the engine packs tokens from multiple concurrent requests into unified matrix operations executed across CUDA tensor cores.

Request A (Concurrency 1):  [Prompt A] --------> [Kernel Execution Order 1] -> Output A

Request A (Concurrency 4):  [Prompt A] \\
Request B:                  [Prompt B]  ===> [Dynamic Batch Tensor Operations] -> Output A'
Request C:                  [Prompt C] /

Floating-Point Arithmetic Is Not Associative

In standard pure mathematics, addition is associative: $(a + b) + c = a + (b + c)$.

In computer floating-point arithmetic (especially reduced precision formats like FP16 or BF16 used in LLM inference), floating-point addition is non-associative: