FP16 Quantization Quality Loss Post-Mortem Correction

Our fp16 quantization post-mortem was wrong in the expensive direction: it told anyone with a mask-gated model to give up on half-precision, and that verdict would have kept a 52%-smaller download off the internet. The real cause was sixteen BatchNorm variance tensors larger than fp16's maximum representable value, silently clamped by the converter, leaving sixteen layers normalizing their output by the wrong divisor. The fix is two lines of algebra. The reward is a model that matches fp32 and downloads 52% smaller.

When developers build high-performance pipelines utilizing advanced LLM APIs or localized ONNX models, quantization efficiency is critical. At n1n.ai, we process massive volumes of inference requests where bandwidth and latency dictate user retention, making reliable compression techniques indispensable.

This is a correction post, so it comes with receipts: the lab notes for this round are checked in next to the code, every figure below is reproducible from them on a CPU-only ONNX Runtime install, and the whole investigation fits in a day or two of work.

The Original Misdiagnosis: Architecture vs. Tooling

Part 3 of this series ends with a case study called "When fp16 quantization quality loss is real: the LaMa failure". The measurements in it are real, and the failure is reproducible: rebuilding that conversion from the same weights today, the unfixed fp16 model averages 15.8 dB against fp32's 29.7 dB across our 14 acceptance cases, and its worst case scores 11.1 dB — the watermark surviving, exactly as described. Part 5 repeats it as "its catastrophic mirror image when we tried it on LaMa". Part 1's series index summarises the lesson as "mask-gated architectures and precision sensitivity".

We offered a mechanism too, and it was a good story. LaMa is mask-gated: the mask doesn't just mark where to work, it gates the network's behavior. A mask-gated inpainter therefore has a decision boundary inside it — which regions are trustworthy context, which must be regenerated — and half-precision noise pushes the model across that boundary. The gating fails softly, the model starts copying the watermarked pixels through instead of repairing them, and the output looks plausible except the watermark is still there.

That story explained the symptom exactly. It was consistent with everything else we'd learned (browser inference is bandwidth-bound, so moving half the bytes is free money). And it produced a tidy rule that we then repeated to readers:

Whether quantization is safe is a property of the model's architecture, not of your tooling.

The rule is wrong. Not "wrong with exceptions" — wrong as a predictor, and expensive: it would have kept a quality-identical model that is 48% smaller out of the product forever, and it tells every reader with an attention head, a gating mask, or a long residual chain to stop investigating. Part 8 of this series was about a quality bug that turned out to be ours, not the model's. This is the same shape of mistake, one layer lower in the stack: our conversion tooling, not the model's architecture.

Isolating Precision from Range

"FP16" bundles two independent changes: mantissa precision (how finely you can split a value) and numeric range (the largest magnitude you can hold at all). If the failure were precision sensitivity, then rounding the mantissa while keeping fp32's range would degrade quality in proportion. So we built models that keep the graph and all arithmetic in fp32, and only round the weights' mantissas:

fp16 has a 10-bit mantissa. The 10-bit row is 0.004 dB off the fp32 baseline, and even throwing away 16 of the 23 mantissa bits (7-bit, coarser than bf16) costs 0.054 dB. LaMa is not precision-sensitive — it is precision-indifferent. Whatever fp16 was doing to it, it wasn't the mantissa.

Finding the Silent Overflow

That left range, and range has a hard, checkable boundary: fp16's largest finite value is 65,504. So we dumped every fp32 initializer in the graph and sorted by magnitude.

The graph is 17,480 nodes with 606 float32 initializer tensors, 51M parameters, 204 MB of weights — 197 MB of it in the FFT path (17,022 of those nodes are Fourier convolutions, 97%). Run the check below over the 606 tensors and sixteen come back above 65,504, and every single one is a BatchNorm running_var:

     484322.56  model.generator.model.5.conv1.bn_l.running_var
     440853.66  model.generator.model.25.running_var
     240497.69  model.generator.model.22.conv1.bn_l.running_var
     218544.31  model.generator.model.11.conv1.bn_l.running_var
     174809.92  model.generator.model.6.conv1.bn_l.running_var
      65564.05  model.generator.model.10.conv1.bn_l.running_var
      ...  16 of 606 tensors, every one of them a running_var

Note the last row: the marginal tensor clears the ceiling by 0.09%, and it gets clamped to exactly the same 65,504 as the one that clears it by 7×. That's the whole hazard — a clamped value sits exactly at fp16's ceiling, which is a value fp16 can barely hold, so the next fp16 pass over that tensor takes it to infinity.

When developers integrate models via enterprise aggregators like n1n.ai, silent data type coercion can introduce similar edge cases if intermediate scaling is unmonitored.

This is a five-line check, and it is the first thing to run before believing any "this model can't be quantized" conclusion:

import numpy as np, onnx
from onnx import numpy_helper

m = onnx.load("lama_fp32.onnx", load_external_data=False)
worst = sorted(
    ((float(np.abs(numpy_helper.to_array(t)).max()), t.name)
     for t in m.graph.initializer if t.data_type == 1),   # 1 = float32
    reverse=True,
)[:16]
for value, name in worst:
    print(f"{value:>14.2f}  {name}")

Sixteen tensors. Not sixteen layers of masking logic — sixteen numbers.

A BatchNorm layer computes y = (x − μ) / sqrt(var + ε) · γ + β. running_var is a scale quantity measured over the layer's input during training. If a layer's inputs live at large magnitudes, its variance lives at the square of those magnitudes, and 484,322 is entirely legitimate in fp32: sqrt(484322) ≈ 696.

Clamp that variance to fp16's ceiling of 65,504 and the divisor becomes sqrt(65504) ≈ 256. The layer now normalizes by 256 instead of 696 — its output is 2.7× too large — and the convolution that consumes it was trained against the correct scale. Sixteen of those, in series, and the network's internal magnitudes are destroyed while the graph still runs and still produces plausible-looking images.

The Algebraic Fix

We tried the blunt version first: keep the sixteen offending BatchNorm nodes in fp32 and convert everything else. That works, costs 34 KB of fp32 parameters, and produced identical quality — but it leaves a graph with fp32 islands and sixteen permanent assertions about which layers are dangerous.

The better fix is to notice that a BatchNorm's variance and its scale are only meaningful relative to each other:

y = (x − μ) / sqrt(var + ε) · γ + β ≡ (x − μ) / sqrt(var/K² + ε/K²) · (γ/K) + β

So pick a power-of-two K per layer, divide the variance by K², divide the scale by K, and divide epsilon by K² — the layer is mathematically identical, and now var/K² fits in fp16. With K = 2^ceil(log2(max|var| / 40000)), that's a twenty-line script touching 19 of the 75 BatchNorm layers:

k = float(2 ** np.ceil(np.log2(max_abs_var / 40000.0)))   # K = 16 for the worst layer
var_init.CopyFrom(numpy_helper.from_array((var / (k * k)).astype(np.float32), var_init.name))
scale_init.CopyFrom(numpy_helper.from_array((scale / k).astype(np.float32), scale_init.name))
# and the epsilon attribute: a.f = a.f / (k * k)

The result is worth stating precisely, because it kills the "mixed precision is inherently fuzzy" intuition: the converted model has 606 fp16 initializers and zero fp32 residue. It is 100% fp16, every tensor, with no casts added around the BatchNorms. And the arithmetic is exact — powers of two, so nothing is lost to rounding.

When orchestrating multi-model architectures through n1n.ai, maintaining precise numerical bounds ensures high-throughput stability across diverse runtime engines.

Benchmarks and Verification

Three independent measurements on the same converted weights, plus a control and a harness sanity check:

The size win is the point of the exercise:

The New Diagnostic Checklist

Our old order was architecture → bit width → runtime. The new one starts one step earlier, because a conversion artifact can impersonate an architectural verdict:

1. Audit the values before you audit the architecture. Dump max |value| for every initializer, compare against the target dtype's ceiling, and check the subnormal floor while you're there.
2. Separate precision from range. Round the mantissa, keep the range. If quality doesn't move, precision is not your variable.
3. Test the structure. Gating, long accumulation chains, adversarial spectral paths — all real failure modes, but hypotheses to test, not automatic diagnoses.
4. Verify in the runtime you ship. A Python CPU pass proves the weights; it does not prove that runtime execution providers produce identical outcomes.
5. Benchmarks decide admission. Always validate against a baseline reference.

Whether you are deploying local weights or connecting via n1n.ai for cloud-accelerated LLM inference, systematic verification prevents costly architectural false positives.

Get a free API key at n1n.ai