Optimizing Keras 3 on PyTorch: Achieving an 11.6x Speedup via Engine-Level Refactoring
- Authors
- Name
- Nino
- Occupation
- Senior Tech Editor
Recent updates in the Keras repository under issue #22561 have transformed how compiled models execute on backend frameworks, particularly PyTorch. Between July and October 2026, a series of 15 pull requests dramatically improved inference throughput. On a standard 2.1M-parameter test language model, compiled Keras[torch] now generates text 11.6 times faster than it did previously.
This performance leap was not achieved by altering hardware or upgrading framework versions—the same NVIDIA RTX PRO 6000 Blackwell GPU, PyTorch build, and JAX versions were maintained across benchmarks. Instead, the gains stem from relentless profiling, eliminating Python-side overhead, and removing graph breaks that previously crippled torch.compile.
Whether you are deploying massive transformer architectures via high-throughput LLM APIs or managing distributed inference clusters using infrastructure aggregators like n1n.ai, understanding these low-level optimizations is crucial for maximizing hardware utilization.
The Performance Transformation: Before and After
In July 2026, compiling a Keras model backed by PyTorch was actually counterproductive. Rather than accelerating execution, torch.compile introduced significant compilation overhead and graph breaks, causing eager execution to outperform the compiled counterpart.
- July 2026: Compiled Keras[torch] ran at 1,458 tokens/second, whereas eager Keras achieved 2,139 tokens/second.
- October 2026: The compiled model hits 16,975 tokens/second—delivering a 6.56x speedup over eager Keras and closing in on native, plain compiled PyTorch performance.
Configuration Performance Comparison
The table below illustrates the execution time (in milliseconds per call) across 12 distinct configurations and workloads, tracking progress from the July baseline to the October milestone.
| Keras config | Workload | Plain framework (ms) | Jul 18 (ms) | Oct 15 (ms) | Change | + #23336 (ms) | A/B check |
|---|---|---|---|---|---|---|---|
| Keras[torch] · eager | CNN forward | 0.064 | 0.668 | 0.477 | 1.40x | 0.466 | 1.39x |
| Keras[torch] · eager | LLM forward | 0.189 | 1.88 | 1.55 | 1.22x | 1.48 | 1.20x |
| Keras[torch] · eager | LLM generate-32 | 5.85 | 59.8 | 49.5 | 1.21x | 47.2 | 1.20x |
| Keras[torch] · torch.compile | CNN forward | 0.076 | 1.28 | 0.160 | 8.03x | 0.161 | 8.41x |
| Keras[torch] · torch.compile | LLM forward | 0.160 | 2.69 | 0.302 | 8.93x | 0.299 | 9.60x |
| Keras[torch] · torch.compile | LLM generate-32 | 4.92 | 87.8 | 7.54 | 11.6x | 7.66 | 12.4x |
| Keras[jax] · eager | CNN forward | 0.508 | 1.34 | 1.23 | 1.08x | 1.18 | 1.15x |
| Keras[jax] · eager | LLM forward | 2.79 | 8.46 | 8.38 | 1.01x | 7.90 | 1.12x |
| Keras[jax] · eager | LLM generate-32 | 89.0 | 293 | 267 | 1.10x | 269 | 1.08x |
| Keras[jax] · jax.jit | CNN forward | 0.077 | 0.079 | 0.081 | 0.97x | 0.080 | 1.02x |
| Keras[jax] · jax.jit | LLM forward | 0.120 | 0.131 | 0.135 | 0.97x | 0.130 | 1.05x |
| Keras[jax] · jax.jit | LLM generate-32 | 2.22 | 2.30 | 2.45 | 0.94x | 2.27 | 1.05x |
Note: A/B check compares July vs. October timing across three separate back-to-back rounds.
Anatomy of the Optimization: Key Pull Requests
Rather than rewriting the entire codebase, the engineering team targeted shared execution paths (Layer.call, op tracebacks, dtype handling, input validation). Because these modules are shared across backends, the performance dividend extends beyond PyTorch, reducing Python function call overhead across JAX, TensorFlow, and NumPy.
The table below breaks down the impact of individual PRs on call overhead and generation latency:
| PR | Merged | What it changed | Δ calls/fwd torch | JAX | TF | NumPy | Δ eager gen-32 (ms) | Δ compiled gen-32 (ms) | Graph breaks |
|---|---|---|---|---|---|---|---|---|---|
| #23183 | 07-28 | Tracebacks built only on op failure | −598 | −586 | −622 | −546 | −2.9 | −5.2 | · |
| #23327 | 08-30 | Removed graph break in dot_product_attention | +6 | 0 | 0 | 0 | +0.40 | −12.3 | LLM 11 → 0 |
| #23200 | 09-05 | Stopped re-setting built/_called on every call | −1,848 | −1,098 | −1,188 | −1,100 | −4.9 | −10.5 | · |
| #23201 | 09-11 | Canonical dict key order for tree code | −142 | +8 | 0 | 0 | +0.55 | −5.2 | CNN 9 → 0 |
| #23647 | 09-30 | Tree code stops comparing id()s | 0 | −8 | 0 | 0 | −0.58 | −49.4 | · |
For developers building enterprise applications that route prompts across diverse foundational models via n1n.ai, these lower latencies translate directly into reduced compute costs and improved user experience.
Profiling Hot Functions in LLM Forward Passes
A granular look at cProfile data reveals precisely where Python execution time was reclaimed during a single Keras[torch] LLM forward pass:
| Function | Jul 18 | Oct 15 | Change | + #23336 |
|---|---|---|---|---|
isinstance | 2,600 | 1,740 | −33% | 1,680 |
| traceback error wrappers | 74 | 48 | −35% | 48 |
| thread-local config lookups | 624 | 513 | −18% | 513 |
standardize_dtype | 177 | 244 | +38% | 184 |
tree.flatten | 93 | 67 | −28% | 67 |
tree.map_structure | 102 | 62 | −39% | 62 |
By optimizing object checks, caching type promotions, and streamlining tree traversals, the framework successfully stripped out thousands of redundant interpreter operations per forward pass.
Pro Tips for PyTorch & LLM API Integration
- Eliminate Dynamic Graph Breaks: Always inspect your compilation boundaries using
torch._dynamo.explain(). Unhandled Python constructs or dictionary re-orderings can silently trigger fallbacks to eager mode. - Leverage Aggregated Endpoints: When orchestrating multi-model pipelines or fallback chains, utilize resilient API gateways like n1n.ai to handle rate-limiting and latency routing seamlessly.
- Keep Tooling Up to Date: Ensure your execution runtime matches your compilation targets; minor version shifts in PyTorch can sometimes alter graph capture behavior.
Get a free API key at n1n.ai