700 Tokens Per Second: How Google's TPUs Are Breaking Nvidia's Grip on AI

A single user can now process 700 tokens per second on Google's seventh-generation Ironwood TPUs, a speed benchmark that significantly outpaces current Nvidia hardware in specific inference tasks. This figure, achieved by Inferact using a tuned setup for the Moonshot AI Kimi K3 model, marks a pivotal moment in the artificial intelligence hardware landscape. It signals that Google's custom silicon is no longer just an internal asset but a competitive force capable of handling some of the most demanding large language model workloads in the market today.

The primary driver behind this performance leap is the official integration of Google's TPUs with the PyTorch framework. Previously, running inference on TPUs required developers to use Google's JAX framework, creating a barrier for teams deeply invested in PyTorch. This week, Google announced a PyTorch-native version of the SGLang inference engine alongside an open-source release of vLLM, another major inference engine. These updates allow the hardware to operate natively within PyTorch environments, largely eliminating the need for code rewrites and lowering the entry barrier for widespread adoption.

Meta, the creator of PyTorch, has moved beyond theoretical testing to active production use. Zongwei Zhou, Meta's PyTorch-TPU lead, confirmed that SGLang-TorchTPU is currently running in production environments. He described the development as the result of a close three-way collaboration between Meta, Google, and the SGLang team. The engine is developed by RadixArk, a startup founded by the creators of SGLang, which worked directly with Google and Meta to build this mature serving stack.

This shift represents a strategic move to diversify hardware dependencies. Jana van Greunen, director of PyTorch engineering at Meta, stated that the company cannot be constrained by a single hardware architecture. She noted that the new TorchTPU technology enables Meta to deploy its existing PyTorch workloads directly onto Google TPUs. Google confirmed it co-developed this technology with Meta's PyTorch team, highlighting a deepening partnership that goes beyond simple software compatibility.

This is not the first time Meta has explored Google's hardware, but it is the first time it has been integrated so seamlessly into their primary development workflow. Google refers to the older PyTorch/XLA bridge as a predecessor to this new technology. Reports from SemiAnalysis indicated that Meta research teams used TPUs via this older bridge between 2020 and 2023 before those contracts were canceled. The current push appears to address Meta's previous reluctance to migrate to JAX, offering a solution that preserves their existing codebase while expanding their hardware options.

The timing of this announcement is notable as it landed one day after Amazon revealed it was open-sourcing TorchNeuron, its own native PyTorch backend for Trainium chips. Both moves appear designed to loosen Nvidia's software lock-in, which has long been a key factor in its market dominance. By making PyTorch, the industry's standard framework, work natively on competing hardware, Google and Amazon are reducing the switching costs for major AI labs and enterprise customers.

Performance comparisons show that the new kernels for models like Kimi K3 and Qwen 3.8 27B deliver roughly 1.4 to 2 times the throughput of the Nvidia GB200 baseline at small batch sizes, even without speculative decoding. The 700 tokens per second benchmark was achieved using 16 Ironwood chips with speculative decoding, a technique that uses a smaller draft model to speed up inference. In contrast, the comparable Nvidia GB200 setup reached 452 tokens per second under similar conditions.

Looking ahead, SGLang-TorchTPU plans to expand its model support, starting with DeepSeek V4.1 Flash and Reflection Beam. RadixArk aims to provide Day 0 TPU support for new open models by the first quarter of 2027. Google, Meta, and RadixArk are scheduled to share further details at the SGLang Summit 2026, taking place in San Francisco in November, which will likely provide more insight into the future of this multi-vendor hardware strategy.