700: The Token Speed That Signals a Shift in AI Hardware

Google’s custom silicon is no longer confined to its own data centers.

For the first time, Tensor Processing Units (TPUs) are serving production AI workloads at Meta, the creator of the PyTorch framework, through a newly announced PyTorch-native version of the SGLang inference engine. This development, detailed in Google’s developer blog, marks a significant departure from the industry standard, which has long been dominated by Nvidia GPUs paired with specific software stacks. A corresponding open-source version of vLLM, another major inference engine, was also released to support this transition. The core innovation lies in the removal of the previous dependency on Google’s JAX AI framework; the new versions allow PyTorch, the dominant ecosystem for AI research and development, to run natively on TPUs without requiring extensive code rewrites.

The timeline of this integration reflects years of underlying tension and collaboration. Until recently, running PyTorch models on TPUs required using the PyTorch/XLA bridge, which Google now describes as a predecessor to the new TorchTPU. SemiAnalysis reported last year that Meta’s research teams utilized TPUs via this bridge from 2020 to 2023 before canceling those contracts. The primary friction was Meta’s reluctance to migrate its entire infrastructure to Google’s JAX framework. The new TorchTPU initiative, co-developed by Google and Meta’s PyTorch team, aims to resolve this by making TPU a first-class citizen in the PyTorch ecosystem. Talks reportedly began last year, with negotiations covering Google Cloud TPU rentals for 2026 and potential purchases in 2027, eventually leading to a multi-year rental agreement, though the physical location of these production TPUs remains unspecified.

This shift is not merely a technical upgrade but a strategic move to dilute Nvidia’s market dominance. The announcement landed just one day after Amazon revealed it was open-sourcing TorchNeuron, a native PyTorch backend for its Trainium2 and Trainium3 chips, at the same PyTorch Conference. Both initiatives appear designed to loosen the software lock-in that has allowed Nvidia to maintain its grip on the AI hardware market. By ensuring that the most popular AI framework, PyTorch, runs efficiently on alternative chips, Google and Amazon are lowering the barrier for other companies to diversify their hardware portfolios.

The practical impact is evident in performance benchmarks. Inferact, a startup founded by the creators of vLLM, reported that a tuned vLLM setup on the seventh-generation Ironwood TPU achieved over 700 tokens per second for a single user when serving Moonshot AI’s Kimi K3 model, which has 2.8 trillion parameters. This test utilized 16 Ironwood chips with speculative decoding. In comparison, the same model running on 16 Nvidia GB200s reached 452 tokens per second. Furthermore, the new kernels for Kimi K3 and Qwen 3.8 27B deliver roughly 1.4 to 2 times the throughput of the GB200 baseline at small batch sizes, even without speculative decoding.

Meta’s engineering leadership has emphasized the strategic flexibility this provides. Zongwei Zhou, Meta’s PyTorch-TPU lead at RadixArk, stated that SGLang-TorchTPU is already running in production at Meta, citing a "close three-way collaboration" between Meta, Google, and the SGLang team. He noted that the engine provides a "remarkably mature, PyTorch-native serving stack." Jana van Greunen, director of PyTorch engineering at Meta, added that the company "cannot be constrained by a single hardware architecture," highlighting how TorchTPU enables the deployment of PyTorch workloads to Google TPUs seamlessly.

Looking ahead, the ecosystem is poised for rapid expansion. RadixArk plans to add support for new models, starting with DeepSeek V4.1 Flash and Reflection Beam. The goal is to achieve Day 0 TPU support for new open models, meaning compatibility on the release day, by the first quarter of 2027. Google, Meta, and RadixArk will share further details at the SGLang Summit 2026, scheduled for November 12-13 in San Francisco. This convergence of major tech giants and specialized startups signals a maturing market where hardware choice is increasingly decoupled from software preference, offering enterprises greater leverage in negotiating AI infrastructure costs and capabilities.

For readers in New York and New Jersey, this technological shift has broader implications for the region’s tech hub status. As major companies seek to optimize AI costs and performance, the diversification of hardware providers could lead to a more competitive market, potentially driving down inference costs for local startups and enterprises. The ability to run high-performance AI models on non-Nvidia hardware without rewriting code reduces the friction of adoption, making it easier for tech firms in the Northeast corridor to experiment with and deploy cutting-edge AI solutions.