OpenAI has revealed new technical details and early benchmark results for its custom Jalapeño inference chip, positioning the processor as a high‑efficiency workhorse for running large AI models at scale.

Presented at the Hot Chips conference, Jalapeño was tested on SemiAnalysis’ InferenceX benchmark, a suite designed to stress modern AI accelerators under realistic, multiuser workloads. According to OpenAI, the chip delivered more tokens per user and higher throughput per kilowatt than today’s leading inference hardware, including systems based on Nvidia’s Blackwell architecture.

Richard Ho, OpenAI’s head of hardware, described the performance jump as a “very, very significant” advance over current systems. He emphasized that Jalapeño is engineered not just for raw speed, but for the economics of serving huge numbers of users simultaneously. The chip is tuned to deliver more AI work per unit of power while keeping response times low, a combination that is crucial for large‑scale deployments of chatbots, copilots, and other latency‑sensitive applications.

Jalapeño is the product of a deep collaboration between OpenAI and Broadcom, with OpenAI’s own models used to assist in the chip’s design. Rather than treating the processor as a standalone component, the company is building Jalapeño as the foundation of a multigenerational platform in which models, software stack, memory hierarchy, and networking are co‑designed with the silicon.

That full‑stack strategy is aimed squarely at the pain points of modern inference. Large language models spend significant time in prefill, when the system ingests and encodes the prompt, and in communication phases, when data shuttles between compute units and memory. These stages often become bottlenecks, especially when thousands of requests are being served in parallel.

OpenAI says Jalapeño attacks those bottlenecks by minimizing data movement and communication delays. The chip is designed so that key model state, including the KV cache used during token generation, can be explicitly placed and kept local to the right mix of compute, memory, and networking resources for each phase of inference. By reducing the distance data has to travel and tailoring the hardware configuration to each step of the workload, Jalapeño aims to squeeze more useful work out of every watt.

OpenAI expects Jalapeño to roll out gradually, with initial, limited deployment followed by broader use in later generations as the platform matures.