OpenAI Shares Test Results for Jalapeño Inference Chip

Data centers are power-limited and OpenAI's Jalapeño sets new frontiers for tokens per second per watt. Impressive and fast work from OpenAI's ASIC program. https://twitter.com/SemiAnalysis_/status/2092253723640598761

No MTP, No PD disaggregation, Pure TP, still beats NVIDIA's Vera Rubin NVL72 on a third-party model, with A0 stepping. And B0 is 25% better. NVIDIA GPUs become HBM wrappers.

Since announcing Jalapeño, our first custom inference chip, we’ve been testing it and the system around it. The results show a major advance: more intelligence from every watt and faster responses, delivering both higher throughput and lower latency in one architecture without… Show more

ultrafast inference reveals new threats. worth thinking about how quickly misaligned frontier class models running 50x faster could infiltrate systems and so on— far too quickly for human responders to stay abreast. you need autonomous detection and shutdown, not just monitoring… Show more

OpenAI chip team presenting their Jalapeno ASIC at Hot Chips They chose InferenceX and open source models as North Star benchmark

OpenAI explaining there are three phases of disaggregated compute now: -prefill (context loading) -draft (speculative decoding) -decode