SambaNova just put a number on a problem most AI buyers feel in their invoices: 763 tokens per second, achieved on a heterogeneous inference setup that mixes four Nvidia H200 GPUs with SambaNova’s SN50 accelerators. The benchmark testing, run by Artificial Analysis, measured performance in MiniMax M2.7 at short context lengths of 10,000 input tokens, and the headline claim is blunt: the platform runs several times faster than competing GPU-only inference providers.

At longer context lengths, the company says it can sustain more than 450 tokens a second. That combination of “fast at short prompts” and “still moving when prompts get bigger” is exactly what matters for real deployments like code assistants, where agents keep conversation history and must generate outputs steadily, not just during a single quick turn.

Here is the core architectural move that makes the benchmark plausible. SambaNova’s platform disaggregates the inference pipeline into two jobs that behave very differently. First comes the prefill phase, where prompts are processed and key value caches are generated. That phase is computationally intensive, so SambaNova assigns it to four Nvidia H200 GPUs. Second comes decode, where output tokens are produced one after another. Decode operations are described as memory-bandwidth-bound, so SambaNova shifts that part to a single SambaNova rack containing 16 SN50 accelerators.

This “split prefill from decode” lever has been a big theme across the industry for the simple reason that token cost is a function of throughput and efficiency over time. Nvidia previously demonstrated this concept with its NVL72 rack systems by varying the ratio of GPUs used for prefill versus decode. Later, the playbook got louder at GTC this spring, when Nvidia revealed its Groq-based LPX racks, further disaggregating the workload. Since then, the source notes that a wide range of players, from AMD to AWS to Cerebras, have announced disaggregated or heterogeneous inference platforms using one or more accelerators.

What makes SambaNova’s pitch more than just another architecture diagram is the specific customer outcome it is aiming for: using its systems as decode accelerators to breathe new life into aging GPU fleets. If you are a CTO or platform lead, the pain is not only that new accelerators are expensive, it is that inference scaling tends to run into a capacity wall. A decode-optimized path can, in theory, let you wring more tokens per dollar out of GPUs you already own, as long as the heterogeneous setup holds performance under real workloads, not just benchmark suites.

There is also a deployment constraint that boards should care about, because it can break timelines and budgets. SambaNova says its systems are air-cooled, which means they can be deployed in existing datacenters. That matters because Nvidia’s latest generation of Rubin GPUs “absolutely need liquid cooling,” according to the source. In practice, air cooling can mean fewer infrastructure upgrades, less capex surprise, and faster time to rollout, especially for teams whose bottleneck is deployment throughput rather than model experimentation.

SambaNova is not stopping at the current configuration. The company plans to show off even more powerful inference setups, with 128 and eventually 256 accelerators, to demonstrate that high token generation rates can be maintained at high throughput. The underlying bet is familiar to anyone who follows the hardware race: GPUs alone have historically struggled with this kind of sustained throughput, which is one driver behind Nvidia’s Groq acquihire late last year, as the source notes.

Now zoom out to the corporate and capital layer. The results come about a month after SambaNova and Intel announced Vector Core Compute would be among the first to deploy the combined GPU plus RDU offering, with TogetherAI as their first large-scale customer. Ramping production of any chip is expensive, and the source flags that capital could have been a stress point. Instead, on Wednesday, SambaNova completed the first close of a $1 billion Series F funding round led by General Atlantic, giving the AI chip startup an $11 billion valuation.

That funding detail changes the decision landscape for everyone trying to build or buy inference capacity. If SambaNova can translate benchmark performance into reliable, high-throughput deployments for long-running agents, it strengthens the case for heterogeneous inference across the board. It also raises the pressure on other hardware strategies that rely purely on scaling GPUs, especially where cooling constraints and token economics make pure GPU scaling harder to justify. For executives, the strategic stake is straightforward: inference is where budgets go to die or to compound, and SambaNova is trying to make “cheaper tokens” real by pairing older GPUs with newer decode accelerators, not by betting everything on the newest GPU generation alone.