OpenAI SemiAnalysis Lens: Vera Rubin NVL72 Beats GB200, but the TCO Case Is Narrower

NVIDIA’s Vera Rubin NVL72 has posted its first measured silicon results, claiming ten times GB200 NVL72 throughput per megawatt at matched interactivity. The OpenAI SemiAnalysis connection comes through Triton, whose Rubin support helps expose the architecture to widely used AI software.

The headline is compelling, but it compares Rubin with an early 2025 GB200 software baseline. SemiAnalysis found a smaller advantage against July 2026 Blackwell results, although Rubin still led across the tested interactivity range.

That distinction defines the real contest. Rubin is not simply a faster GPU replacing an older GPU. It is a rack-scale system designed to sustain responsive inference when model size, memory traffic, and token demand rise together.

The result puts pressure on GB200 operators, competing accelerator vendors, and developers maintaining custom kernels. It also leaves an important verification gap. The initial test used an engineering-sample rack, an older reasoning model, and a single-turn workload.

Rubin’s First Result Changes the NVL72 Comparison

Rubin’s early lead looks real, but the ten-times figure is a best-case comparison rather than a universal performance ratio.

CoreWeave published the first measured Vera Rubin NVL72 silicon benchmark on July 21, 2026. Its engineers ran DeepSeek R1 across Rubin and GB200 NVL72 with the same major inference optimizations enabled.

The test measured output-token throughput per megawatt against interactivity, expressed as tokens per second for each user. This matters because an inference service must balance total capacity with acceptable response speed.

At matched interactivity, the measured silicon results showed up to ten times more output-token throughput per megawatt. The comparison used a 2025 GB200 NVL72 baseline.

CoreWeave says both systems used NVFP4 precision, speculative decoding, wide expert parallelism, and disaggregated prefill and decode. TensorRT-LLM and NVIDIA Dynamo provided the serving software.

NVFP4 is NVIDIA’s four-bit numerical format for reducing model storage and computation. Speculative decoding generates candidate tokens ahead of final verification, increasing output speed when its predictions succeed.

Disaggregated serving separates prefill and decode across different GPU groups. Prefill processes the prompt, while decode generates the response one token at a time.

Those optimizations matter because benchmark results often measure software quality as much as silicon. An older kernel, scheduler, or parallelization recipe can leave a large part of a GPU’s capacity unused.

SemiAnalysis adjusted its own InferenceX data to match CoreWeave’s accounting. The original benchmark counted power for both prefill and decode GPUs while reporting only output tokens.

Against July 2026 GB300 NVL72 results, Rubin delivered nearly twice the throughput through 100 tokens per second per user. Its advantage expanded to roughly four times around 200 tokens per second.

At 300 tokens per second, the reported ratio reached 5.4 times. However, SemiAnalysis noted that GB300 was operating at the edge of its viable performance curve.

GB200 could not reach that interactivity level in the tested configuration. Rubin continued to 350 tokens per second, where it produced 70,703 output tokens per second per megawatt.

This does not invalidate CoreWeave’s claim. Early Rubin clearly sustains higher interactivity than the measured Blackwell systems. It does change what buyers should infer from the headline.

A ten-times figure describes one workload, operating point, software baseline, and accounting method. It does not mean every Rubin deployment immediately replaces ten GB200 racks.

The stronger conclusion is narrower and more useful. Rubin maintains efficiency as serving moves toward smaller batches and faster individual responses, where Blackwell’s throughput falls sharply.

That behavior directly affects coding agents, search systems, and security applications. These services create many sequential model calls, making latency difficult to hide with large batches.

Why Performance per Megawatt Now Matters More Than Peak FLOPS

Power capacity has become a practical ceiling, so useful tokens per megawatt can matter more than theoretical arithmetic throughput.

An accelerator’s peak FLOPS count measures its maximum floating-point operations under specific conditions. It does not describe how efficiently a complete rack serves a reasoning model.

Inference moves weights, activations, and KV-cache data between memory and compute units. KV cache stores attention information from earlier tokens, preventing the model from recalculating the entire sequence.

Longer contexts enlarge that cache. Mixture-of-experts models also send tokens among specialized subnetworks, creating communication traffic across GPUs.

Rubin addresses those constraints as a complete NVL72 system. The rack combines 72 Rubin GPUs, 36 Vera CPUs, ConnectX-9 networking, BlueField-4 processors, and NVLink 6 switches.

NVIDIA lists 20.7 terabytes of HBM4 memory across the rack. It also specifies 260 terabytes per second of aggregate NVLink 6 switch bandwidth.

Each GPU receives 3.6 terabytes per second of all-to-all scale-up bandwidth. Scale-up networking connects accelerators within one large computing domain, allowing them to behave like a unified device.

Rubin therefore attacks inference efficiency at several layers. Faster tensor operations handle matrix math, HBM4 supplies weights, and NVLink moves tokens among experts.

Vera CPUs manage data movement and CPU-heavy agent tasks. These can include tool calls, code compilation, orchestration, and sandboxed execution.

NVIDIA’s NVL72 specifications claim 3,600 petaflops of rack-level NVFP4 inference performance. The company also claims one-tenth the token cost compared with GB200 NVL72.

Those figures remain workload-dependent company claims. Still, the rack architecture explains why Rubin’s advantage grows when interactivity increases.

Large batches help a GPU remain busy because many users share each weight load. Fast interactive service reduces batching opportunities, increasing pressure on memory bandwidth and scheduling.

Rubin’s 22-terabyte-per-second HBM4 bandwidth per GPU gives it more room under those conditions. SemiAnalysis estimates 2.8 times Blackwell Ultra’s global memory bandwidth.

Higher bandwidth does not automatically reduce memory latency. It moves more data per second, but an individual access can still take similar time.

Rubin’s wider advantage at high interactivity therefore reflects several mechanisms working together. No single peak specification fully explains it.

The power metric also includes infrastructure choices. Cooling equipment, networking, host processors, storage, and conversion losses all consume electricity beyond the GPU package.

NVIDIA designed Rubin for liquid coolant entering at 45 degrees Celsius. In compatible facilities, that temperature supports dry cooling without traditional chillers.

The company says its closed-loop approach can reduce water consumption and cooling overhead. Those gains depend on facility design and should not be assumed for every existing data center.

SemiAnalysis used the same power usage effectiveness assumption across liquid-cooled systems. That conservative choice prevents Rubin’s facility design from inflating the silicon comparison.

The result still favors Rubin. More importantly, it shows why rack architecture has become inseparable from accelerator economics.

A buyer cannot evaluate Rubin by comparing GPU FLOPS alone. The relevant unit is the serving system that delivers a required response speed within a fixed power envelope.

OpenAI SemiAnalysis Software Support Gives Rubin an Earlier Starting Line

Rubin can reuse important Blackwell kernels, shortening deployment work before developers begin architecture-specific tuning.

The OpenAI SemiAnalysis angle is not an OpenAI hardware partnership. It refers to Rubin support appearing in OpenAI Triton alongside PyTorch, vLLM, CUDA, and other public projects.

Triton is an open-source language and compiler for writing GPU kernels with Python-like syntax. A kernel is a specialized program that runs a compute operation on the GPU.

OpenAI introduced Triton programming to make high-performance kernel development more accessible than low-level CUDA code. PyTorch compilers and inference projects now rely on Triton-generated kernels for many operations.

NVIDIA has released a CUDA 13.4 developer preview with Rubin support and updated PTX instructions. PTX is NVIDIA’s intermediate instruction language for describing GPU operations.

The CUDA preview lets developers inspect new Rubin capabilities and begin porting software. NVIDIA warns that this preview is pre-release software and unsuitable for production benchmarking.

SemiAnalysis reports that Rubin changes have also reached PyTorch, vLLM, and OpenAI Triton repositories. That public exposure gives framework developers time to adapt before broad cloud availability.

Rubin’s streaming multiprocessor uses the SM107 target. More importantly, it can run major Blackwell SM100-family kernels across libraries such as CUTLASS, DeepGEMM, and FlashMLA.

Blackwell did not inherit the same convenience from Hopper. Its tensor-core programming model required substantial kernel rewrites before developers could approach the hardware’s potential.

Rubin preserves more of that investment. Teams can start with functional Blackwell kernels, deploy earlier, and optimize the most valuable operations afterward.

Compatibility should not be confused with maximum performance. SemiAnalysis says architecture-specific tuning remains necessary to reach the practical speed limit.

Rubin increases shared memory from Blackwell’s 228 KiB to an optional 328 KiB mode. Tensor Memory also grows to 256 KiB, giving kernels more space for accumulators and scaling information.

The architecture adds inline Tensor Memory Accelerator descriptor updates. TMA is a hardware mechanism that moves multidimensional data without requiring normal execution units to manage every transfer.

In a mixture-of-experts layer, each expert owns a separate weight matrix. Blackwell may need to rewrite and synchronize a descriptor when the active expert changes.

Rubin can pass the new address with the transfer instruction. One descriptor can then serve several experts without an intervening memory rewrite.

That reduces dispatch overhead during low-batch decoding. It also illustrates why real inference gains come from small data-movement improvements, not only larger matrix engines.

Rubin doubles FP8 and FP4 Tensor Core throughput compared with Blackwell, according to NVIDIA’s architecture disclosures. It also adds finer synchronization between dependent thread blocks.

These features help developers build larger fused kernels. Fusion combines multiple operations, reducing repeated launches and unnecessary movement through memory.

The software advantage extends beyond launch readiness. Compatible tools let more developers examine Rubin behavior, report defects, and optimize common model architectures.

However, public support remains early. CUDA’s preview restrictions show that available code is not equivalent to a mature production stack.

PyTorch and vLLM integration can establish functionality while leaving substantial optimization unfinished. Operators should distinguish “runs on Rubin” from “uses Rubin efficiently.”

The historical pattern supports that caution. GB200 inference improved during its first year as kernels, schedulers, and distributed-serving recipes matured.

Rubin begins from a stronger compatibility position. Its eventual lead will still depend on whether framework maintainers convert new instructions into dependable, model-level gains.

The 3-Bit LUT Tensor Core Targets the Memory Bottleneck

Rubin’s most interesting inference feature compresses weights inside the Tensor Core, reducing memory traffic without a separate dequantization pass.

Rubin adds a lookup-table B operand mode to its matrix multiply-accumulate instruction. SemiAnalysis describes it as NVIDIA’s first Tensor Core format using an internal non-uniform codebook.

In this mode, each stored weight becomes a three-bit index. That index selects one of eight eight-bit E4M3 values from a lookup table shared across a weight block.

The Tensor Core reconstructs the selected value inside the matrix operation. Software does not need to generate a separate decompressed weight matrix before multiplication.

Including the shared codebook, SemiAnalysis calculates a stored footprint of 3.125 bits per weight. The codebook contains 64 bits shared across 512 weights.

The design differs from NVFP4 and MXFP formats. Those formats use block scaling, applying a uniform scale to a group of low-precision values.

A lookup table can place its eight values unevenly. It can concentrate entries near dense weight clusters or represent asymmetric positive and negative distributions.

That flexibility can preserve more information than uniform rounding at a similar bit count. It does not guarantee better model quality.

One codebook covers 512 weights, while NVFP4 can adapt its scale across much smaller groups. Results will depend on calibration data, codebook fitting, and model sensitivity.

Some layers may also need higher precision. An aggressive quantization recipe can save memory while damaging reasoning accuracy or destabilizing output.

The hardware mechanism still addresses a central inference constraint. During low-batch decoding, GPUs frequently wait for model weights to arrive from HBM.

Reducing each weight’s stored size lets memory deliver more weights per second. It also lowers the energy spent moving those bits through the system.

SemiAnalysis used a hypothetical 2.8-trillion-parameter model to illustrate the capacity effect. Its calculation placed the raw weight payload near 1.09 terabytes with Rubin’s format.

The comparison excluded KV cache, activations, replication, and serving overhead. It therefore demonstrates weight storage, not total deployment memory.

At 288 gigabytes of HBM4 per Rubin GPU, the compressed weights would require roughly four packages. An alternative low-precision representation required about six in the example.

Fewer participating GPUs can reduce communication and replication. It can also leave more memory for longer contexts, larger batches, or the KV cache.

However, the LUT mode has implementation constraints. SemiAnalysis notes that it cannot transpose the B matrix, which limits the operations that can use it directly.

The public benchmark also appears not to use this feature. Rubin’s initial lead therefore cannot be credited to three-bit lookup-table inference.

That is both encouraging and uncertain. Rubin has additional silicon capability that future software can exploit, but its accuracy and production value remain unverified.

NVIDIA has also added runtime 2:4 activation sparsity. This method retains two values in every group of four and skips the other two during supported operations.

Unlike earlier weight sparsity, runtime activation sparsity does not require permanently pruning and retraining the model. The hardware can compress intermediate values while the model runs.

Yet NVIDIA has not published accuracy evidence for discarding half of selected activation values before later operations. The CoreWeave result does not appear to use this feature either.

These unused capabilities represent Rubin’s optimization runway. They should not enter current TCO calculations as guaranteed future gains.

Buyers should demand model-level quality tests alongside throughput. A lower-bit format only reduces serving costs if the resulting model meets the same accuracy and reliability target.

Rubin Wins the TCO Test, but the Baseline Changes the Margin

Rubin appears cheaper per delivered token despite higher ownership costs, although its advantage shrinks against fully tuned Blackwell systems.

Total cost of ownership combines hardware, electricity, facilities, networking, maintenance, and operational expenses. It offers a broader view than performance per watt alone.

SemiAnalysis applied its operator-ownership model to normalized output throughput. The analysis avoided cloud rental rates, which can include scarcity, contract terms, and provider margins.

The analysis found Rubin cheaper per output token at every measured interactivity level against July 2026 GB200 and GB300 results. The advantage increased as response speed rose.

Against the current GB200 baseline, Rubin was about 1.5 times cheaper through 100 tokens per second per user. The relative advantage reached roughly three times around 200 to 250 tokens.

Comparing Rubin with the 2025 GB200 software baseline produced a larger result. Rubin peaked near eight times cheaper around 150 tokens per second per user.

That old baseline explains much of the difference between NVIDIA’s marketing headline and a practical buying comparison. A tuned 2026 GB200 is more capable than its early deployment state.

SemiAnalysis also estimates that Rubin carries higher ownership cost per GPU than either GB200 or GB300. Its throughput gain must overcome that higher system burden.

It does so in the examined workload, especially at demanding interactivity targets. The economics become less decisive at slower response speeds, where Blackwell can use larger batches effectively.

This creates a workload-specific purchasing decision.

For highly interactive inference

• Rubin sustains response speeds that GB200 cannot reach in the tested configuration.

• Its memory bandwidth and rack fabric become more valuable as batch size falls.

• Coding agents and real-time search services fit this profile.

For throughput-focused inference

• Mature Blackwell software can narrow Rubin’s relative gain.

• Existing infrastructure and reserved capacity can outweigh a theoretical efficiency advantage.

• Migration costs deserve inclusion in the operator’s TCO model.

For long-context models

• Rubin’s larger memory pool gives operators more space for weights and KV cache.

• The initial single-turn benchmark does not directly measure that advantage.

• Multi-turn agent tests will provide a more representative signal.

The benchmark used DeepSeek R1 671B with an 8,000-token input and 1,000-token output. That model and sequence shape do not represent every 2026 production workload.

SemiAnalysis argues that newer multi-trillion-parameter models should favor Rubin’s capacity and bandwidth. That claim remains a technical expectation until comparative results arrive.

CoreWeave also ran the test on a Dell engineering-sample rack without a scale-out fabric. Scale-out connects multiple racks, while the internal NVLink backplane provides scale-up connectivity.

The successful test supports the rack’s internal expert-parallel operation. It does not establish performance, reliability, or efficiency across a large multi-rack deployment.

Competition adds another constraint. AMD’s MI455X offers 432 gigabytes of HBM4 per accelerator, compared with Rubin’s 288 gigabytes.

Across 72 accelerators, AMD’s Helios design provides 31.1 terabytes of memory. Rubin supplies 20.7 terabytes across its NVL72 rack.

AMD’s capacity advantage can matter for large models and long contexts. NVIDIA counters with its established software ecosystem, NVLink fabric, and earlier rack-scale operating experience.

Google’s TPU systems offer another integrated path. They combine custom accelerators, interconnects, compilers, and cloud services under one operator.

The TCO contest is therefore not limited to Rubin and GB200. Buyers must compare entire serving recipes using the same model, accuracy target, latency, context length, and power accounting.

Rubin currently holds the strongest public early result. The evidence does not yet establish one fixed cost ratio across production inference.

What the Next Benchmarks Must Prove

Three signals will determine whether Rubin’s early engineering result becomes a durable production advantage.

The first signal is NVIDIA’s promised InferenceX submission in the third quarter of 2026. SemiAnalysis says NVIDIA has committed to providing independently verifiable Rubin numbers through that benchmark.

A credible submission should test current models under documented serving configurations. It should report interactivity, output throughput, power boundaries, and optimization settings together.

Results against July 2026 GB200 and GB300 systems would strengthen the present comparison. A repeat of the older GB200 baseline would leave the central methodological concern unresolved.

The second signal is a multi-turn agent workload. Single-turn inference cannot reproduce repeated tool calls, growing KV caches, and changing prompt lengths across an agent session.

SemiAnalysis is developing an AgentX scenario with infrastructure and open-source serving contributors. Its Rubin analysis identifies long-context agentic work as a likely architectural strength.

Rubin should widen its lead when memory capacity and bandwidth dominate. If the advantage remains unchanged or contracts, the rack’s long-context case becomes less convincing.

The third signal is mature public software. Developers should watch production releases across CUDA, PyTorch, vLLM, Triton, TensorRT-LLM, and Dynamo.

Functional compatibility will arrive before optimized performance. The decisive evidence will come from stable kernels that use Rubin-specific memory movement, synchronization, and low-precision features.

Three-bit LUT inference deserves particular scrutiny. Published tests must report accuracy, calibration methods, affected layers, and end-to-end throughput against NVFP4.

Activation sparsity needs the same treatment. Higher arithmetic rates mean little if quality loss forces operators to use a larger model or repeat failed requests.

Feynman also creates a longer-term software question. NVIDIA has identified its next architecture as SM140, while Rubin uses SM107.

SemiAnalysis expects Feynman to require broader kernel rewrites, resembling the difficult Hopper-to-Blackwell transition. Rubin’s compatibility advantage may therefore last only one generation.

For infrastructure buyers, the immediate decision is not whether Rubin has better specifications. It does.

The harder question is whether a specific workload benefits enough to justify deployment changes. Operators need tests based on their model, context distribution, response target, and existing power capacity.

For developers, the useful action is to track kernel support and benchmark recipes now. Early profiling can reveal whether an application is limited by compute, HBM traffic, networking, or scheduling.

For AI product teams, watch user-level outcomes. Faster tokens only matter when they reduce task completion time, improve agent reliability, or support more simultaneous customers.

The OpenAI SemiAnalysis thread ultimately connects three layers: open kernel tooling, independent performance analysis, and NVIDIA’s rack-scale hardware. None of those layers can validate Rubin alone.

Will Rubin preserve its advantage on modern models, multi-turn agents, and independently reproducible tests? That evidence, rather than one headline multiple, should decide the next inference infrastructure commitment.