NVIDIA Releases Nemotron-Labs-3-Puzzle-75B-A9B Compressed Hybrid MoE Model With 2.03x Server Throughput

NVIDIA released Nemotron-Labs-3-Puzzle-75B-A9B on July 9 2026. The new model cuts total parameters from 120.7 billion to 75.3 billion and active parameters from 12.8 billion to 9.3 billion. It keeps the same 88-block hybrid layout of 40 Mamba blocks, 40 MoE blocks, and 8 attention blocks.

The release targets inference cost rather than raw capability. On an 8x B200 node the model reaches 2.03 times higher server throughput once user throughput stays at or above 100 tokens per second in 8K or 64K context windows. One H100 now handles eight concurrent 1-million-token sessions instead of one, and model weights drop from 70 GB to 44.5 GB.

Model Size Reduction Without Architecture Change

Nemotron-Labs-3-Puzzle-75B-A9B comes from an iterative Puzzle compression process applied to Nemotron-3-Super. The method prunes experts inside MoE layers and merges redundant Mamba states while preserving the original block count and routing pattern.

Hugging Face now hosts BF16, FP8, and NVFP4 checkpoints. The FP8 version already matches the BF16 accuracy on most internal benchmarks, and the NVFP4 version runs on next-generation Blackwell GPUs without additional quantization code.

The compression did not alter the 88-block skeleton. NVIDIA kept every Mamba and MoE block in its original position so existing inference kernels continue to work without changes.

Throughput Gains Measured on Production Hardware

NVIDIA reported the 2.03 times server throughput figure on an 8x B200 node under matched user throughput of at least 100 tokens per second. The same test on H100 nodes produced a 1.7 times gain, showing the benefit scales with newer hardware.

Single-GPU memory footprint fell from 70 GB to 44.5 GB. That change lets one H100 run eight separate 1-million-token sessions instead of one. The 1M-token context length remained unchanged because the hybrid layout already supports long sequences through Mamba state compression.

The throughput test used two context lengths, 8K and 64K, and both showed nearly identical gains. This indicates the compression did not introduce extra latency that would appear only at longer contexts.

Accuracy Tradeoffs Appear on Hard Benchmarks

Arena-Hard-V2 scores dropped 4.2 points and SWE-Bench scores dropped 2.6 points compared with the uncompressed Nemotron-3-Super model. Both drops stay within the range NVIDIA expected from the parameter reduction.

The iterative Puzzle method scored 0.57 points higher on average than a single-step baseline during internal ablation tests. NVIDIA presented the number as evidence that repeated compression rounds recover some quality that a one-pass prune loses.

No benchmark showed an outright regression beyond those two scores. Other standard suites remained within one point of the original model. The company states the quality loss stays acceptable for most production workloads that already tolerate small accuracy differences.

Puzzle Method Relies on Iterative Expert Pruning

The Puzzle compression runs multiple rounds of expert importance scoring followed by targeted removal. Each round measures the contribution of every expert to downstream tasks and removes the lowest-scoring set before the next round begins.

NVIDIA kept the same number of active experts per token after pruning. The routing network was retrained for two epochs on the same data mixture used for the original model so token distribution stayed balanced.

The 40 Mamba blocks received separate compression that merged less-used state channels. This step produced most of the memory reduction because Mamba state size grows linearly with sequence length.

The Main Pressure Falls on Existing Dense and MoE Deployments

Existing dense 70B-class models and uncompressed MoE variants now face a direct cost comparison. Operators who run high-concurrency long-context workloads see the clearest difference in cost per token once the new checkpoints are deployed.

Competitors that rely on pure transformer MoE without Mamba blocks have not published equivalent compression results at this scale. Their published numbers still assume the full original parameter count for long-context serving.

The release does not claim to beat every existing model on every benchmark. It claims a measurable reduction in inference cost at a given quality level, shifting the conversation from absolute accuracy to accuracy per dollar on large batches.

Quality Loss Limits Certain High-Stakes Uses

The 4.2-point drop on Arena-Hard-V2 and the 2.6-point drop on SWE-Bench matter for applications that already sit near decision thresholds. Code-generation services that target SWE-Bench scores above a fixed bar may need to keep the larger model for the hardest tasks.

NVIDIA has not released a side-by-side error analysis. It remains unclear whether the lost points come from a small number of hard cases or from a broad shallow decline across many queries.

Users who fine-tune the model on domain data may recover part of the gap. The base compression is fixed, but continued training on the smaller checkpoint is still possible and is expected to be cheaper than training the original size.

Three Signals Worth Watching Over the Next Quarter

Watch for third-party throughput numbers on clusters larger than eight GPUs. NVIDIA supplied only the 8x B200 configuration. Independent runs at 16x or 32x scale will show whether the gains hold or shrink with more communication overhead.

Watch for updated Arena-Hard-V2 and SWE-Bench numbers after users release fine-tuned versions. If the gap narrows below two points on either benchmark, the model becomes viable for a wider set of production workloads.

Watch for similar compression announcements from other labs that use pure MoE or pure Mamba stacks. A direct comparison on the same hardware and context lengths will clarify whether the hybrid layout itself enables the observed compression ratio or whether the gains come mainly from the Puzzle procedure.

NVIDIA positioned the release as an engineering step rather than a capability leap. The practical effect is lower cost per token for long-context, high-concurrency inference while keeping the same context length and routing structure. Teams running large inference fleets now have a concrete option to reduce hardware requirements without rewriting kernels.