SemiAnalysis 4-hi HBM Analysis: Why Shorter Stacks Could Win AI Inference

SemiAnalysis has challenged the AI memory industry’s race upward, arguing that 4-hi HBM can deliver full bandwidth with one-third as many DRAM dies.

That claim matters because accelerator makers have spent years adding taller memory stacks. More layers increased capacity, supported larger models, and helped establish high-bandwidth memory as an essential part of AI computing.

The SemiAnalysis 4-hi HBM analysis says inference now changes that calculation. Rack-scale systems provide more aggregate capacity, while quantization and cache offloading reduce the amount each GPU must hold.

The report does not say every workload needs less memory. Training, large batches, and future models can still reward taller stacks. Its sharper argument concerns interactive inference, where bandwidth often limits token generation before capacity becomes scarce.

That creates a direct contest between two design priorities. One favors maximum memory capacity per accelerator. The other favors maximum token bandwidth from every constrained DRAM wafer.

Nvidia’s reported Rubin Ultra memory reduction provides the immediate signal. SemiAnalysis says the accelerator will carry 192GB of HBM, down from 288GB on standard Rubin and Blackwell Ultra.

The company has not publicly detailed a 4-hi Rubin Ultra configuration. However, the reported move toward 8-hi stacks suggests that ever-taller HBM is no longer an automatic choice.

If the reasoning extends to 4-hi HBM4E, AI infrastructure could use fewer DRAM dies without sacrificing each stack’s external bandwidth. That would lower memory costs and produce more usable HBM packages from limited wafer supply.

Nvidia’s Memory Roadmap Is No Longer Moving in One Direction

The meaningful change is not that Nvidia suddenly needs little memory. It is that additional capacity no longer appears valuable at any cost.

Blackwell Ultra established the high-capacity direction clearly. Nvidia lists 288GB of HBM3E per Blackwell Ultra GPU and 20TB across a GB300 NVL72 rack.

The company’s GB300 specifications also list aggregate GPU memory bandwidth reaching 576TB per second. Those figures show why Nvidia previously emphasized both capacity and bandwidth.

SemiAnalysis now reports that Rubin Ultra will reverse part of that trend. Its expected 192GB per GPU represents a one-third reduction from Blackwell Ultra and conventional Rubin.

The comparison requires some care. Rubin Ultra’s architecture reportedly changed from four compute dies to two. Its earlier expected memory capacity therefore belonged to a different system design.

Even after normalizing for that change, SemiAnalysis estimates that memory falls from an earlier 256GB per compute die to 96GB. That is a substantial architectural reset, not a cosmetic specification adjustment.

The report identifies supply as one reason. Nvidia has reportedly secured significant advanced logic and packaging capacity, but available HBM wafers cannot support equivalent volumes of 12-hi stacks.

A 12-hi stack contains twelve core DRAM dies above its base die. An 8-hi stack uses eight, letting the same processed DRAM supply support more finished stacks.

This matters because HBM consumes more fabrication capacity per bit than conventional DRAM. Through-silicon vias, known as TSVs, carry signals and power vertically through the memory stack.

Those TSVs require additional processing. HBM dies also include area for vertical connections, lowering bit density compared with ordinary memory made on a similar process.

SemiAnalysis previously estimated that HBM consumes roughly three times more wafer capacity per bit than commodity DRAM. Its estimate approaches four times as production shifts toward HBM4.

That figure is an analyst estimate, not an industry standard. However, the underlying manufacturing burden is well established across memory suppliers.

SK hynix, Samsung, and Micron must process, thin, connect, test, and package multiple known-good dies. Each extra layer increases material use and exposes the package to additional yield risks.

Taller stacks have historically justified that burden because accelerators needed their capacity. Model weights, activations, gradients, optimizer states, and key-value caches all compete for memory during different workloads.

The new decision suggests that capacity has crossed a threshold for some inference systems. Once a rack holds the required working set, adding memory to every GPU produces less value.

Nvidia has not publicly confirmed SemiAnalysis’s exact Rubin Ultra capacity breakdown. The claim should therefore remain a reported roadmap change, not a finalized product specification.

Still, supply-chain preparations can reveal direction before a public launch. SemiAnalysis says suppliers are preparing 8-hi configurations to become standard after an industry push toward 12-hi and 16-hi products.

The reported change creates the opening for 4-hi HBM. If eight layers are sufficient, designers must ask whether four layers can serve selected inference deployments more economically.

That question would have sounded backward during the recent capacity race. Under severe DRAM constraints, it becomes one of the industry’s most practical architectural choices.

SemiAnalysis 4-hi HBM Keeps the Bandwidth While Removing Dies

A shorter HBM stack reduces capacity, but it does not automatically reduce the package’s external bandwidth.

HBM transfers data through a very wide interface rather than relying only on extreme signaling speed. HBM4 expands that interface to 2,048 data connections per stack.

The HBM4 standard was published by JEDEC in April 2025. It defines the foundation for a new generation of AI and high-performance memory.

According to SemiAnalysis, each HBM4 DRAM die can access as many as 512 of the stack’s data connections. Four dies can therefore populate all 2,048 connections.

Adding a fifth through twelfth die increases capacity. It does not widen the external interface beyond those 2,048 connections.

The available pins are divided among more memory layers in a taller stack. The stack still presents the same overall interface to the accelerator.

That is the central mechanism behind the short-stack argument. A 4-hi, 8-hi, and 12-hi package can offer comparable nominal bandwidth when they use the same generation and signaling rate.

Their capacities differ sharply. SemiAnalysis models 32-gigabit HBM4E core dies, producing 16GB for 4-hi, 32GB for 8-hi, and 48GB for 12-hi.

Those are modeled future configurations, not generally available retail products. They illustrate how layer count changes capacity while leaving the package interface intact.

The economic distinction follows from what suppliers consume. DRAM content drives much of an HBM stack’s cost, while inference customers often value the bandwidth feeding their accelerators.

A 4-hi buyer gets fewer gigabytes. The buyer can still receive the data path needed to keep compute units supplied.

That changes the relevant efficiency measure. Capacity-focused procurement asks how many gigabytes fit beside each accelerator. Bandwidth-focused procurement asks how many tokens those memory links can support.

For inference, the second question can dominate. Autoregressive decoding generates output sequentially, and each step must read the model’s active weights from memory.

The process often performs relatively little arithmetic for each byte moved. That makes decoding memory-bandwidth-bound, meaning data delivery limits throughput before raw computation does.

A taller stack helps only when the workload uses its additional capacity. Otherwise, the extra dies carry information that the available bandwidth cannot read frequently enough.

SemiAnalysis gives an illustrative HBM4E bandwidth of 3,328GB per second per stack. It derives that number from 2,048 pins operating at 13 gigabits per second.

At 100 generated tokens per second, that represents 33.28GB of theoretical bandwidth for each token. The useful working set cannot exceed that budget if every token must read it once.

A 48GB stack would then contain more data than the theoretical bandwidth can scan per token. Some capacity would remain unavailable to that specific workload at that speed.

Real systems never sustain every unit of nominal bandwidth. Protocol overhead, access patterns, communication, and software behavior reduce effective throughput.

That limitation can strengthen the short-stack case. Lower usable bandwidth means the workload reaches its bandwidth ceiling before it can exploit all installed capacity.

The 4-hi HBM explained by this mechanism is not simply cheaper memory. It is a configuration that separates external bandwidth from maximum vertical density.

The distinction also affects manufacturing output. A wafer used for four-layer stacks can theoretically support three times as many packages as one used for 12-layer stacks.

Actual gains depend on yields, base-die availability, testing, and packaging throughput. Shorter stacks should also avoid some cumulative assembly losses associated with additional layers.

SemiAnalysis argues that the resulting bandwidth per HBM wafer matters as much as tokens per watt. Both metrics measure useful AI output against a resource that cannot expand quickly.

Why Inference Values Bandwidth More Than Maximum Capacity

Interactive inference rewards memory that can feed the processor quickly, while unused capacity contributes little to token throughput.

Training and inference place different demands on HBM. Training stores weights, activations, gradients, and optimizer data while processing large batches through forward and backward passes.

That workload can consume enormous capacity. It also performs enough arithmetic to become compute-bound during many operations.

Inference decode is different. The system repeatedly reads active weights and the user’s key-value cache, or KV cache, while producing one token after another.

The KV cache stores attention information from earlier tokens. It prevents the model from recomputing the entire conversation whenever it generates another token.

More users and longer contexts enlarge that cache. However, larger capacity is valuable only if the system can serve those users within an acceptable latency target.

Batching complicates the picture. By processing several requests together, a server can amortize one model-weight read across multiple users.

Larger batches improve total throughput, but they also require more KV-cache capacity. This is where 8-hi or 12-hi stacks can regain an advantage.

The tradeoff is interactivity. A provider can wait longer to form larger batches, or it can return tokens quickly with smaller batches.

SemiAnalysis models this relationship using Kimi K3 on a future Rubin Ultra NVL576 configuration. It finds that taller stacks provide diminishing throughput gains as required per-user speed rises.

At a modeled 213 tokens per second for each user, the report finds no throughput benefit beyond 4-hi. The bandwidth ceiling arrives before extra memory capacity becomes useful.

At other points on the curve, 8-hi raises maximum throughput by 8 percent. A 12-hi configuration raises it by 10 percent over 4-hi.

Those improvements are real within the model. The question is whether they compensate for the additional memory and system cost.

SemiAnalysis estimates that its modeled 8-hi Rubin Ultra system costs 12.1 percent more than the 4-hi baseline. Its 12-hi configuration adds 26.3 percent.

These are analyst estimates based on future components and assumed memory prices. They are not vendor quotations or measured ownership costs from deployed Rubin Ultra systems.

Within those assumptions, throughput grows more slowly than system cost. The resulting cost per token is higher for both taller configurations.

This is the strongest claim in the SemiAnalysis 4-hi HBM case. Shorter stacks do not merely reduce the hardware bill; they improve modeled output per unit of spending.

Rack-scale computing makes the argument more plausible. Earlier servers connected eight GPUs, so the memory on each device strongly constrained the entire serving system.

An H100 HGX system supplied 640GB of aggregate HBM. Large model weights could occupy most of that capacity before the system stored any substantial KV cache.

The H200 relieved that pressure by adding memory to each GPU. At that stage, greater per-device capacity had immediate operational value.

GB300 NVL72 changes the scale. Nvidia connects 72 Blackwell Ultra GPUs through NVLink, creating a much larger shared communication domain.

The rack carries about 20TB of GPU memory. Its aggregate capacity is far greater than that of an eight-GPU Hopper server.

SemiAnalysis compares this growth with Kimi K3, a 2.8-trillion-parameter mixture-of-experts model. Such models activate only a subset of their experts for each token.

The report estimates that a quantized Kimi K3 replica occupies 1,561GB. That would consume less than 8 percent of a roughly 21TB GB300 NVL72 configuration.

Quantization represents numbers with fewer bits, reducing weight storage and memory traffic. It trades some numerical precision for substantially lower hardware requirements.

Large NVLink domains also spread experts across more GPUs. The arrangement increases communication demands but reduces the number of model weights each device must hold.

Rubin Ultra reportedly expands the domain from NVL72 to NVL576. That eightfold increase in connected GPUs can outweigh a one-third reduction in HBM per accelerator.

Capacity therefore moves from an individual-GPU problem toward a rack-level allocation problem. A deployment can hold a large model while using thinner local memory stacks.

This does not eliminate memory constraints. It changes where architects solve them and which resource becomes scarce first.

Cache Offloading Helps, but 4-hi HBM Is Not a Free Lunch

The short-stack strategy works only when software, secondary memory, and workload behavior keep reduced HBM capacity from becoming the bottleneck.

SemiAnalysis tested part of that assumption with an InferenceX workload using Kimi K3 and 16 GB300 GPUs. Eight GPUs handled prefill, while eight handled decode.

Prefill processes the prompt before token generation begins. Separating it from decode lets operators tune each phase for different compute and memory behavior.

The experiment reduced permitted HBM utilization from 92 percent to 85 percent. That is much smaller than the capacity cut between 8-hi and 4-hi stacks.

However, the limit reduced available KV-cache space substantially. Aggregate-serving KV capacity fell from 53GB to 34GB per GPU, according to the report.

For disaggregated pairs, the available budget fell from 44GB to 24GB. Those changes represented reductions of 36 percent and 44 percent.

Throughput remained similar across most tested concurrency levels. The restricted configuration moved inactive KV-cache data into ordinary server DRAM.

That process is called KV-cache offloading. It keeps frequently accessed information in HBM while moving colder conversation state to slower, larger memory.

Agentic workloads can suit this approach. They often pause while tools run, CPUs execute code, or external services return information.

During those pauses, the GPU does not need every conversation’s cache immediately. Moving colder data can release expensive HBM for active requests.

The restricted test encountered a clear limit. Once concurrency exceeded 70, throughput fell by almost 30 percent compared with the higher-memory configuration.

GPU KV occupancy had reached 100 percent. Preemption, repeated transfers, and queueing then reduced useful work.

The constrained run also recorded more than ten times as many reads from the DRAM-based KV cache. Offloading relieved capacity pressure but increased traffic through a slower tier.

This result is both supportive and cautionary. It shows that software can preserve performance after a meaningful memory reduction, but only below a workload-dependent threshold.

A production service with many simultaneous long-context users can cross that threshold. In that case, taller HBM stacks can support larger batches and higher total throughput.

Network capacity also matters. Spreading model experts and caches across hundreds of accelerators creates intensive all-to-all communication.

A system can stop being memory-capacity-bound and become network-bound instead. That outcome still strands extra HBM, but it does not guarantee efficient overall performance.

Secondary DRAM presents another constraint. Server memory is experiencing its own supply pressure as AI systems increase CPU-side capacity.

Offloading also consumes power and adds operational complexity. Software must decide what remains hot, what moves, and when data must return.

Poor decisions can turn capacity savings into latency spikes. The architecture needs careful scheduling, cache management, and workload isolation.

Future model growth presents the larger uncertainty. SemiAnalysis acknowledges that its analysis applies a present workload to hardware expected later.

AI servers often remain deployed for more than five years. Models, contexts, user concurrency, and reasoning workloads can change dramatically during that period.

The report stress-tests a model with three times Kimi K3’s size and triple the per-user cache requirement. Taller stacks become more useful under that assumption.

Its 8-hi configuration delivers 36 percent more total tokens than 4-hi. The 12-hi version delivers 47 percent more, although both operate at lower per-user speed.

After including estimated costs, 8-hi becomes favorable below 180 tokens per second per user. The 12-hi system still fails to offset its modeled cost increase.

These results demonstrate why 4-hi HBM cannot become a universal prescription. The preferred stack height depends on model size, latency targets, batching, and system lifespan.

Training systems remain an especially poor target for aggressive capacity reduction. Their activations, gradients, and optimizer states can use all available memory.

Inference systems serving many independent models also need capacity. Operators may value flexibility more than the lowest cost for one optimized workload.

Hardware buyers face an optionality problem. A 4-hi accelerator can be efficient for today’s service but restrictive after workloads change.

Researchers might adapt models around the installed hardware. Looped computation, quantization, sparse experts, and smaller caches can reduce dependence on stored parameters.

That adaptation is not guaranteed. Model quality improvements might again come from greater parameter counts or memory-intensive architectures.

The best reading is therefore narrower. Four-layer HBM offers a strong configuration for carefully characterized inference, not an automatic replacement for taller memory.

Fewer Layers Put Pressure on Memory Suppliers and System Builders

If buyers optimize bandwidth instead of gigabytes, the economic pressure moves from DRAM production toward base dies, packaging, networking, and complete-system integration.

SK hynix, Samsung, and Micron have spent years developing denser and taller HBM. Their roadmaps emphasize capacity, signaling speed, packaging yield, and thermal control.

SK hynix began supplying 12-layer HBM4 samples in 2025. Its HBM4 announcement described 36GB packages with more than 2TB per second of bandwidth.

That product direction remains useful for training and capacity-heavy inference. A meaningful shift toward 4-hi would add a very different purchasing priority.

Customers could demand full interface bandwidth with fewer core dies. Suppliers would ship more packages but fewer DRAM bits inside each one.

At first glance, that appears negative for HBM revenue. Vendors have benefited from selling more specialized DRAM content into every accelerator generation.

SemiAnalysis argues the outcome can be more balanced. Shorter stacks could improve assembly yields and increase the number of sellable packages produced from each wafer.

A supplier could also charge for bandwidth value rather than treating HBM primarily as a gigabyte product. Whether customers accept that model remains uncertain.

The supply benefit is clearer. Moving from 12-hi to 4-hi theoretically triples the number of stack-sized die groups available from a fixed amount of DRAM.

The realized increase can exceed the simple ratio if shorter stacks raise packaging yield. It will fall below that ratio when other components become constrained.

Every HBM package still needs a base die, testing, stacking, and attachment beside an accelerator. Multiplying package output increases demand for all those steps.

HBM4 makes the base die more important because it includes more logic and can use an advanced foundry process. DRAM savings do not create equivalent foundry capacity.

More memory packages also need more accelerator packages. Those packages require interposers, organic substrates, power delivery, cooling, and high-density assembly.

The bottleneck can therefore migrate rather than disappear. Logic wafers, base dies, substrates, circuit boards, and system integration would all face higher demand.

Large scale-up domains intensify another constraint. More accelerators require extensive switching and networking before their aggregate memory behaves like a practical shared resource.

Nvidia’s GB300 NVL72 uses nine NVSwitch trays to connect 72 GPUs. Its fifth-generation NVLink fabric supplies 130TB per second of aggregate bandwidth.

Future NVL576 systems must extend that scale substantially. Their economics depend on the network remaining fast enough to support distributed experts and cache movement.

Power remains the final boundary. Doubling the number of bandwidth-rich HBM packages helps only when operators can deploy the accelerators attached to them.

A 4-hi strategy can stretch DRAM supply without producing new electrical capacity. Data-center construction, cooling, and grid access still determine usable compute.

This pressure explains why tokens per HBM wafer is a helpful metric, but not a complete one. Operators must optimize tokens per watt, rack, network port, and capital budget together.

The change would also affect conventional memory markets. HBM production competes with server, PC, and mobile DRAM for fabrication resources.

Using fewer HBM core dies can return some wafer capacity to those products. That relief matters as CPU-side memory grows alongside agentic AI deployments.

However, the benefit depends on actual adoption. A 4-hi design that enables far more accelerator shipments could consume some savings through greater total unit volume.

Memory suppliers may also resist a rapid shift if it weakens bit demand. Their response will appear through product availability, qualification schedules, and capacity allocation.

The main contest is therefore not Nvidia against one memory vendor. It is bandwidth-oriented inference design against capacity-oriented HBM economics.

That opponent map keeps the industry consequences clear. Short stacks win when bandwidth creates revenue and installed gigabytes do not.

Taller stacks win when customers can convert extra capacity into larger batches, more models, or longer-lived hardware flexibility.

Three Signals Will Show Whether 4-hi HBM Really Wins

The short-stack thesis becomes credible only when product roadmaps, production availability, and measured inference results converge.

The first signal is Nvidia’s final Rubin Ultra configuration. Public documentation must confirm memory capacity, stack height, bandwidth, and NVL576 system architecture.

A confirmed 192GB configuration using 8-hi HBM would support the directional argument. A return to 12-hi would weaken claims that capacity requirements have broadly relaxed.

A commercial 4-hi Rubin product would provide much stronger evidence. Until then, the SemiAnalysis 4-hi HBM proposal remains an informed forecast about future ASICs and accelerators.

The second signal is supplier qualification of high-speed 4-hi HBM4 or HBM4E. Vendors must offer those packages at the signaling rates required by accelerator designers.

Nominal interface width alone is insufficient. Products need acceptable yields, thermals, reliability, and sustained performance in complete systems.

Watch whether SK hynix, Samsung, or Micron adds 4-hi configurations to public roadmaps. Customer sampling would show that short stacks have moved beyond internal architectural studies.

Also watch the pricing model. If 4-hi costs scale mainly with its lower capacity, its bandwidth economics become compelling.

If suppliers price most of the value into the base die and interface, expected savings may narrow. Package shortages could also preserve high premiums despite lower DRAM content.

The third signal is independent inference testing across varied workloads. Benchmarks must include interactive assistants, agentic services, long-context requests, and large-batch deployments.

Average token throughput will not be enough. Tests need per-user speed, queueing behavior, cache hit rates, offload traffic, and tail latency.

Results should also compare models that differ in size and architecture. A configuration optimized for Kimi K3 cannot establish the best choice for every future frontier model.

The decisive evidence will be a stable cost-per-token advantage under realistic service-level targets. That advantage must survive heavy concurrency and changing workload mixes.

Developers should care because hardware constraints shape model design. Available memory influences quantization, expert placement, context handling, and cache policy.

Enterprise buyers should care because headline capacity can be misleading. More HBM does not guarantee more useful inference when bandwidth, networking, or latency sets the limit.

Infrastructure teams should evaluate working sets at the rack level. They should separate training, prefill, decode, and agentic workloads before selecting one memory profile.

They should also preserve operational headroom. A narrowly optimized 4-hi system can lose its advantage when demand shifts toward larger batches or more simultaneous models.

The broader lesson is not that less memory always wins. It is that memory should be purchased against the resource a workload actually consumes.

For interactive inference, that resource is often bandwidth. Four-layer HBM can retain the full external data path while using fewer scarce DRAM dies.

That makes the SemiAnalysis 4-hi HBM analysis a serious challenge to the industry’s tallest-stack assumptions. The next product disclosures will show whether hardware teams agree.

Before committing to a future accelerator fleet, ask three questions. Does the working set fit at rack scale, can cold cache data move safely, and does extra capacity increase useful throughput?

If the answers point toward bandwidth, the short king deserves a place on the roadmap.