DeepSeek Engram Offloading Moves AI Memory Beyond HBM

DeepSeek Engram offloading has moved 189 GiB of model memory outside GPU HBM, with DRAM tests sometimes outperforming an all-HBM configuration. The result challenges a basic assumption behind large-model infrastructure. The fastest memory is not automatically the best location for every parameter.

SemiAnalysis published the new experiments on September 18, using DeepSeek-V4.1-Flash and its InferenceX serving framework. Its tests placed Engram tables in host DRAM or memory-mapped files on local NVMe drives. The DRAM path improved one B300 configuration by reducing tensor parallelism, while the first SSD implementation remained slower and less economical.

That distinction matters. Engram does not eliminate demand for Nvidia GPUs or high-bandwidth memory. It changes which model parameters deserve that scarce capacity. The near-term contest is therefore not HBM versus SSD. It is an all-HBM deployment model versus a tiered system that assigns different workloads to HBM, DRAM, and storage.

DeepSeek Engram Offloading Changes Parameter Placement

The central change is architectural: a large block of learned memory can leave HBM without forcing the whole model to behave like an offloaded neural network.

DeepSeek introduced Engram as a conditional memory module for language models. It extends ordinary token embeddings with learned lookups for recurring multi-token patterns. These patterns can include names, code fragments, boilerplate, and common relational phrases.

An ordinary Transformer often reconstructs such local patterns through attention and feed-forward layers. Engram gives the model a separate lookup path. It hashes token sequences, retrieves a small collection of embedding rows, and combines them with the model’s hidden state.

The original conditional memory paper describes this approach as a second form of sparsity. Mixture-of-Experts, or MoE, activates only part of the model’s computation for each token. Engram activates only a tiny part of a much larger memory table.

This distinction determines how the hardware can serve the model. An MoE expert contains weight matrices used for substantial computation. Moving one between storage tiers can require large transfers at exactly the wrong time.

Engram addresses are different. They depend on token IDs and their local sequence, rather than an intermediate hidden state. A runtime can know which rows it needs before later model layers finish computing.

That predictability creates a prefetch window. The server can retrieve rows from host memory while the GPU handles earlier operations. If retrieval finishes within that window, the model avoids much of the latency normally associated with offloading.

DeepSeek’s published research scaled an experimental Engram table to 27 billion parameters. Under controlled comparisons, the authors reported gains over an MoE baseline with equivalent parameters and computation. Reported improvements covered factual knowledge, reasoning, code, mathematics, and long-context retrieval.

Those figures came from research models, not the exact production system tested by SemiAnalysis. They still explain why Engram cannot be treated as optional metadata. It becomes part of the trained model’s reasoning path.

SemiAnalysis reinforced that point by removing Engram during inference. According to its analysis, factual benchmarks retained only 29 to 44 percent of their original performance in the earlier paper’s ablation. Reading comprehension retained 81 to 93 percent.

That loss does not prove Engram beats every conventional model. The network was trained to depend on its memory module. Removing that module creates a training and inference mismatch.

The experiment does show that operators cannot simply delete the table when memory becomes tight. They must serve it efficiently, compress it, or place it elsewhere.

DeepSeek-V4.1-Flash makes the placement problem concrete. DeepSeek says the model has a 552-billion-parameter MoE backbone, with 8 billion parameters active during input processing and 16 billion during output generation. Engram adds another large pool of conditional memory.

The V4.1-Flash release also claims a KV cache requiring one-quarter the HBM and one-eighth the SSD storage of its predecessor. KV cache stores reusable attention state from earlier tokens. It is separate from Engram, but both features reshape server memory requirements.

SemiAnalysis used roughly 189 GiB for the model’s Engram table. Yet each processed token position requested only 24 rows at two Engram layers. That represented about 12.4 KiB across the model, or 3.1 KiB per GPU in a four-GPU configuration.

The table is enormous, but the active read is tiny. That asymmetry is the reason DeepSeek Engram offloading works at all.

DRAM Offloading Pressures the All-HBM Design

Engram weakens the link between total parameter count and required HBM capacity, even when GPU bandwidth remains essential.

HBM has two valuable properties for AI inference. It provides high bandwidth, and it sits close to GPU compute. Those advantages make it the natural home for frequently accessed weights and a growing KV cache.

Capacity remains expensive in system terms, however. An operator often adds GPUs because a model does not fit, even when the workload does not need their full compute. The extra accelerators then introduce communication costs and operational complexity.

Engram offers a way to separate capacity from computation. The server can keep dense layers, active experts, and latency-sensitive state in HBM. It can place the sparse memory table in host DRAM.

SemiAnalysis tested that arrangement through Unified Virtual Addressing, or UVA. UVA lets a GPU address pinned host memory within one shared address space. The same GPU kernel selected and dequantized Engram rows from HBM or DRAM.

This was not ordinary CPU-managed offloading. The GPU directly accessed the pinned host allocation. Async execution allowed retrieval to overlap with other model work.

The surprising result appeared on Nvidia’s B300. SemiAnalysis reported that moving Engram into DRAM allowed its configuration to switch from four-way tensor parallelism to two-way tensor parallelism.

Tensor parallelism splits model operations across multiple GPUs. It can make a large model fit, but every split adds synchronization and communication. Reducing the split can therefore offset the slower memory tier.

The resulting B300 throughput and interactivity frontier improved by as much as 1.6 times, according to the test. Moving the table back into HBM did not deliver a measurable improvement beyond run-to-run variation on that early software stack.

That finding reverses the simplest memory hierarchy argument. HBM made the individual lookup faster, but those lookups represented only one part of the complete serving path. Engram consumed capacity that could otherwise support KV cache, batching, or a smaller replica.

DRAM improved the whole system by changing its topology. The benefit did not come from DRAM beating HBM at raw access speed.

This is the main pressure on GPU system design. Operators have traditionally evaluated model fit by adding the checkpoint size, runtime state, and safety margin. Engram requires them to classify memory by access pattern.

Hot, dense, bandwidth-intensive weights still favor HBM. Predictable sparse rows can tolerate DRAM when compute hides the transfer. Long-tail rows might eventually sit on NVMe, provided caching and software overhead remain controlled.

The shift has implications for memory suppliers. Engram does not end HBM demand. The SemiAnalysis experiments instead suggest that bandwidth can matter more than maximum HBM capacity for some inference configurations.

At the same time, server DRAM becomes part of the model-serving performance envelope. Capacity planning must account for pinned allocations, memory channels, PCIe behavior, and contention across replicas.

NVMe also gains a potential role beyond storing checkpoints and KV cache. Its addressable market grows if model parameters become actively served from local storage. Yet that opportunity depends on software making storage reads economically useful.

The immediate winners are not necessarily the vendors selling the largest memory pool. They are systems that coordinate several tiers without stalling expensive accelerators.

For AI infrastructure buyers, total model size becomes a weaker procurement signal. Active parameters, bytes retrieved per token, prefetch distance, and replica topology offer more useful guidance.

A 748-billion-parameter system can impose very different hardware requirements from a dense model of similar size. DeepSeek-V4.1-Flash combines a sparse backbone with conditional memory and a reduced KV cache. Each part stresses a different resource.

That complexity also makes comparisons harder. A benchmark that measures only tokens per second can hide latency at individual concurrency levels. A cost result can change when a configuration adds GPUs solely for memory capacity.

The most useful comparison maps throughput against user-visible interactivity. It then asks how much useful work each complete server produces, not how quickly one isolated kernel runs.

Why Engram Works Outside GPU Memory

Deterministic addressing gives the runtime time to fetch memory, while sparse access keeps each transfer small enough to overlap with computation.

Engram starts with N-grams, which are short sequences of adjacent tokens. The architecture compresses the tokenizer’s vocabulary, hashes local sequences through multiple heads, and maps those hashes to learned rows.

Vocabulary compression matters because tokenizers often assign different IDs to visually similar text. Capitalization, spacing, and Unicode forms can fragment repeated patterns. The paper reports a 23 percent reduction in effective vocabulary for one 128,000-token vocabulary.

The retrieved embeddings do not enter the model unchanged. A context-aware gate controls how much each lookup influences the current hidden state. Lightweight convolution adds nearby context before the memory contribution reaches later layers.

This design helps explain both Engram’s value and its limitations. The table stores reusable statistical patterns, but the model still decides how much to use them. It is not a conventional database containing clean factual records.

SemiAnalysis scanned high-gate patterns in DeepSeek-V4.1-Flash. It found names, code fragments, relational wording, licenses, bibliography fragments, and web boilerplate. One unusual example referenced the Ace Attorney game series.

These results suggest that the table optimizes next-token prediction, not human judgments about valuable knowledge. Repeated formatting can become useful if it provides a prediction shortcut.

That finding complicates caching. A strong gate does not necessarily identify a frequently accessed row. A row can have high value when retrieved but appear rarely across production traffic.

The runtime also cannot avoid every weak lookup after seeing its gate. Calculating the gate requires the corresponding key, which has already triggered retrieval. Skipping that read would require a separate predictor operating before the memory access.

Access frequency should still follow a long-tail distribution. Common words, programming patterns, and routine syntax should appear more often than rare entities. This makes a multi-level cache plausible.

Frequently requested rows could remain in HBM. A larger working set could stay in host DRAM. Rare rows could live on NVMe and enter faster tiers as traffic makes them useful.

The CXL memory research extends the same idea beyond one server’s directly attached DRAM. Its authors propose pooling Engram memory through Compute Express Link, which supports fine-grained access to shared memory devices.

That proposal remains research, not proof of production economics. CXL adds its own latency, topology, and software considerations. Still, it illustrates how conditional memory changes system boundaries.

An operator could scale the lookup pool separately from GPU compute. Several accelerators might share capacity rather than duplicating the entire table inside every GPU replica.

Prefetching is the central mechanism across these designs. Engram layers cannot sit arbitrarily early if storage latency needs to be hidden. More preceding computation creates a longer window for retrieval.

Model quality can pull in the opposite direction. Early memory intervention helps the network avoid spending its first layers reconstructing common local patterns. Placing Engram too late can weaken that benefit.

This creates a genuine codesign problem. Model researchers choose insertion layers, table dimensions, hashing behavior, and gates. Runtime engineers design prefetch queues, caching, kernels, and transfer paths around those choices.

Hardware teams then decide how much bandwidth and capacity each tier should provide. None of these decisions can be optimized independently.

DeepSeek’s paper reported a U-shaped relationship between parameters allocated to MoE computation and parameters allocated to Engram memory. Too little memory left repeated patterns to the neural backbone. Too much memory reduced the compute available for other tasks.

That result argues against treating Engram as unlimited cheap capacity. More table rows can improve validation loss, but only within a balanced system. Training data quality also determines what those rows learn.

The broader architectural implication is important. Scaling no longer means placing every new parameter beside GPU arithmetic. A model can add capacity through predictable sparse retrieval, while reserving expensive bandwidth for dynamic computation.

SSD Offloading Is Not Yet the Cheap Answer

The first NVMe experiment saved committed DRAM capacity, but its unoptimized data path lost to pinned DRAM on both speed and efficiency.

SemiAnalysis replaced the 189 GiB Engram allocation with memory-mapped files on local SSDs. A memory-mapped file lets the operating system expose storage through virtual memory pages. Recently accessed pages can remain in the filesystem cache.

That approach offers operational flexibility. The operating system can reclaim cached pages when other applications need memory. Operators avoid permanently pinning the complete Engram table in DRAM.

However, a warm page cache does not make the SSD path equivalent to native DRAM offload. The prototype still moved row identifiers to the CPU, removed duplicates, gathered rows into pinned buffers, and copied them back.

It then dequantized the selected rows on the GPU. These steps ran between segments of GPU execution rather than through the direct UVA kernel used for pinned host memory.

Every step adds coordination. CPU scheduling, identifier transfers, row gathering, buffer management, and GPU copies can cost more than the physical storage read. A cached file can therefore trail pinned DRAM even when no request reaches NAND flash.

The B200 results exposed that gap. Near 125 tokens per second for each user, the DRAM configuration produced 121 million total tokens per unit of spending. The SSD path produced 52 million.

DRAM therefore delivered roughly 2.3 times as many tokens under that measured comparison. It also dominated the observed SSD configurations on P90 interactivity, which measures the experience of slower requests near the tail.

The researchers could not enable GPUDirect Storage, or GDS, for the test. GDS can move data between NVMe and GPU memory with less CPU involvement. Its absence limits what the experiment says about an optimized storage path.

The result should not be framed as proof that NVMe cannot serve Engram. It shows that cheap storage media does not automatically create cheap inference.

The four B200 GPUs remained in the server. So did the CPU, networking, power, and most supporting infrastructure. Replacing DRAM with SSD reduced one capacity requirement without reducing the expensive parts of the configuration.

An economic win requires a second effect. SSD offloading must enable a cheaper server, fit more productive replicas, support a larger model, or release DRAM for another valuable workload.

The prototype achieved none of those outcomes in its measured setup. Its filesystem cache could also consume much of the DRAM that file backing was intended to save.

Workload locality creates another uncertainty. A stable service with repeated prompts might maintain a useful warm cache. A diverse agent workload could touch a wider range of N-grams and produce more storage misses.

Batching changes the equation again. Multiple requests can reuse rows within a batch, while deduplication reduces transfers. Yet larger batches also increase KV cache requirements and can alter latency.

SSD hardware specifications alone will not predict the outcome. Random-read latency, queue depth, filesystem behavior, page size, CPU overhead, and cache policy all contribute.

Recommendation systems provide a useful historical precedent. They have served large embedding tables from tiered memory for years. Some systems cache popular rows in DRAM while placing the long tail on SSDs.

Language-model Engram traffic is similar enough to borrow ideas, but not identical. Autoregressive generation imposes tight sequential latency. A delay at one token position can stall later positions.

AgentX makes that concern more visible. It represents long-context, multi-turn agentic coding traffic rather than one short synthetic prompt. Such workloads alternate substantial input processing with latency-sensitive generation.

The storage case will improve only when software treats Engram as a first-class serving primitive. A generic memory-mapped file is useful for experimentation, but it leaves too much work in the CPU-controlled path.

Future designs need direct I/O, asynchronous queues, row-aware caching, and predictable prefetch schedules. They may also require storage layouts aligned with Engram’s quantized rows.

NVMe therefore remains an opportunity rather than the current default. DRAM has already shown that tiering can improve the system. SSD still needs a serving stack designed around the model’s sparse access pattern.

AgentX Shows the Software Moat Around New Architectures

Engram changes memory placement, but launch-day software support still determines which accelerator can turn the architecture into usable service.

SemiAnalysis evaluated DeepSeek-V4.1-Flash across Nvidia H100, H200, B200, B300, GB200, and GB300 systems. It also tested AMD’s MI355X through the InferenceX framework.

The InferenceX measurements use real hardware and trace throughput against interactivity. AgentX supplies a long-context coding workload intended to resemble repeated tool-oriented sessions.

According to SemiAnalysis, Nvidia’s vLLM path worked across all six tested Nvidia platforms when the model launched. AMD’s referenced container image was not publicly available during the first 23 hours.

AMD support arrived later, but the early performance gap remained significant in the analyst’s measurements. Seven days after release, SemiAnalysis placed MI355X performance per unit of spending between two and four times behind B200.

These are vendor-sensitive claims from one benchmarking organization. They depend on cloud assumptions, runtime versions, quantization, topology, and rapid software changes. They should not be generalized to every model or AMD workload.

They do reveal an important constraint. An architecture can be designed for offloading, but deployment still depends on kernels, graph capture, memory addressing, quantization, and distributed scheduling.

DeepSeek Engram offloading needs more than adequate PCIe bandwidth. The runtime must overlap transfers without breaking decode graphs. It must select and dequantize rows efficiently on each platform.

A new model therefore tests an accelerator ecosystem before hardware teams have months to tune it. Maintainer familiarity and working release pipelines become part of performance.

This dynamic strengthens Nvidia’s software position even when model architecture reduces the HBM capacity required per replica. Fewer GPUs per replica can lower communication, but each remaining GPU needs mature kernels and runtime integration.

The effect on total accelerator demand is not straightforward. Better memory use can reduce the number of GPUs required for one model replica. Lower serving costs can also expand demand by making more applications economically viable.

Engram could encourage larger models because conditional memory scales separately from active computation. Operators might use the saved HBM for more concurrent sessions rather than buying fewer accelerators.

DeepSeek has also reduced KV cache requirements in V4.1-Flash, according to its release materials. Engram offload and KV compression together free memory from two different directions.

That combination matters for agentic inference. Agents carry long histories, tool outputs, code, and intermediate plans. Their KV cache can occupy substantial capacity even when model weights already fit.

A server that moves sparse lookup tables to DRAM can devote more HBM to those active sessions. The practical benefit may appear as higher concurrency, not a smaller physical model.

The skeptical view remains necessary. SemiAnalysis’s Engram reproduction used released code and training settings because DeepSeek did not release the original paper’s two trained research models. Production behavior can differ from controlled scaling experiments.

The SSD path was explicitly unoptimized. Nvidia and AMD results captured early software stacks that will change. Even the strongest DRAM result depended on a specific B300 topology and workload frontier.

Engram also stores whatever training rewards. More capacity can preserve boilerplate and incidental patterns alongside useful entities or code. Better memory allocation does not guarantee better data selection.

The evidence supports a narrower conclusion. Conditional memory is technically compatible with lower memory tiers, and DRAM can improve complete serving performance. It does not yet establish one universal configuration.

Three Signals Will Decide the DRAM and NVMe Impact

The next phase depends on optimized SSD serving, repeatable topology gains, and broad runtime support beyond one model release.

The first signal is an optimized NVMe implementation using a direct GPU-oriented path. The important comparison is not peak SSD bandwidth. It is the complete throughput and P90 interactivity frontier against pinned DRAM.

Support for GPUDirect Storage would remove part of the CPU round trip. Native row deduplication, async prefetching, and Engram-aware storage layouts could remove more overhead.

If an optimized path approaches DRAM performance while materially reducing required host memory, the NVMe opportunity becomes credible. If cached SSD remains far behind, Engram will strengthen DRAM demand instead.

The second signal is whether smaller replica topologies repeat across hardware and workloads. SemiAnalysis’s B300 move from four-way to two-way tensor parallelism produced the most consequential result.

Independent tests should examine B200, H200, MI355X, and future systems. They should include varied context lengths, batch sizes, concurrency levels, and cache states.

Repeated gains would confirm that Engram offload changes server economics through reduced communication. Failure to reproduce them would narrow the finding to one early configuration.

The third signal is ecosystem adoption. DeepSeek-V4.1-Flash provides a visible test, but Engram-like memory must spread across more model families and serving engines to reshape hardware demand.

The SemiAnalysis report points to related N-gram mechanisms in LongCat and Qwen models. Researchers have also proposed pooled Engram capacity through CXL. These approaches need stable support in vLLM, SGLang, and vendor runtimes.

Broad adoption would strengthen demand for balanced servers with more DRAM bandwidth, capable local NVMe, and flexible interconnects. Limited adoption would leave Engram as an important DeepSeek-specific optimization.

Developers should watch deployment recipes, not parameter headlines. Enterprise buyers should request performance at their own context lengths and latency targets. Infrastructure teams should measure cold, warm, and mixed cache states separately.

Knowledge workers will experience the outcome indirectly. Better memory placement can support longer sessions, lower latency, and wider access to capable models. Those benefits matter only when complete applications use them reliably.

Teams evaluating AI systems should preserve their own evidence alongside benchmark claims. A searchable engineering knowledge base can keep model cards, runtime versions, and internal test results connected.

DeepSeek Engram offloading is not the end of HBM-led inference. It is evidence that model architecture can decide which parameters deserve HBM in the first place. The next question is whether optimized NVMe and shared memory can repeat DRAM’s result without moving the bottleneck into software.