top of page

tandem.cpp Update Adds AMD Support and SSD Gains

46 minutes ago
11 min read
Illustrative photograph. Photo: Bibek ghosh / Pexels.

The tandem.cpp update reports DeepSeek-V4.1-Flash running at 14.2 tokens per second on a PC with 64 GB of memory and a 16 GB graphics card. Its author lists plain llama.cpp at 7.4 tokens per second on the same configuration. The claim concerns faster inference, meaning running an existing model, while much of its data remains outside graphics memory.

The important detail is where that data lives. The tested model’s expert weights occupy 139 GiB, according to the author, exceeding the machine’s system memory. Experts are specialized model components selected for particular inputs. The patch keeps frequently used experts on the graphics card and coordinates remaining work across the processor, memory, and storage.

That puts tandem.cpp against a practical constraint: how much waiting users will tolerate when their preferred model exceeds available memory. Its reported advantage over llama.cpp is substantial on the highlighted machine. Comparisons with Strata, another inference engine, show a narrower and more complicated contest. These are author-reported measurements, not independently reproduced results.

What the tandem.cpp update actually changes

The update targets computers that cannot keep a large model entirely in their fastest memory. It also adds an AMD execution path, broadening the hardware owners can investigate.

In the author’s DeepSeek offload benchmarks, the central configuration combines an RTX 5070 Ti, 64 GB of RAM, and an i5-14600KF. RAM is the computer’s working memory. The graphics card has 16 GB of VRAM, its dedicated memory for graphics and computation.

On that machine, the author reports 14.2 tokens per second for tandem.cpp, 7.4 for llama.cpp, and 10.4 for Strata. Tokens are the text units a model consumes and generates. These numbers measure decode speed, the rate at which an engine produces output after processing a prompt.

The benchmark uses DeepSeek-V4.1-Flash IQ2_XS. That final label identifies a quantization format, which stores model weights with reduced numerical precision. The author says each engine used the same quantized model, making this a comparison between execution approaches rather than differently compressed models.

The stated conditions also matter. The tests involve one user, GSM8K math questions, and greedy decoding, which selects the highest-scoring next token. The author says none of the compared engines skipped selected experts. Those controls narrow the comparison, although they do not independently validate its implementation.

A second machine pairs the same graphics card and memory capacity with an i5-12400F and a budget SSD without dedicated onboard DRAM. The reported result is 10.4 tokens per second for tandem.cpp and 5.7 for Strata. The post supplies no llama.cpp result for that row.

That missing entry should remain missing. It does not establish a failure, an inability to load the model, or a particular performance gap. It simply prevents a complete comparison across all engines on that configuration.

The larger configuration combines an RTX 5090 with 32 GB of graphics memory, 128 GB of RAM, and an i5-14600KF. Here, the author reports 33.1 tokens per second for tandem.cpp, 11.5 for llama.cpp, and 28.0 for Strata.

Taken together, the rows support a specific claim worth testing: coordinating memory and storage can improve generation speed on oversized models. They do not establish that any computer with a 16 GB graphics card will deliver the highlighted result.

This is also an inference-engine update, not a DeepSeek model announcement. The patch changes how an existing model runs. It does not train a new model or establish that the compressed version matches the original model’s quality.

The pressure falls on memory management, not just graphics cards

The consequential comparison is between execution strategies under the same memory constraint. A faster graphics card alone does not explain the differences reported between engines on identical hardware.

Consider the reader who already owns the highlighted class of desktop. Their immediate question is whether software can make an oversized model usable without replacing the machine. The author’s figures give that question a concrete experimental starting point.

They also shift attention toward the complete computer. Once a workload depends on storage, the processor, available RAM, and drive behavior belong in the performance discussion. Treating graphics memory as the only meaningful specification misses the configuration described here.

That does not mean memory capacity stops mattering. The central limitation remains visible in the setup: the expert data exceeds available RAM. Software must manage movement and computation around that shortage. The reported improvement makes the shortage less expensive in time; it does not remove it.

For developers choosing an engine, this creates pressure to compare the same model under matching conditions. A result measured with plentiful memory answers a different question from one measured while repeatedly accessing storage.

The author’s Qwen comparison reinforces that distinction. Qwen3.8-Flash-Next IQ3_S, listed at 78 GiB, reportedly reaches 34.3 tokens per second on a 32 GB PC with a 16 GB RTX 5070 Ti. The corresponding figures are 14.9 for llama.cpp and 30.3 for Strata.

That test includes MTP, or multi-token prediction, an approach that proposes multiple future tokens for validation. It belongs to the stated Qwen configuration. Readers should not transfer its result to the DeepSeek configuration or assume the same acceleration applies universally.

The practical stakes differ across applications. Someone experimenting with a local assistant might accept a noticeable startup delay followed by steady output. A developer running repeated short requests might care more about the delay before each response begins.

A background job introduces another standard. Its owner might accept slower interaction if the job completes reliably while leaving enough resources for other applications. A benchmark that occupies an otherwise idle computer does not automatically represent that experience.

These are evaluation scenarios, not documented deployments of tandem.cpp. Their purpose is to show why a single generation-speed figure cannot settle an engine choice.

For teams considering local document workflows, the model is also only one component. Retrieval, source access, and answer checking remain separate concerns. A searchable knowledge base addresses a different layer from the inference engine’s memory strategy.

The reasonable pressure on engine developers is therefore specific. Publish measurements that reveal when memory coordination helps, when it stops helping, and which other parts of the user experience remain slow.

Why moving fewer experts can matter more than fitting everything

The mechanism depends on selective computation and placement, while preserving the work required by the tested model. It does not require the entire model to reside on the graphics card.

A mixture-of-experts model, commonly shortened to MoE, selects subsets of specialized components for each input. Its total stored parameters can therefore exceed the subset used during an individual computation. That distinction creates opportunities for different memory arrangements.

The author describes tandem.cpp as keeping frequently used experts on the graphics card while letting the processor help with others. When the model exceeds system memory, storage also participates. The engineering question becomes which data to keep close and how to handle the rest.

A cache is a faster storage area that retains data for reuse. In this setting, useful cached experts can avoid repeated transfers. A cache miss occurs when required data is absent and must come from elsewhere.

The analytical implication is straightforward: avoiding a slow transfer can matter even when the amount of arithmetic stays unchanged. That is why a memory-management patch deserves attention alongside improvements to numerical computation.

There are limits to that reasoning. A placement strategy benefits from the actual pattern of expert use. Different prompts and models can create different patterns. A favorable result on math questions does not establish the same advantage for code generation or long document analysis.

The baseline also deserves accurate treatment. Local model inference through llama.cpp provides the underlying engine that tandem.cpp patches. This is a comparison within an existing software foundation, rather than evidence that the foundation has become irrelevant.

That relationship matters for maintenance. Readers should evaluate both the measured benefit and the consequences of using a modified engine. Compatibility with their model, application, and future software updates belongs in the decision.

The project’s November 16 repository update adds another relevant development. Its expert-cache changes describe larger prompt-processing batches and temporary use of cache memory for processing buffers. The author reports improved prompt latency, while showing that Strata remains faster in some comparisons.

This newer material matters because prompt processing and text generation are different stages. Prefill is the initial processing of input tokens. Decode is the subsequent production of output tokens. Improving either stage does not automatically improve the other.

For someone submitting a long document, a faster output stream can follow a substantial initial wait. That user experiences the total interaction, not merely the portion highlighted by a decode benchmark.

The tandem.cpp update should consequently be read as work on a memory-constrained execution pipeline. Its appeal comes from improving how existing resources cooperate. The evidence does not support describing it as a way to make model size irrelevant.

Nor does avoiding expert skipping establish equivalence to an uncompressed model. The chosen quantization remains part of the experiment. Preserving selected computation and preserving original numerical precision are separate claims that require separate evidence.

AMD support broadens access without establishing a speed lead

AMD compatibility is a meaningful addition, but the reported AMD results do not put tandem.cpp ahead of every alternative. The author’s own comparison makes that limitation clear.

The patch now builds with HIP, AMD’s C++ runtime and programming interface for GPU computation. AMD describes HIP programming as part of ROCm, its software platform for heterogeneous computing across processors and graphics hardware.

That distinction helps interpret the announcement. Supporting a programming interface opens an execution path. It does not establish equal performance across all supported cards, processors, operating systems, or software configurations.

The author reports testing an RX 7900 XTX with an i7-8700, 64 GB of DDR4 memory, and a PCIe 3.0 connection. PCIe is the interface connecting components such as graphics cards to the computer. The workload uses Qwen3.8 IQ3_S with MTP.

On that setup, tandem.cpp reportedly delivers 68.6 tokens per second. Stock Strata reaches 69.7, while an RDNA3-focused Strata fork reaches 80.1. RDNA3 identifies the graphics architecture used by the tested AMD card.

Those figures support a restrained conclusion: tandem.cpp runs competitively with stock Strata in the author’s test, while the specialized fork runs faster. They do not support calling tandem.cpp the fastest ROCm engine.

The post also lists Avalon at 83.6 tokens per second. However, that entry uses its own IQ3_KS quantization. Because the stored model representation differs, the number should not be presented as a controlled engine-only comparison.

This is where benchmark presentation can quietly become misleading. Placing results in the same list makes them easy to scan, but does not erase differences in model files. A useful comparison explains those differences before drawing a ranking.

The author also says Vulkan support is absent. Vulkan is another graphics and compute API. Readers whose preferred setup depends on it should not treat the HIP announcement as evidence that their existing software route is covered.

RDNA4 remains a separate uncertainty. In the discussion, the author says they cannot test that architecture with their available hardware. An expectation that code should work is weaker evidence than a successful build, a completed workload, and reproducible measurements.

For AMD owners, the immediate value is therefore an additional option to evaluate. The announcement expands the set of plausible experiments without settling which engine best serves every Radeon configuration.

It also illustrates the main narrative’s boundary. Coordinating memory can help, but the effectiveness of that coordination still depends on the execution platform. Shared code does not make hardware-specific constraints disappear.

The appropriate response is to compare like with like on the intended machine. Architecture names and compatibility labels provide context; they cannot replace measurements of the application readers actually intend to run.

What the benchmarks leave unresolved

The largest verification gap is the distance between an author’s controlled comparison and another user’s complete workload. Publicly available claims make investigation possible, but they do not constitute independent replication.

The author states that the DeepSeek comparison uses identical quantization, greedy decoding, and no expert skipping. Those are useful controls. They reduce obvious sources of mismatch between engines while leaving other questions open.

The central unresolved issue is reproducibility. Another evaluator needs enough information to match the model files, software versions, configuration, and measurement procedure. Agreement across those details would make a repeated result more informative than a loosely similar screenshot.

Storage conditions deserve particular attention. A cold run begins without the same useful data already retained in memory. A warm run can reuse previously accessed data. Comparing different starting conditions risks attributing a memory-state advantage to the engine itself.

The desktop’s other activity also matters for a practical assessment. Applications compete for working memory and processing time. A configuration that performs well in an isolated test deserves another look under the owner’s normal usage conditions.

That is an evaluation concern, not evidence of a defect in tandem.cpp. The available benchmark simply cannot answer every question about how a machine behaves during ordinary work.

The test workload imposes another boundary. GSM8K contains math word problems. Using its questions to measure generation speed does not automatically establish mathematical accuracy, coding ability, factual reliability, or successful tool use.

A model can generate an incorrect answer quickly. Conversely, a slower configuration might still be adequate for a task whose main requirement is careful verification. Throughput and usefulness need their own measurements.

The same caution applies to the word “exact.” In this comparison, the author uses it to distinguish computation without expert skipping. It should not become a blanket statement about identical outputs across hardware or equivalence to the original model.

Quantization adds its own question. The reported DeepSeek configuration uses a compressed representation. An evaluation of that representation’s answer quality would require evidence beyond the engine speed figures discussed here.

Energy use, storage activity, responsiveness, and stability are also outside the headline result. Readers should avoid inferring improvement in those areas solely because output arrives faster. Different users will assign them different importance.

There is no independent enterprise adoption evidence in the cited material. The results therefore support investigation by technically comfortable users, rather than a claim that organizations have already validated the approach for production.

The author’s own concession about Strata helps keep the discussion grounded. In the Reddit thread, they acknowledge that Strata can be better when sufficient memory is available. That is consistent with a workload-dependent advantage rather than a universal winner.

A cautious reading does not make the update uninteresting. It identifies the actual proposition worth examining: whether this patch improves a specific oversized model on a specific constrained machine.

That proposition is both useful and testable. Expanding it into claims about every model, every AMD card, or every local AI workflow would go beyond the evidence.

Three signals will show whether the gains travel

The next useful evidence should test reproducibility, AMD coverage, and complete response time, in that order. Those signals address the largest gaps without turning the update into a prediction about the whole AI market.

First, watch for independent replication of the memory-constrained DeepSeek results. The strongest reports would identify the exact model files, engine versions, hardware, settings, and starting memory conditions. They would also report variation across repeated runs.

A matching result would strengthen the claim that the gain survives outside the author’s environment. A smaller gain would still be informative if the difference could be connected to storage, processor behavior, or configuration.

A reversal would matter too. If another engine wins under matched conditions, the boundary of tandem.cpp’s advantage becomes clearer. That would help users choose software rather than merely add another number to a leaderboard.

Second, watch for AMD tests beyond the reported RX 7900 XTX configuration. RDNA4 results would directly address the author’s stated hardware gap. Reports should distinguish successful compilation from correct execution and sustained application performance.

Broader AMD coverage would strengthen the compatibility story. It would not erase the existing Strata fork result or establish a permanent ranking. Different architectures need their own evidence, especially when the host processor also participates in inference.

Third, watch complete interaction timing on realistic prompts. That means measuring the initial wait, output generation, and successful completion together. The November 16 prompt-processing work makes this a particularly relevant next step.

Long document requests and repeated conversational turns would test different demands from a short math prompt. Independent evaluators should explain the workload rather than present an isolated throughput number as a complete user-experience score.

These signals are worth following over the coming months, but they are not promised milestones. Neither the source post nor the repository establishes a deadline for independent validation, broader hardware testing, or application-level comparisons.

For now, the tandem.cpp update offers a credible subject for hands-on evaluation, supported by detailed author claims and candid competitive limits. Its strongest reported case is a machine forced to move expert data beyond available memory. The practical next step is to compare that case with your own workload. Start with the same model files and a clearly recorded baseline. Measure the wait before output as carefully as the output rate. Check the answers before treating speed as progress. If the gain persists under those conditions, it becomes evidence for your machine rather than a borrowed headline. Which matters most for your local workload: steady generation, a shorter initial wait, or keeping the desktop responsive?

Give every agent the context to do better work

Connect your agents to the knowledge, decisions, and history already organized in remio.

remio currently supports Windows 10+ (x64) and Macs with Apple silicon.

Your AI Partner at Work
Get more done with remio

Plan. Create. Deliver.
All in one place.

bottom of page