llama.cpp CUDA Fusion Merges With Up to 10.7% Faster Prompt Processing

llama.cpp merged a CUDA optimization on October 25, with contributor benchmarks reporting prompt-processing gains reaching 10.7% on DGX Spark. The llama.cpp CUDA fusion change combines operations inside a model’s feed-forward network, the component that transforms token representations between attention stages. Its benefit concerns reading prompts, rather than producing every subsequent answer token.

Contributor anujj’s measurements also show improvements on NVIDIA’s RTX 5090, although the reported percentage gains are smaller. These are author-reported comparisons, not independently reproduced results. The underlying change is verifiable in the merged code, while its practical value still depends on model format, hardware, and workload.

The central comparison is between fused execution, which combines related operations, and an existing execution path that schedules those operations separately. An earlier proposal pursued the same opportunity differently. The merged approach reorganizes how values reach GPU threads, reducing intermediate work without changing the stored model weights. For developers running long prompts locally, that makes this a specific software improvement worth measuring against an existing deployment.

What the llama.cpp CUDA fusion merge changes

The merged change extends an existing optimization to larger prompt-processing workloads on supported NVIDIA hardware.

CUDA, NVIDIA’s GPU computing platform, provides the execution environment for this implementation. MMQ, llama.cpp’s quantized matrix multiplication path, handles calculations involving compressed model weights. Quantization represents those weights with reduced numerical precision to lower storage and memory requirements.

The merged fusion change targets a recurring sequence inside supported dense feed-forward layers. Separate gate and up projections transform the same input. A gated linear unit, or GLU, then combines their results through an activation and elementwise multiplication.

Previously, larger workloads on this path handled the projections and their combination as separate operations. According to anujj, the new implementation executes that sequence through a single matrix multiplication kernel. A kernel is a function launched across GPU threads.

That consolidation matters because the surrounding work also consumes resources. Preparing inputs, launching operations, writing intermediate results, and reading them again can all add overhead. Combining operations creates an opportunity to remove some of that traffic while retaining the intended mathematical transformation.

The change does not introduce this idea across every execution path simultaneously. The contributor says small-batch fusion already existed through other matrix-vector implementations. Matrix-vector calculations typically serve smaller workloads than the matrix-matrix calculations emphasized here.

That distinction explains why the announcement centers on prompt processing. Prefill, the phase that processes an input prompt before answer generation, can expose larger batches of work. The newly supported execution path addresses that part of inference.

The chronology also matters. The pull request’s opening description dates to October 4, while the project merged it on October 25. The merge moves the implementation into the project’s master branch. It does not establish that every packaged application already includes the change.

A desktop application, container, or downstream project can incorporate llama.cpp on its own schedule. Users therefore need to identify the actual runtime version behind an interface. An application update and an upstream source merge are separate events.

For existing model files, the compatibility story is relatively straightforward. The contributor states that the stored weight layout remains unchanged. Users should not interpret the title’s reference to interleaved rows as a requirement to download specially rearranged model files.

The immediate news is therefore narrow and concrete. A supported execution sequence now has an additional optimization available upstream. Whether a particular request uses it depends on the runtime’s eligibility checks.

The benchmark gains favor Spark, with a smaller lift on RTX 5090

The reported results show a consistent direction across the listed configurations, but their magnitude varies with hardware and quantization.

The author tested DGX Spark using its GB10 processor and separately tested an RTX 5090. Both benchmark tables compare a baseline labeled master against the pull request. Every listed row reports higher prompt-processing throughput for the modified implementation.

On DGX Spark, the largest listed percentage improvement belongs to Llama-3.1-8B using IQ4_XS, a compressed weight format. Throughput increases from 2,933.9 to 3,249.0 tokens per second. The author reports a 10.7% gain for that configuration.

Other Llama-3.1-8B results on Spark fall below that peak while retaining substantial reported improvements. The Q4_K_M configuration moves from 2,749.3 to 3,000.6 tokens per second. The Q5_K_M configuration increases from 2,699.4 to 2,960.3 tokens per second.

Those labels identify different quantized model configurations, rather than interchangeable measurements of an identical binary. Their results show why users should match the exact file format when assessing an optimization. A model family name alone leaves out a relevant variable.

The Spark table also lists models labeled Qwen3.8-27B and Qwen3.6-27B. Those names are reproduced here as benchmark labels from the contributor’s report. The performance table itself does not independently establish their training details, capabilities, or broader availability.

For the entry labeled Qwen3.8-27B, Spark improvements range from 4.6% to 7.4%. The Q4_K_M row rises from 840.1 to 899.9 tokens per second. The IQ4_XS row increases from 882.4 to 948.0 tokens per second.

The RTX 5090 results present a different scale of improvement. Across its listed configurations, the author reports gains ranging from 1.9% to 4.7%. That smaller percentage range makes hardware-specific interpretation essential.

For Llama-3.1-8B in IQ4_XS, the RTX 5090 result increases from 12,997.6 to 13,603.3 tokens per second. The reported improvement is 4.7%. For the Qwen3.8-27B Q6_K entry, throughput rises from 3,220.5 to 3,281.0 tokens per second, or 1.9%.

The llama.cpp fusion benchmarks therefore support a bounded conclusion. In the author’s selected long-prompt tests, the optimization improves measured input throughput across the reported matrix. They do not establish an identical uplift for every supported GPU.

Nor does the percentage comparison reveal why Spark benefits more. Different memory behavior, execution characteristics, and existing bottlenecks can affect optimization results. Establishing their relative contribution would require additional profiling, which these headline measurements alone do not provide.

For buyers, the useful implication concerns evaluation discipline. These results do not justify choosing hardware from percentage gains alone. Absolute throughput, model fit, memory capacity, and actual application latency remain separate considerations.

Interleaved rows remove work between calculations

The implementation’s central move is to arrange related results together before applying the gate operation.

Anujj describes loading gate and up rows in alternating chunks of eight. This arrangement places corresponding values within the same thread’s working results. The thread can then combine those values when writing the final output.

Registers, the GPU’s small storage locations assigned to executing threads, hold those intermediate results. Applying the combination there avoids a separate round trip through intermediate output buffers for that portion of the computation. The intended saving comes from execution organization.

The implementation keeps one accumulator arrangement rather than introducing a second accumulator tile for the gate projection. An accumulator holds partial sums during multiplication. A tile is a block of matrix elements processed together.

That design choice distinguishes this implementation from the earlier fusion proposal. According to the contributor’s description, the newer approach follows a suggestion from am17an. Both proposals target related operations, but they organize the calculation differently.

The distinction is practical rather than cosmetic. Holding more intermediate state can increase pressure on a GPU’s limited execution resources. Combining operations only helps when the savings exceed any additional scheduling or storage cost.

This is why “fewer kernels” does not automatically mean “faster inference.” A larger combined operation can use hardware less efficiently under some conditions. An optimization must preserve enough parallel work while avoiding new constraints that cancel its benefits.

The merged code reflects that concern through eligibility checks. It evaluates whether the workload is appropriate for this fusion path. Supported mathematical operations alone do not guarantee that fusion should run.

The implementation also competes with stream-k, a scheduling method that divides matrix multiplication work along its reduction dimension. The fused path cannot use that decomposition. It therefore needs another way to decide whether its tile arrangement provides sufficient efficiency.

Johannes Gaessler raised that issue during review, requesting an advance comparison between expected tiling efficiency and stream-k. The merged code includes an efficiency decision before selecting the fused path. That makes conditional execution part of the design.

The fusion scheduling review provides the most useful skeptical perspective on the optimization. It focuses on circumstances where an attractive simplification could lose performance. That is a technical limitation, rather than a disagreement about the benchmark’s existence.

The primary competition here consequently remains inside the runtime. Fused execution must beat the existing scheduling approach for the eligible operation shape. There is no evidence in these results of a broader victory over another inference framework.

This mechanism also clarifies what users should expect from llama.cpp CUDA fusion. It reorganizes work for an existing calculation. It does not increase a model’s reasoning ability, expand its trained knowledge, or remove unrelated memory limits.

The same model still performs the same general task. The engineering opportunity lies in spending less time reaching the output of a particular sequence.

Faster prompt processing changes one part of the waiting time

The benchmark measures input processing, so its practical value depends on how much input processing contributes to a real request.

The reported command processes a prompt containing 16,384 tokens and requests zero generated tokens. Tokens are the text units a model consumes and produces. This setup isolates prompt throughput instead of measuring a completed conversational response.

The command uses batch and microbatch sizes of 2,048. A microbatch is a smaller execution unit carved from a larger batch. It also enables Flash Attention, an attention implementation designed to reduce memory traffic during attention calculations.

The author specifies half-precision storage for the key-value cache, which holds attention information from previously processed tokens. Five repetitions appear in the command. These settings define the tested conditions and should travel with any quoted result.

The project’s benchmark methodology distinguishes prompt processing from token generation. That separation is crucial for reading these numbers. Input tokens per second and output tokens per second answer different performance questions.

Consider a developer asking a local model to inspect a large source file. Processing that input can contribute meaningfully to the delay before the answer begins. A faster eligible prefill path addresses that portion of the interaction.

A brief conversational exchange presents another workload. Its prompt might require relatively little processing compared with the time spent generating a long answer. The same input-throughput improvement would then affect a smaller share of the total wait.

Cached prompts create another distinction. Prompt caching reuses computation from input the system has already processed. When an application successfully reuses that work, less fresh input remains for this optimization to accelerate.

Document analysis also includes work beyond inference. An application might retrieve files, extract text, rank passages, or prepare an instruction before sending anything to the model. Faster model input processing does not establish that those surrounding stages finish sooner.

For teams maintaining a searchable knowledge base, that separation supports better performance decisions. Retrieval quality and context selection determine what reaches the model. Runtime throughput determines how quickly eligible calculations process it.

The distinction also places pressure on application maintainers. They need to know which runtime their product ships and whether its common workloads reach the optimized path. A generic release note about faster local AI cannot answer those questions.

The useful response is measurement at the application boundary. Time to first token measures the delay before output begins. Total response time measures the entire interaction, while generation throughput measures the pace after output starts.

Each metric can move differently after a runtime update. A prefill improvement can reduce initial waiting without changing the rate of subsequent text. Reporting those outcomes separately gives users a more accurate account of the upgrade.

For llama.cpp prompt processing, the strongest use case remains fresh, substantial input on an eligible configuration. The author’s benchmark deliberately emphasizes that workload. Extending its result to every chat session would go beyond the evidence.

Supported formats and correctness checks set the boundaries

The merged code contains meaningful restrictions, and the author’s correctness report should be read within its stated test coverage.

The pull request describes support for Q4_K, Q5_K, Q6_K, Q8_0, and IQ4_XS weight types on NVIDIA Turing or newer architectures. Those are supported weight families. Model files can combine types, so a file’s overall label does not describe every internal tensor.

The implementation also checks that relevant gate and up weights match the conditions required for fusion. Shape, type, and layout matter because the combined operation expects corresponding calculations. An unsupported arrangement should retain another execution path.

The merged code excludes cases requiring its row-boundary fallback. A fallback handles shapes that do not fill the normal processing tiles cleanly. That restriction matters because the original description discusses tests covering boundary cases during development.

Readers should therefore distinguish the opening test summary from the final code’s selection rules. Testing a broader development implementation does not establish that every tested shape remains eligible in the merged version. The final guards determine actual execution.

There are restrictions on operation parameters as well. The code checks particular activation settings before selecting fusion. These checks prevent the runtime from treating superficially similar operations as interchangeable when their required behavior differs.

The scope also centers on dense feed-forward operations. Mixture-of-experts models, which route tokens through selected subnetworks, introduce different execution requirements. The review explicitly identifies additional support work rather than establishing universal coverage for those models.

Other hardware backends do not receive this optimization merely because they support llama.cpp. The contributor explicitly separates the change from other backends. Results from NVIDIA hardware therefore cannot establish equivalent benefits on an Apple GPU or an AMD configuration.

Correctness evidence is encouraging but remains author-reported. Anujj says the full backend-operation test suite passes on GB10. The report also states that matrix-multiplication suites pass on an RTX 5090.

The contributor additionally reports identical perplexity results across the specified tests with fusion enabled and disabled. Perplexity measures how well a model predicts an evaluation text. It provides a useful numerical check, although it does not represent every possible application behavior.

That result supports a narrower statement than universal output identity. Identical scores on a stated evaluation do not prove identical generated answers across all prompts, settings, or platforms. Broader validation requires broader evidence.

The llama.cpp fusion benchmarks also lack an independent reproduction in the sources examined for this report. Upstream review and merge establish that the project accepted the code. They do not turn contributor measurements into an external performance audit.

None of these boundaries erase the reported gain. They define where the claim becomes useful. A reader with matching hardware and model types has a concrete reason to test, while other configurations require separate evidence.

The strongest interpretation combines both sides. The change has a clear mechanism and published measurements, alongside explicit conditions governing when it runs.

Three signals will show how broadly the improvement travels

The next evidence should connect upstream code, reproducible measurements, and the latency users experience.

The first signal is downstream adoption. Over the coming months, application releases and packaged builds should establish which products include the merged implementation. That matters because users often interact with llama.cpp through another application.

The project’s CUDA build instructions describe the upstream compilation route. Packaged distributions can follow different release schedules. A version identifier tied to included source is more informative than a general claim of using the latest backend.

If widely used applications incorporate the change, the immediate adoption case strengthens. If they remain on older revisions, the merge’s practical reach remains limited. That outcome would concern distribution timing rather than invalidate the underlying optimization.

The second signal is independent reproduction with explicit settings. Useful reports should identify hardware, model files, runtime revisions, and benchmark arguments. They should also separate fresh prompt processing from cached input and generated output.

Reproductions across different prompt lengths would clarify how sensitive the gain is to workload size. Reports across supported GPU generations would test whether the direction persists beyond the listed machines. Results showing regressions would help identify limitations in the selection logic.

The relevant question is not whether every run matches the headline maximum. The question is whether comparable workloads show a repeatable improvement with acceptable variation. A smaller consistent gain can still be operationally useful.

The third signal is application latency. Teams should measure whether an updated runtime reduces the wait before an answer begins on representative requests. They should retain generation throughput and total response time as separate measurements.

That evidence would connect the kernel change to user experience. A lower initial delay on long, uncached prompts would support the implementation’s practical promise. An unchanged total response time would indicate that other stages dominate the measured workflow.

Expanded format or architecture support would be additional development, rather than something established by this merge. Future claims should identify their own implementation and measurements. The present results provide a baseline for comparison, not advance confirmation of those extensions.

For now, llama.cpp CUDA fusion offers a specific reason to revisit local inference performance. It targets repeated work inside eligible feed-forward calculations and reports measurable gains under disclosed prompt-processing conditions. That is enough to justify evaluation without promising an across-the-board speed increase.

Before updating a working deployment, preserve its current runtime identifier and a representative set of requests. Compare the same model files, prompt lengths, cache conditions, and output settings after the update. Record initial waiting time separately from answer generation, and retain the benchmark arguments alongside each result. For long-document workflows, include fresh inputs that resemble everyday usage rather than relying entirely on synthetic throughput. For conversational workloads, check whether a smaller input-processing delay changes the experience enough to matter. Then use your own measurements to decide where the upstream improvement belongs in your release schedule. Does the updated runtime shorten the part of the request your users actually wait for?