Kimi Linear Just Beat Full Attention on Its Own Terms

Kimi Linear gave Moonshot AI a result that linear attention researchers had chased for years: better reported quality than full attention across three demanding settings. The company says its architecture led in short-context tests, long-context evaluations, and reinforcement learning under matched training conditions.

The efficiency numbers sharpen that challenge. Moonshot reports up to 75% less key-value cache use and 6.3 times faster decoding at a one-million-token context. Those gains come from replacing most global attention layers with Kimi Delta Attention, while retaining occasional global layers.

That hybrid design puts full attention, specifically Multi-Head Latent Attention, under pressure. Full attention has remained the safer quality choice despite its growing memory and computation costs. Moonshot is arguing that model builders no longer need to accept that tradeoff.

The evidence is substantial but not definitive. The comparisons come from Moonshot's own experiments, using one model family and its internal training system. Open weights, kernels, and serving code make the claims unusually testable, but production behavior still depends on software correctness and hardware conditions.

Kimi Linear Changes the Attention Tradeoff

Moonshot is not presenting Kimi Linear as a smaller compromise. It is presenting the architecture as a stronger replacement for full attention.

The technical report describes a mixture-of-experts model with 48 billion total parameters and 3 billion active parameters. Active parameters are the subset used for each token, which largely determines the computation required during inference.

The model combines Kimi Delta Attention, or KDA, with Multi-Head Latent Attention, or MLA. KDA is a recurrent linear attention mechanism that maintains a fixed-size state instead of storing every previous key and value.

MLA is a compressed form of global attention associated with DeepSeek's model designs. It reduces cache requirements compared with conventional multi-head attention, but its cache still grows with the sequence.

Moonshot uses three KDA layers for every MLA layer. The global layers preserve direct access across the context, while the linear layers handle most token processing through recurrent state updates.

That ratio matters because pure linear attention still faces a memory-capacity problem. A fixed state must compress an entire sequence into a limited representation. Details can collide, fade, or become difficult to retrieve as inputs grow.

Full attention avoids that specific bottleneck by preserving representations for previous tokens. Every new token can compare itself directly with the stored history, although the resulting cache grows with context length.

Kimi Linear does not pretend this finite-state constraint disappeared. Instead, the architecture assigns different work to two mechanisms. KDA supplies efficient local and recurrent processing, while periodic MLA layers restore global access.

Moonshot trained its principal comparison models on 1.4 trillion tokens with the same recipe. According to the report, Kimi Linear outperformed the full-MLA baseline on every listed short-context pretraining evaluation.

Its MMLU-Pro score reached 51.0, compared with 47.2 for full MLA. Kimi Linear also scored 73.8 on MMLU, while MLA reached 71.6.

The pattern extended across HellaSwag, ARC-Challenge, Winogrande, and Big-Bench Hard. Individual margins varied, but Moonshot reported no short-context loss in the listed comparison.

That result reverses the usual argument against linear attention. Historically, teams accepted linear methods because they reduced inference costs, not because they consistently improved conventional benchmark quality.

Research such as linear transformers established the computational case years ago. Reordering attention calculations could make autoregressive processing linear in sequence length and maintain a constant-size recurrent state.

Quality remained the harder problem. Softmax attention could form precise, content-dependent connections across a sequence. Early linear alternatives often lost accuracy, especially when tasks required exact retrieval or complex associations.

Kimi Linear therefore changes more than one benchmark ranking. It provides evidence that a carefully designed hybrid can move efficiency and quality in the same direction.

That evidence creates the article's central tension. If independent tests support Moonshot's results, full attention stops being the automatic default for quality-sensitive long-context models.

Why Long-Context Decoding Puts Full Attention Under Pressure

The decisive advantage appears during generation, when every additional token forces a full-attention model to revisit an expanding cache.

A key-value cache stores the attention representations of previous tokens. It prevents a model from recomputing its entire history during each decoding step.

The cache still grows as the conversation, document, or agent trajectory becomes longer. That growth consumes accelerator memory and reduces the number of requests a server can process together.

MLA already compresses this cache substantially. Moonshot nevertheless reports that Kimi Linear reduces cache use by up to another 75% compared with its full-MLA baseline.

The reason comes from the layer mix. Only one quarter of the attention layers use MLA and require a sequence-length-dependent cache. The three KDA layers maintain recurrent states whose size does not expand with every token.

That distinction becomes important for decoding-heavy workloads. Long-running coding agents, research systems, and tool-using assistants can generate extensive trajectories after receiving large inputs.

A conventional context benchmark often emphasizes prefill, the initial processing of an input. Agent workloads add another burden because generation may continue through many reasoning steps and tool responses.

Moonshot measured time per output token across increasing decoding lengths. At one million tokens, the report lists 1.84 milliseconds for Kimi Linear and 11.48 milliseconds for MLA.

That is the basis for the reported 6.3-times advantage. The gap was smaller at shorter lengths, then widened as the MLA cache placed greater pressure on memory and batching.

The comparison does not mean every deployment will produce six times more tokens each second. Moonshot's result reflects a specific hardware setup, model configuration, batching regime, and implementation.

Throughput also differs from single-request latency. A smaller cache lets a server fit larger batches, which can raise total throughput even when individual computation changes less dramatically.

The report shows a more modest prefill advantage. At a one-million-token input length, Kimi Linear reportedly processed the prompt 2.9 times faster than MLA.

These measurements explain why the architecture targets agents rather than document summarization alone. A system that reads once and answers briefly receives less value from constant-state decoding.

A system that repeatedly reasons, calls tools, receives observations, and continues generating faces a different cost curve. Its history grows during the task, precisely when cache pressure becomes harder to manage.

This pressure reaches model developers, inference platforms, and enterprise buyers. Developers must decide whether global attention everywhere still justifies its serving cost.

Inference platforms need kernels and schedulers that understand recurrent attention state. Enterprise buyers need evidence that lower memory use survives realistic concurrency, retrieval, and reliability requirements.

For knowledge-intensive work, long context also needs organization rather than raw capacity alone. A searchable knowledge base can select relevant evidence before a model processes it.

Kimi Linear does not remove that architectural need. It changes the cost of processing and extending whatever context the application supplies.

The near-term pressure therefore falls most heavily on builders of long-running AI agents. They now have an open implementation that claims better quality alongside a flatter decoding cost curve.

How Kimi Linear Uses a Smarter Recurrent Memory

KDA improves linear attention by controlling what its recurrent memory erases and preserves at a finer level.

Kimi Delta Attention extends Gated DeltaNet, an earlier architecture that combines decay gates with a delta-rule memory update. The Gated DeltaNet paper reported gains over Mamba2 and DeltaNet across retrieval and language tasks.

A delta rule updates memory according to prediction error. In simplified terms, the model tries to replace an outdated key-value association instead of continually adding new information.

That update behavior helps prevent memory from becoming an uncontrolled sum of everything seen previously. It also gives the recurrent state a way to revise earlier associations.

Gated DeltaNet adds a decay mechanism that determines how quickly memory fades. However, one gate can still apply too broadly across a state with many distinct channels.

KDA makes that control more granular. Its finer-grained gates can let different parts of the state forget at different rates, depending on the current token and learned projections.

That matters because useful context does not expire uniformly. A temporary formatting cue may become irrelevant quickly, while a user constraint or function definition should remain available.

The architecture gives the model more control over this allocation. It still has finite recurrent memory, but it can use that capacity more selectively.

Moonshot also designed a chunkwise training algorithm for KDA. Chunking lets the model process groups of tokens in parallel during training instead of applying every recurrent update sequentially.

The report uses a specialized diagonal-plus-low-rank transition structure. This mathematical form makes the chunkwise computation more efficient while preserving the intended delta-rule behavior.

That implementation work is essential. An architecture with attractive complexity can still fail if its kernels introduce enough overhead to erase the theoretical benefit.

Moonshot's prefill measurements indicate that KDA adds negligible latency compared with the Gated DeltaNet hybrid tested in the report. Yet it delivered better benchmark results than that baseline.

The comparison with Gated DeltaNet also reveals why global attention remains in the design. Kimi Linear and the Gated DeltaNet hybrid were close on short-context evaluations, with Kimi generally ahead.

At a 128,000-token context, the Gated DeltaNet hybrid averaged 51.2 across Moonshot's long-context suite. Full MLA averaged 52.2, while Kimi Linear reached 54.5.

The pure efficiency route therefore did not dominate long-context quality. The stronger result came from pairing KDA with periodic MLA and using Moonshot's chosen positional treatment.

On RULER, a synthetic suite that tests retrieval and manipulation across long contexts, Kimi Linear scored 84.3. Full MLA scored 81.3, while the Gated DeltaNet hybrid scored 80.5.

Kimi Linear also led RepoQA with 68.5, compared with 63.0 for both baselines. RepoQA tests question answering over software repositories, making it relevant to coding and code-search systems.

It did not lead every long-context task. MLA scored 36.1 on LongBench v2, compared with 35.0 for Kimi Linear. MLA also led the Frames evaluation by 60.5 to 58.8.

Those exceptions matter. They show that KDA has not eliminated every advantage of full attention, even inside Moonshot's own evaluation.

The aggregate result still favored the hybrid. More importantly, its largest efficiency gains appear where the full-attention baseline becomes most expensive.

This is how Kimi Linear works without becoming a product tutorial. It treats recurrent memory and global token access as complementary resources, then budgets them layer by layer.

That allocation is the mechanism behind the reversal. Moonshot did not make linear attention imitate full attention everywhere. It kept enough global attention to cover recurrent memory's weakest point.

Reinforcement Learning Tests a Different Kind of Memory

Moonshot's reinforcement learning results suggest the architecture can survive long generated trajectories, not merely retrieve facts from large prompts.

Reinforcement learning introduces a different test from pretraining. The model generates answers, receives rewards, and updates toward behaviors that score better under the training objective.

For reasoning models, those outputs can become long. A model may explore calculations, revise intermediate steps, or produce extended tool-use sequences before reaching an answer.

Moonshot compared Kimi Linear with full MLA using reinforcement learning with verifiable rewards. RLVR uses outcomes that can be checked automatically, such as the correctness of a mathematical answer.

The company says both models used the same algorithm and hyperparameters. Moonshot evaluated their progress on mathematical tests including AIME 2025 and MATH 500.

According to the paper, Kimi Linear maintained better performance through the training process. Its advantage appeared in both training accuracy and held-out evaluation curves.

This result matters because linear attention's fixed state could have struggled with extended reasoning. Important earlier steps might disappear as the generated sequence expands.

Moonshot's curves suggest that KDA's memory controls, combined with periodic global layers, preserved enough useful state for the tested reasoning workloads.

The report does not provide an independent replication of that conclusion. It also uses Moonshot's internal mathematics training set, which outside researchers cannot fully inspect.

Matched hyperparameters improve fairness, but they do not settle every comparison. An architecture-specific tuning effort might change the absolute results or narrow the gap.

There is another interpretation to consider. The Kimi Linear architecture might provide an optimization advantage for Moonshot's training system, rather than a universal quality advantage.

That would still be useful, but it would make the claim narrower. Other model sizes, data mixtures, reward designs, and training frameworks might produce different rankings.

The 48-billion-parameter mixture-of-experts configuration also complicates generalization. A 3-billion-active model does not behave exactly like a dense model of the same active size.

Its expert layers hold much more total capacity than the active parameter count suggests. Results could depend on the interaction between sparse experts and the hybrid attention stack.

Developers should therefore separate three questions. Does KDA improve Moonshot's model, does hybrid linear attention improve comparable architectures, and does it improve a specific production workload?

The paper provides encouraging evidence for the first question and a credible research lead for the second. Only workload-level evaluation can answer the third.

For an enterprise agent, benchmark accuracy is not enough. The system must preserve instructions, cite the right evidence, handle tool failures, and remain stable over repeated interactions.

A recurrent-state architecture can also be harder to inspect. Full attention exposes a cache tied directly to prior tokens, while KDA compresses history into learned state matrices.

That compression is its source of efficiency. It also makes failures less intuitive when the model forgets or overwrites something important.

Reinforcement learning therefore strengthens Moonshot's argument without closing the case. It shows the architecture remained competitive under a long-output training regime that directly stresses memory.

Open Code Makes the Claims Testable, and the Software Stack Adds Risk

Kimi Linear's open release lowers the verification barrier, but implementation correctness now becomes part of model correctness.

Moonshot released pretrained and instruction-tuned checkpoints alongside the paper. It also published the KDA kernel and an implementation for the vLLM inference engine.

The official repository describes an OpenAI-compatible vLLM deployment with a maximum model length of 1,048,576 tokens. It documents the same three-to-one KDA and MLA layer ratio.

Researchers can inspect the architecture, run controlled evaluations, and compare memory consumption. Infrastructure teams can profile the kernels rather than relying only on charts in a report.

The model weights also allow targeted failure testing. Evaluators can vary the location of evidence, introduce competing facts, or extend generation after a long prefill.

This openness is particularly important for a new attention mechanism. Small errors in state layout, chunk boundaries, caching, or batching can quietly corrupt long-context behavior.

A later vLLM bug report illustrates that risk. The report identified an apparent recurrent-state layout error affecting KDA chunked prefill in several vLLM versions.

The reporter found that short prompts remained coherent while long-context retrieval failed. A needle test returned nearby filler instead of the requested phrase.

The issue was marked done, but its existence carries a broader lesson. A benchmarked model architecture and a production serving path are not the same artifact.

Full-attention serving has years of optimization, testing, and operational experience behind it. KDA requires newer kernels and state-management logic across multiple hardware backends.

vLLM now exposes a Kimi Linear module, which improves accessibility. Compatibility does not guarantee identical performance or reliability across releases.

Teams evaluating the model should pin exact software versions. They should test long-prefill retrieval, batched decoding, prefix caching, and state behavior across chunk boundaries.

They should also measure actual concurrency. The paper's largest throughput benefit depends partly on using saved memory for larger batches.

If an application serves only one low-volume request, the operational gain may differ. A high-concurrency agent platform could benefit much more from the smaller cache.

Hardware matters too. Specialized kernels can behave differently across NVIDIA architectures or alternative accelerators. Kernel availability may determine whether theoretical efficiency reaches production.

The strongest validation would combine independent quality tests with end-to-end serving measurements. Evaluators should report hardware, precision, batch size, input length, output length, and software commit.

They should also compare against optimized MLA rather than conventional multi-head attention alone. Moonshot chose MLA as its baseline, making the claimed cache reduction more demanding.

The release deserves credit for enabling that work. Many architecture claims arrive without weights or efficient code, leaving outsiders unable to reproduce the central result.

Still, open artifacts do not make the paper's conclusions automatic. They turn a company claim into a falsifiable engineering proposition.

What to Watch After the Kimi Linear Release

The next verdict depends on independent replication, stable serving support, and adoption in larger production models.

The first signal is benchmark replication under matched training conditions. Independent teams need to compare hybrid KDA against full attention using equivalent parameter counts, data, and training budgets.

Inference-only comparisons cannot isolate the architecture's contribution. A model may differ in data quality, expert routing, optimization, or post-training.

The most valuable studies will train multiple architectures from scratch. They should cover short-context language modeling, long-context retrieval, code tasks, and reinforcement learning.

Consistent gains would strengthen Moonshot's central claim. Mixed results would suggest Kimi Linear works best under particular scales or training recipes.

The second signal is serving reliability across vLLM releases and hardware platforms. Long-context tests should become routine regression checks for KDA state handling.

Stable support would make the architecture credible for production agents. Repeated state or kernel failures would weaken the case, regardless of the paper's model-level results.

Teams should watch whether deployment tools support prefix reuse, quantization, speculative decoding, and continuous batching without sacrificing KDA correctness. Those features shape real serving economics.

The third signal is architectural adoption. The strongest endorsement would be KDA appearing in larger Moonshot systems or unrelated model families.

Adoption would show that Moonshot considers the design scalable beyond one research checkpoint. External use would indicate that other teams can reproduce its training and inference advantages.

Larger deployments will also expose the design to harder workloads. Multi-agent tasks, repository-scale coding, research synthesis, and prolonged tool use can stress memory differently from academic suites.

Readers should not reduce the story to a six-times speed claim. That figure represents the most favorable long-context decoding point in Moonshot's reported setup.

The larger development is a credible challenge to full attention's status as the quality-first default. Kimi Linear combines recurrent efficiency with enough global access to avoid the usual compromise.

For developers, the practical question is specific: does the architecture preserve the evidence and instructions your application needs throughout long generation?

For infrastructure teams, the question concerns usable capacity. Does the reduced cache translate into larger safe batches, lower latency, or fewer accelerators under representative traffic?

For enterprise buyers, reliability should come before architecture labels. Ask vendors for long-context retrieval tests, software versions, concurrency assumptions, and failure analysis.

Kimi Linear now gives the industry something concrete to test. Run it against your longest real prompts, extend the output, and verify what survives. If independent results match Moonshot's report, full attention will need a stronger defense than familiarity.