RedKnot’s Long-Context Speedup Matters for AI Workflows, Not Just Benchmarks

RedKnot splits the KV cache along the attention head dimension to raise throughput on long inputs. The change moves past pure benchmark numbers and into daily use cases that need fast handling of full documents and transcripts.

Engineers measured the effect on 8 H800 cards. Time to first token improved between 1.6 times and 3.54 times. Single card concurrency rose 4.7 to 7.8 times. Prefill compute fell 67 to 79.5 percent. On DeepSeek-V4-Flash the same 128 K context produced a 5.16 times TTFT gain and cut KV transfer volume by up to 6.3 times. Accuracy stayed above 95 percent of the dense baseline in most tests.

These numbers matter because many current agents still stall when the input grows past a few tens of thousands of tokens. The stall shows up as long waits before any answer appears.

Head-level split changes cache behavior

RedKnot keeps only the heads it classifies as local. Between 83.4 and 96.8 percent of heads fall into this group on the tested models. The remaining heads stay dense. The method also adds a sparse feed-forward network and SegPagedAttention so that both compute and memory stay aligned to the same head groups.

The three pieces work together. Sparse classification decides which heads to keep. The sparse FFN reduces activation cost. SegPagedAttention then pages the surviving KV blocks without extra copies. The result is a storage layout that matches the compute schedule and avoids the usual cross-device shuffle that slows prefill.

Office agents feel the difference first

Teams that rely on full meeting records or multi-file research hit the old limits every day. A 90-minute call plus attached slides and emails often exceeds 40 K tokens. Under the previous KV layout the first useful sentence could take 30 seconds or more.

With RedKnot the same request returns in roughly one-third the time on the same hardware. The agent can therefore finish the next step, such as extracting decisions or drafting a follow-up document, before the user moves to another task. The saved seconds accumulate across every session that touches recent context.

Long document search becomes usable

Product teams and analysts already run repeated queries across past project files. Each query previously required either truncation or slow full-context evaluation. RedKnot raises the practical limit because the prefill cost drops and the cache stays resident longer on one card.

The same hardware that once supported four concurrent 128 K requests now supports roughly six to eight. Scheduling fewer cards for the same workload frees capacity for other jobs or reduces the monthly cloud bill.

Accuracy stays within reach of dense models

The reported F1 scores stayed above 95 percent of the full dense run on the tasks measured. That margin holds because the pruned heads carry low attention weight for the local patterns that dominate most office text. The method does not claim zero loss on every possible prompt. It simply keeps most of the signal while cutting most of the cost.

Teams that need absolute top-percentile recall on obscure facts still run dense passes for final verification. The faster path serves the first pass and the interactive loop.

Hardware efficiency replaces raw scale

The measured gains came from existing H800 cards without new interconnect or larger memory. The engineering focus moved from adding GPUs to using the same GPUs more completely. That shift matters for groups that already own capacity and want to stretch it rather than order more.

Remaining limits and open questions

RedKnot still requires an initial classification pass to label heads. The cost of that pass is small but adds a fixed overhead on every new model deployment. The classification also depends on the training distribution. A model fine-tuned on code rather than meetings may show different sparsity ratios.

No public head-to-head data yet exists against other sparsity methods released in the same quarter. Until those comparisons appear, the 5.16 times number on DeepSeek-V4-Flash stands alone.

What to watch in the next quarter

Three signals will show whether the approach moves from research release to production default. First, whether the next DeepSeek or similar model ships with RedKnot-style head classification enabled by default. Second, whether an open weights model reproduces the same sparsity ratios on public long-document benchmarks. Third, whether any major inference platform reports production adoption metrics such as average TTFT on 100 K-plus inputs.

Each of these events will appear in the next model card or infrastructure report. Any one of them will confirm that the KV cache change has moved past the benchmark stage and into standard tooling for long document AI workloads.

Users who run agents against accumulated transcripts and files can test the new path today on supported backends. The measured speedups translate directly into shorter waits between question and usable answer.