HYPIC Brings Position-Independent Caching to Hybrid-Attention LLMs, Challenging the Prefix Cache

HYPIC has introduced the first reported position-independent caching system for hybrid-attention LLMs, with average first-token latency falling by 3.25 times in its evaluation. Researchers from Xiaohongshu, Peking University, and Shanghai Jiao Tong University designed the system for long, modular prompts used by RAG applications and AI agents.

The important conflict is not simply cache versus no cache. Hybrid-attention models already reduce long-context costs by replacing most full-attention layers with linear attention. However, existing position-independent caching techniques depend on per-token data that those linear layers do not retain.

That incompatibility forced serving teams toward an awkward choice. They could deploy newer hybrid models with smaller recurrent states, or use flexible caching designed for conventional transformers. HYPIC claims to combine both approaches through state composition, limited boundary recomputation, and parallel processing of uncached segments.

The results are promising but remain research findings rather than broad production proof. The system was tested on four hybrid models and five workloads, including one production RAG trace. Its quality also stayed an average of 1.71 points below full recomputation, leaving deployment teams with a measurable tradeoff.

What the HYPIC Position-Independent Caching System Changes

HYPIC treats independently reusable prompt segments as composable model states, extending flexible caching beyond conventional full-attention transformers.

The authors released the second version of the HYPIC paper on July 12, 2026. Its seven researchers represent Xiaohongshu, Peking University, and Shanghai Jiao Tong University. Yifei Liu and Juntong Wu are listed as equal contributors, while Junhao Hu is the corresponding author.

The system targets retrieval-augmented generation, multi-document analysis, and long-running agents. These applications rarely send a simple user question directly to a model. They assemble prompts from system instructions, retrieved documents, memory files, tool definitions, examples, and execution history.

Many of those segments appear repeatedly across requests. A support assistant might retrieve the same policy document for hundreds of users. A coding agent might load the same repository files while investigating different tasks. An enterprise search system may repeatedly combine familiar documents in new orders.

Standard prefix caching can reuse a processed segment only when its preceding tokens also match. If the same document moves from the second position to the fourth, the changed prefix can invalidate the cache hit. The server then performs the expensive prefill computation again.

Position-independent caching, or PIC, changes that rule. It allows the system to cache a self-contained prompt segment once and reuse it after different prefixes or at different locations. The approach better matches how retrieval systems and agents construct prompts from modular material.

Earlier PIC work focused on full-attention models. These models retain key-value vectors for individual tokens, giving cache systems a direct object to store, move, and selectively recompute. The model can splice cached document representations together and repair the places where their surrounding context changed.

Hybrid-attention models create a different data structure. They replace most full-attention layers with linear-attention layers, which compress earlier tokens into a fixed-size recurrent state. That state can reduce memory use and processing costs, but it removes the per-token handles required by existing PIC methods.

The incompatibility matters because hybrid designs are moving into major model families. The HYPIC researchers identify MiniMax-M1, Ring-2.5, Qwen3.5, and Kimi-Linear as examples of models that mix linear and full attention.

In the researchers' Qwen3.5-35B-A3B example, 30 of 40 layers use linear attention. Each linear layer holds roughly 2 MB of recurrent state per request. A full-attention layer can require roughly 256 MB of KV cache at a 128,000-token context, according to the paper.

That is more than a 100-fold per-layer difference in this configuration. Yet the smaller state also prevents established cache systems from manipulating each segment token by token.

HYPIC changes the serving layer rather than retraining the model. It adds a representation that lets the server combine recurrent states from independently processed prompt segments. It then repairs the remaining full-attention layers near segment boundaries.

The researchers implemented HYPIC on SGLang, an inference framework designed for structured language model workloads. This makes the project a serving-system proposal, not a new foundation model or attention architecture.

The central claim is therefore narrow but important. HYPIC does not make hybrid attention possible. It claims to make modular cache reuse practical when a model already uses a hybrid-attention stack.

Why Prefix Caching Falls Short for RAG and AI Agents

The pressure falls on inference systems that still organize cache reuse around identical prompt beginnings, even as applications assemble context dynamically.

Prefix caching works well when requests share a stable opening sequence. A provider can process a common system prompt once, retain its intermediate state, and reuse that work for later requests containing the same prefix.

This pattern appears in chat applications with fixed instructions. It also works for batch workloads where every request begins with an identical template. Modern serving engines exploit it because the rule is simple and maintains exact causal attention.

RAG and agent workloads are less orderly. Retrieval results can change by query, document order, permission level, freshness, or reranking decision. Agents also insert tool responses, memory records, and execution traces between previously seen segments.

Consider an internal research assistant that has already processed three long reports. A later question may use two of those reports, add a fourth, and place the most relevant report first. The information is familiar, but the full prefix is not.

Prefix caching sees much of that request as new work. A position-independent system sees reusable building blocks.

This difference grows more important as prompt contexts expand. Prefill is the phase where a model processes all input tokens before generating the first output token. When a prompt contains tens of thousands of tokens, prefill can dominate the user's waiting time and the provider's compute cost.

The user experiences this bottleneck through time to first token, or TTFT. Generation may proceed quickly after the first token appears, but a long silent wait can still make an agent feel unresponsive.

HYPIC targets this delay by increasing reuse at the segment level. A retrieved file does not need the same absolute position or preceding documents to qualify for reuse. The system attempts to compose its cached state with the states of other segments.

The approach challenges prefix caching as the default production baseline for hybrid models. The paper reports an average 3.25-fold TTFT reduction over prefix caching across its tested configurations.

Under a one-second TTFT service-level objective, HYPIC reportedly supported 1.66 times as many queries per second. The metric matters because an optimization that helps isolated requests can still fail once a server faces sustained traffic.

The reported gains varied by model. At the same one-second latency objective, HYPIC raised sustainable QPS over prefix caching by 1.85 times on Ring-mini. The corresponding gains were 1.49 times on Ring-flash, 1.58 times on Qwen3.5-35B, and 1.71 times on Qwen3.5-122B.

Peak token throughput per GPU increased by smaller amounts. HYPIC reported gains of 1.50 times on Ring-mini, 1.46 times on Ring-flash, 1.32 times on Qwen3.5-35B, and 1.30 times on Qwen3.5-122B.

Those figures describe more than a faster cache hit. They suggest that broader reuse can change the amount of traffic a fixed GPU deployment sustains within a latency target.

That is especially relevant for enterprise AI systems grounded in recurring information. A company may have millions of possible documents, yet actual workloads often concentrate requests around popular policies, code modules, customer records, or current projects.

The reusable unit is no longer the entire prompt. It is the document, memory entry, tool specification, or other meaningful segment inside that prompt.

Previous work established this direction for full-attention models. The peer-reviewed EPIC system, published at ICML 2025, formalized position-independent caching and reported substantial latency and throughput gains.

However, EPIC and related systems manipulate token-level KV representations. HYPIC addresses the architectural gap that appears when most layers no longer keep those representations.

This places pressure on serving frameworks, model providers, and infrastructure teams. If hybrid models continue spreading, cache systems built only for conventional transformers will cover a shrinking portion of the model stack.

It also changes model selection. Buyers evaluating a long-context model must consider more than benchmark quality, context length, and generation speed. The model's attention architecture can determine which serving optimizations remain available.

How HYPIC Composes Linear-Attention States

HYPIC's core mechanism is a cached transition operator that describes how each segment transforms the recurrent state entering it.

A linear-attention layer processes tokens through a recurrent update. Instead of storing a separate key and value for every earlier token, it maintains a compact state that evolves as each token arrives.

That state depends on the segment's own tokens and the state created by everything before it. This dependency makes naive segment reuse unsafe.

Suppose a system independently processes document A and document B from empty states. Simply adding their final states does not reproduce the state obtained by processing A followed by B. Document B should transform the state left by A, not behave as if no earlier context existed.

The HYPIC paper reports that naive addition loses 66.9 percent of the full-recompute quality score across its evaluation cells. The result shows why a basic merge cannot support reliable hybrid-attention caching.

HYPIC stores two items for each independently processed segment. The first is the segment's final recurrent state when processing starts from zero. The second is a segment-cumulative transition operator.

That operator captures how the segment transforms any state that existed before it. During reuse, the server applies the cached transition to the running state and adds the segment's zero-start contribution.

The authors describe the resulting composition as near-exact and constant time with respect to segment length. A longer cached document does not require the system to replay every token during composition.

This property is central to the performance claim. Position-independent caching would offer limited value if combining a cached segment still required work proportional to every token it contains.

The method also separates one-time work from reuse work. Creating a segment's transition and state adds overhead during its initial prefill. Later requests can reuse those cached objects without repeating the main computation.

In experiments using Qwen3.5-35B-A3B, transition and state construction accounted for about 5.2 to 6.7 percent of segment prefill time across tested segment lengths. This overhead remains relevant when segments rarely repeat.

The economic logic depends on reuse frequency. A segment used once pays the construction cost without receiving a later cache benefit. A frequently retrieved document can amortize that cost across many requests.

This makes cache admission and eviction policy important, even though it is not the paper's headline contribution. A production deployment must predict which segments deserve GPU memory, host memory, or persistent cache space.

The mechanism also assumes the system can divide prompts into semantically independent segments. Documents and memory files are natural candidates. Arbitrary token fragments may not preserve the same useful self-containment.

Hybrid models are not identical either. Their linear layers can use different recurrent update rules. HYPIC formulates its transition mechanism across several linear-attention families, including scalar, diagonal, and dense transition forms.

The authors evaluated two internal Ring variants and two Qwen3.5 models. This model range supports the idea that the method is not tied to one parameter scale.

Still, four models cannot establish universal compatibility. New hybrid architectures may use state updates, attention schedules, or routing systems that require different composition logic.

Related PIC work is also advancing quickly. The 2026 MiniPIC design focuses on position-free KV storage and user-controlled reuse inside vLLM. It reports 49 percent higher prefill throughput than baseline vLLM on a large RAG workload.

MiniPIC addresses conventional KV-based reuse with a small implementation footprint. HYPIC instead targets recurrent states inside hybrid stacks. The projects are therefore better viewed as responses to different cache representations than as direct substitutes.

The broader shift is clear. Cache design is becoming architecture-aware. A single universal caching layer may not capture the best reuse strategy across full attention, linear attention, state-space components, and hybrid combinations.

Boundary Repair Keeps Full Attention From Losing Context

State composition solves the linear layers, but HYPIC still has to repair the full-attention layers where independently cached segments lack cross-segment context.

Hybrid models retain some full-attention layers because global token-level attention remains useful. These layers let later tokens inspect earlier details directly rather than relying entirely on a compressed recurrent state.

Independent segment prefill creates a problem for those layers. When the server processes a document alone, its tokens cannot attend to the documents or instructions that will eventually precede it.

Cached representations therefore differ from those produced by full recomputation of the complete prompt. A PIC system needs a correction step to recover enough cross-segment interaction.

Existing methods can selectively recompute token KV entries inside full-attention models. Hybrid stacks make this harder because intervening linear layers do not preserve every token's intermediate hidden state.

HYPIC's answer is a seam window, a small region of recomputed tokens at the beginning of each segment boundary. The system propagates hidden states for these tokens through the hybrid stack and uses them to restore cross-segment attention.

The design follows an empirical observation in the paper. The largest attention deviations appear near segment beginnings, where tokens suddenly encounter a preceding context absent during isolated prefill.

Rather than recomputing every token, HYPIC concentrates work around those seams. The strategy limits correction costs while allowing the retained full-attention layers to look backward across segment boundaries.

This is also where the system's main quality tradeoff appears. HYPIC approximates full recomputation because most cached token representations remain based on independent segment processing.

Across 16 reported model and workload combinations, HYPIC scored an average of 1.71 points below full recomputation. The average hides meaningful variation between model families.

On Qwen3.5-35B, HYPIC reportedly scored 0.47 points above full recomputation. The paper treats this small advantage as effectively lossless variation rather than evidence of better reasoning.

On Qwen3.5-122B, it trailed full recomputation by 0.56 points. The gaps were larger on the Ring models, reaching 3.44 points for Ring-mini and 3.29 points for Ring-flash.

These differences deserve more attention than a single average. A three-point reduction may be acceptable for one retrieval task and unacceptable for another. Aggregate benchmark scores do not automatically translate into safe production thresholds.

The quality gap can also interact with workload structure. A prompt containing loosely related reference documents may tolerate limited boundary correction. A legal analysis or code-debugging task may rely on precise dependencies across distant segments.

Segment order matters too. Position independence does not mean context order becomes irrelevant to the model. It means cached computation can be reused when a segment appears in a new position, after the system applies composition and correction.

The paper's evaluation covers four public datasets and one production RAG trace. That is stronger than testing only synthetic sequences, but it does not represent every agent workflow.

Long-running agents can produce unusually entangled prompts. Tool output may refer directly to an earlier observation, while a later memory entry changes the interpretation of an earlier file. Clean segment boundaries may become harder to identify.

There is also an operational tuning question. A larger seam window should recover more cross-segment information but requires more recomputation. A smaller window saves time while increasing approximation risk.

Deployment teams would need quality tests tied to their own prompts, models, and failure costs. The paper's settings provide starting points, not a universal seam size.

This limitation does not erase the result. HYPIC's contribution is showing that selective boundary work can preserve most measured quality despite the absence of per-token states in linear layers.

It does mean the latency headline should not stand alone. The relevant comparison is latency, throughput, cache overhead, and task quality together.

Cold Requests Become a Parallel Workload

HYPIC extends caching beyond reuse by parallelizing uncached segments, attacking the tail latency that cache-hit averages can conceal.

Every cache system eventually misses. Documents change, new files arrive, unpopular segments get evicted, and first-time queries retrieve material the server has never processed.

A long cold request can contain several uncached segments. Prefix caching and earlier PIC systems commonly process those segments as one sequential request on a single model instance.

HYPIC observes that PIC has already made each segment self-contained during its initial prefill. That independence allows the server to scatter cold segments across multiple workers.

Each worker processes one or more segments in parallel. A combine worker then assembles the resulting transition operators, recurrent states, and repaired boundary data into the request's running state.

The scheduler uses a longest-processing-time-first policy. It assigns longer segments early to reduce the risk that one slow worker delays the entire request.

HYPIC also pipelines computation with data transfer. This reduces the time the combine worker spends waiting for all segment results to arrive.

On cold-only requests, the paper reports a 5.7-fold TTFT speedup using eight instances. In another reported breakdown, segment parallelism shortened the critical computation path by 3.6 times.

This feature changes the role of a cache miss. A miss still requires full segment processing, but it no longer has to remain entirely sequential when the prompt contains independent components.

The distinction matters for tail latency. Average TTFT can look healthy when popular documents produce frequent cache hits. A small share of long cold requests can still violate service objectives and create poor user experiences.

Parallel cold prefill does require spare capacity across multiple workers. A lightly loaded cluster can scatter one request aggressively. A busy cluster must decide whether reducing one request's delay justifies occupying several instances.

That choice introduces a scheduling tradeoff between individual latency and cluster-wide fairness. A system optimized for interactive agents may prioritize short tail latency. A batch platform may prefer higher aggregate utilization.

Data transfer can also become more significant as cached state grows or network bandwidth tightens. The paper reports that communication and combination remained small in its tested configuration, but larger deployments may have different topology constraints.

Fault handling adds another consideration. A distributed cold request now depends on several workers and a combine stage. Production implementations need timeouts, retries, cancellation, and cleanup when one segment fails.

Cache consistency also matters when documents update. If workers process different versions of a segment or reuse stale states, composition can produce a prompt representation that never corresponded to one coherent source set.

These are engineering questions, not contradictions in the paper's mechanism. They determine whether research speedups survive production realities.

For AI agents, the potential benefit is tangible. An agent often loads instructions, tools, memory, repository files, and retrieved evidence before it can act. Processing independent components concurrently can reduce the waiting period before the first visible response.

Knowledge-intensive applications can also improve their own input pipeline. Teams building a searchable knowledge base already organize documents into reusable units. That structure can align naturally with segment-aware inference.

Still, application developers cannot enable HYPIC through prompt wording alone. The serving infrastructure must recognize boundaries, manage cached transitions, coordinate correction, and schedule distributed cold prefills.

The result therefore pressures inference providers more than end users. If HYPIC's findings hold at scale, providers that expose only conventional prefix caching may leave substantial long-context performance unused.

What the HYPIC Results Still Need to Prove

The next test is not another isolated speed benchmark. HYPIC must show that its quality, cache economics, and distributed scheduling remain dependable under changing production traffic.

The first signal to watch is independent implementation. The paper says the researchers built HYPIC on SGLang, but broad validation requires reproducible code, documented kernels, and results from teams outside the original institutions.

Independent testing should compare identical model versions, hardware, cache budgets, prompts, and service objectives. Small differences in batching, quantization, kernels, or request arrival patterns can materially affect inference results.

The second signal is model coverage. The initial evaluation includes Ring-mini, Ring-flash, Qwen3.5-35B, and Qwen3.5-122B. Those models cover useful scales, but the hybrid-attention category is broader.

Tests should examine other recurrent update rules, mixture-of-experts routing, longer contexts, quantized deployments, and multi-node inference. A reusable serving abstraction must survive architectural variation.

The third signal is workload-specific quality. An average 1.71-point gap is encouraging, but enterprises need failure analysis rather than only aggregate scores.

Researchers should report which questions change answers, how errors relate to segment boundaries, and whether larger seam windows recover them. Evaluations should include code agents, legal retrieval, deep research, and multi-step tool use.

The source paper remains an arXiv preprint, not a peer-reviewed conference publication as of July 16, 2026. Its claims should be treated as reported experimental results until other teams reproduce them.

The “first system” description also requires a narrow reading. HYPIC appears to be the first system specifically designed to combine position-independent caching with hybrid-attention LLM serving. It is not the first PIC system or the first optimization for hybrid models.

Competition will likely come from several directions. Conventional PIC systems can improve their support for mixed architectures. Model designers can expose more reusable intermediate state. Serving engines can also develop native segment interfaces that reduce integration complexity.

Prefix caching will not disappear. It remains exact, comparatively simple, and effective when prompts share stable beginnings. Many chat and batch workloads fit that pattern well.

HYPIC becomes more compelling when requests repeatedly rearrange long, independently meaningful segments. Its value should rise with document reuse, segment length, context diversity, and strict first-token latency targets.

Providers will need to measure those properties rather than assume every long prompt benefits equally. A cache with poor hit rates can consume memory and add management overhead without producing enough saved computation.

The same caution applies to segment parallelism. Eight-worker acceleration looks attractive for an isolated cold request, but production schedulers must balance it against competing traffic.

A meaningful next benchmark should report cluster utilization, tail latency, queueing, and throughput together. It should also include cache churn, document updates, and realistic eviction pressure.

The paper's one-second TTFT service objective offers a useful operational frame. Future reports should show whether HYPIC maintains its QPS advantage across stricter and looser objectives, not only at one threshold.

Security and isolation also deserve examination. Enterprise serving systems may cache segments belonging to different users, teams, or permission domains. Reuse mechanisms must prevent unauthorized state sharing and reliably invalidate cached representations when access changes.

For developers, the immediate lesson is architectural. Long-context performance now depends on how model architecture, prompt assembly, caching, and scheduling interact. Evaluating these layers independently can hide the actual bottleneck.

For enterprise buyers, HYPIC suggests several questions for inference vendors. Does the platform reuse documents outside identical prefixes? Does it support hybrid-attention models without disabling flexible caching? How does it measure quality loss from approximate composition?

Teams managing agent memory should also examine how their systems divide information. A well-structured AI knowledge base can provide stable, reusable segments, while tangled prompt histories make safe reuse harder.

HYPIC's reported 3.25-fold latency reduction and 1.66-fold sustainable QPS gain make it worth watching. The deeper result is the removal of a false choice between efficient hybrid architectures and flexible segment caching.

Whether that result changes production inference will depend on reproducibility, broader model support, and workload-level quality tests. Over the next several months, watch for public code, independent benchmarks, and deployments that report real cache-hit patterns.

If those signals arrive, HYPIC position-independent caching could become an important serving layer for RAG and agent systems. If they do not, the paper will still have identified a central problem that inference platforms must solve.