Xiaohongshu Introduces PIPO Architecture to Compress LLM Inputs

Xiaohongshu released the PIPO architecture, a method that compresses two tokens into one latent on the input side and expands hidden states into extra tokens on the output side.

The change directly targets inference speed. On Qwen3.5-4B and 9B backbones, the system produced up to a 7.15 point lift in pass@4 on AIME 2025 tasks. Deployment tests recorded a 1.23 times improvement in time-to-first-token and a 1.86 times improvement in time-per-output-token.

PIPO differs from earlier length-reduction approaches because it keeps training in two clean stages. Supervised fine-tuning first aligns the model, then on-policy distillation transfers verifier signals into a lightweight confidence head. The result is a model that stays accurate even after aggressive compression.

Deployment gains appear across multiple model sizes

Engineers tested the architecture on both 4B and 9B parameter backbones. The smaller model saw the largest relative speedups, while the 9B model preserved more accuracy under the same compression ratio. Both versions maintained or improved benchmark scores compared with the uncompressed baseline.

The measured TTFT and TPOT numbers came from controlled GPU runs using identical prompt sets. No additional hardware was required beyond standard serving frameworks.

Two-stage training keeps quality stable

The first stage uses ordinary supervised fine-tuning to teach the model basic response patterns. The second stage applies on-policy distillation, in which a stronger verifier model labels the student outputs and the confidence head learns to predict those labels.

This separation prevents the compression layers from learning incorrect shortcuts. The final model therefore shows consistent behavior on math and reasoning tasks where token-level errors would otherwise compound.

Performance tradeoffs remain visible

While average latency improved, peak memory usage during training rose slightly because the latent compressor and multi-token prediction head must run in parallel. Production teams must budget extra GPU memory when fine-tuning from scratch.

The confidence head also introduces a new hyper-parameter that controls how aggressively the model accepts early stopping. Setting the threshold too high reduces the output speedup; setting it too low can drop accuracy on harder questions.

Next signals to watch

Teams will track whether later Qwen releases adopt the same compressor design. They will also monitor open-source replications on other base models to see if the 1.23 times and 1.86 times speedups hold under different training regimes. Finally, any public release of the confidence-head weights would allow independent verification of the distillation claims.