Tencent HPC Lands in SGLang, Challenging Default Inference Kernels

Tencent HPC has entered SGLang’s main branch with three optimized operator paths and a reported 48.8% reduction in Hy3 token latency. The contribution brings Dynamic Attention, Router GEMM, and Fused MoE implementations from Tencent Hunyuan’s HPC-Ops library into a widely used open-source inference engine.

The headline number deserves careful framing. TPOT, or time per output token, measures the average latency between generated tokens after the first one appears. Tencent and SGLang report improvements of up to 48.8% under tested Hy3 configurations, rather than a universal gain across models, GPUs, and traffic patterns.

Still, this is more than another collection of isolated CUDA benchmarks. SGLang users can access the kernels through an upstream framework instead of maintaining a private fork. That puts Tencent’s operator work into direct competition with established SGLang, FlashInfer, CUTLASS, Triton, and vendor-backed inference paths.

Tencent HPC Moves From a Kernel Library Into SGLang

The important change is distribution, not simply another fast benchmark.

HPC-Ops is an open-source inference operator library developed by Tencent’s Hunyuan AI Infrastructure team. Its operator repository covers attention, GEMM, Mixture-of-Experts computation, sampling, normalization, and fused communication operations.

An operator is a low-level computational building block that performs a specific model task on the GPU. Frameworks such as SGLang assemble these blocks into the execution path used to serve model requests.

Before upstream integration, teams interested in HPC-Ops needed to install and connect its components within their own serving environment. That work involved compatibility testing, backend selection, and ongoing adaptation when either project changed.

Integration into SGLang’s main branch changes that relationship. The framework can recognize and dispatch supported Hy3 operations to the Tencent kernels through maintained backend paths. Users no longer need a long-lived SGLang fork merely to evaluate the implementation.

The initial contribution centers on three latency-sensitive areas:

• Dynamic Attention distributes uneven decoding work across GPU execution units.

• Router GEMM accelerates a precision-sensitive matrix multiplication used to select MoE experts.

• Fused MoE combines several expert-processing stages to reduce launches and memory traffic.

Each targets a different source of inference delay. Together, they address the irregular workloads created by long contexts and sparse models.

Hy3 is a useful test case because it combines both problems. Tencent describes Hy3 as a sparse Mixture-of-Experts model designed for agentic execution, coding, and extended reasoning. Its architecture activates only part of the model for each token, but it still requires attention over growing context and routing across many experts.

That combination creates latency outside the large matrix multiplications that usually dominate simplified benchmarks. Scheduling, data movement, precision conversion, routing, and small kernel launches can become equally important during online serving.

The reported end-to-end result is a TPOT reduction of up to 48.8% on tested Hy3 workloads. This number comes from the project authors’ benchmark environment and has not been independently reproduced across a broad hardware set.

The distinction matters because “up to” captures the strongest measured case. It does not describe the average improvement for every request pattern. A server with short, uniform prompts may see a different result from one handling mixed-length agent sessions.

HPC-Ops also remains hardware-specific. Its published requirements identify NVIDIA’s SM90 architecture, Python 3.8 or later, and CUDA 12.8 or later. The library says its kernels are particularly tuned for NVIDIA H20 GPUs.

That limits immediate portability, but it does not erase the significance of upstream availability. SGLang has become a common integration point for model builders, infrastructure teams, and kernel developers. Landing there exposes Tencent’s work to more realistic testing than a standalone repository usually receives.

It also follows a similar integration into vLLM. In July, HPC-Ops attention and MoE backends entered vLLM’s main branch as first-class options. The accompanying vLLM benchmarks reported lower first-token and output-token latency on eight H20 GPUs.

SGLang therefore becomes the second major serving ecosystem where Tencent can test whether its production-oriented kernels travel beyond Tencent’s own infrastructure. That is the real event behind the benchmark.

Why Tencent HPC Matters for Online Model Serving

Modern inference performance depends on managing irregular work, not only maximizing raw matrix throughput.

Offline benchmarks often use fixed prompt lengths, predictable batches, and stable tensor shapes. Production traffic behaves differently. Requests arrive continuously, contexts grow at different rates, and users stop generation at unpredictable points.

One request may carry a 16,000-token context while another has just begun with 1,000 tokens. A static attention schedule can assign much more work to one GPU block than another. The shorter task finishes early, while the longest task determines when the entire launch completes.

Dynamic Attention tries to reduce that imbalance. HPC-Ops divides key-value cache work into smaller tiles and assigns those tiles according to the current workload. The mapping is rebuilt as request lengths change during decoding.

A key-value cache stores previous attention states so the model does not recompute its entire history for each new token. Longer histories require more cache data to be read and processed.

HPC-Ops describes a scheduler that splits requests into uniform tiles and distributes them across cooperative thread arrays. A cooperative thread array, or CTA, is a group of GPU threads that executes a scheduled work unit.

The library reports Dynamic Attention performance up to 2.88 times its static split-k baseline across selected variable-length tests. Its broader attention results include gains against FlashInfer, FlashAttention, and TensorRT-LLM under specified configurations.

Those operator-level numbers should not be confused with end-to-end server improvement. A faster attention kernel affects only the share of total latency spent in that operation. Request scheduling, communication, model loading, sampling, and other layers remain in the path.

Yet attention becomes increasingly consequential as contexts grow. Agentic applications repeatedly append tool results, retrieved documents, and intermediate reasoning to an active session. That creates exactly the uneven cache lengths targeted by dynamic scheduling.

A coding assistant provides a concrete example. One request may contain a small function and a short instruction. Another may include a repository map, several files, build logs, and the full conversation history.

Placing both requests in one continuous batch improves utilization, but it creates uneven attention work. A static split policy either leaves some GPU units idle or launches empty chunks for shorter requests.

Dynamic scheduling attempts to make those requests coexist more efficiently. The largest benefit should appear when sequence lengths differ substantially and the server has enough concurrent work to rebalance.

This is why TPOT matters more than a single throughput number. Throughput measures aggregate output across all requests. TPOT more directly reflects how quickly an individual user sees successive tokens during generation.

For interactive agents, slow token delivery compounds across long responses and repeated tool calls. A lower TPOT can shorten each generation segment, which then allows the next tool or model step to begin sooner.

The integration also pressures default kernel selection inside inference frameworks. SGLang already offers multiple attention and MoE implementations, each optimized for different models, precisions, and GPUs. A new backend must outperform those options without creating fragile configuration rules.

That competition benefits operators only when framework dispatch remains understandable. An infrastructure team needs to know when HPC-Ops activates, which shapes it supports, and what fallback handles an unsupported request.

It also needs evidence from its own traffic. Public benchmarks can identify a promising backend, but they cannot reproduce every combination of sequence length, batch size, quantization, parallelism, and service-level objective.

Teams evaluating the contribution should measure median and tail TPOT, time to first token, request throughput, GPU memory use, and output consistency. Average latency alone can hide stalls that affect the slowest users.

The same discipline applies to technical knowledge surrounding an evaluation. Engineering teams can preserve benchmark commands, deployment notes, and failure analysis in a searchable knowledge base rather than relying on scattered chat threads.

Tencent HPC matters because it gives those teams another maintained execution path to test. Its value will come from repeatable serving improvements, not the maximum figure on a launch chart.

Dynamic Attention and Fused MoE Attack Different Bottlenecks

The reported Hy3 gain comes from removing several small delays that accumulate during every generated token.

Dynamic Attention targets load imbalance during attention. Fused MoE targets fragmentation after the model routes tokens to selected experts. Router GEMM sits between them and accelerates the decision that assigns those experts.

MoE, or Mixture-of-Experts, replaces some dense neural layers with a larger collection of specialized feed-forward networks. A router selects a small subset of experts for each token, limiting the computation performed per token.

That sparsity reduces active computation, but it introduces irregularity. Different experts can receive different token counts. The server must calculate routing scores, organize tokens, execute several smaller matrix multiplications, and combine weighted outputs.

Conventional implementations often separate those actions into multiple kernels. They gather tokens into expert-specific buffers, launch matrix operations, apply activations, quantize intermediate values, and reduce the selected outputs.

Every separation can produce another launch and another trip through high-bandwidth memory. These costs are small in isolation but significant during low-batch decoding, where each expert operation handles relatively few tokens.

HPC-Ops uses a fused FP8 MoE path. FP8 is an eight-bit floating-point format that reduces memory traffic and increases available tensor-core throughput compared with wider formats.

The fused path combines routing-related preprocessing, the gate and expansion matrix multiplication, activation quantization, the projection back to model width, and weighted reduction. Tencent says the implementation reads original tokens through routing indices instead of first copying them into gathered expert buffers.

That design aims to remove both memory movement and kernel launch gaps. It also changes how the implementation hides latency.

Rather than relying heavily on software pipelining inside one GPU block, HPC-Ops increases the number of resident blocks for low-latency shapes. Hardware scheduling can then switch among blocks while other work waits for data.

The library reports its Fused MoE operator as up to 1.6 times faster under tensor parallelism and 1.5 times faster under expert parallelism. Its published comparisons include SGLang, vLLM Triton, and vLLM CUTLASS implementations.

Tensor parallelism divides model calculations across GPUs. Expert parallelism places different MoE experts on different workers, requiring tokens to travel to the workers holding their selected experts.

The best choice depends on model shape, interconnect speed, request concurrency, and memory limits. A fused local kernel cannot eliminate communication required by a distributed expert layout.

Router GEMM addresses a subtler constraint. GEMM means general matrix multiplication, the core operation behind many neural network layers.

A router multiplies lower-precision activations by weights whose small numerical differences can affect expert selection. Reducing those weights too aggressively can change the routing decision and therefore alter model output.

Using full FP32 arithmetic protects precision, but current tensor cores do not provide the same FP32 throughput available for lower-precision formats. A straightforward implementation may fall back to slower CUDA-core computation.

HPC-Ops decomposes each FP32 weight into two BF16 components. BF16 preserves FP32’s exponent range while using fewer mantissa bits.

One component carries the high portion of the weight. The second carries a scaled residual, with the project documenting a scale of 1/256. The kernel then performs two BF16 tensor-core multiplications and combines the results.

The two calculations remain inside one kernel. They share input movement, retain intermediate accumulators in registers, and write the final result once.

Tencent reports performance up to 3.22 times cuBLAS FP32 or TF32 baselines for selected router and state-compression shapes. The claim represents tested operator shapes, not all GEMM workloads.

The mechanism matters because it tries to preserve routing behavior while capturing lower-precision hardware throughput. Speed without comparable expert selection would be a poor trade for production inference.

This precision question is also why output-validation details matter. A benchmark should compare routing decisions or final model outputs, not only execution time.

The earlier vLLM integration reported matched output quality for its MoE comparison. The SGLang integration now creates another opportunity for maintainers and users to test numerical behavior across framework implementations.

Together, the three operators form a coherent execution story. Dynamic Attention distributes context work. Router GEMM decides where sparse computation goes. Fused MoE performs that computation with fewer boundaries.

The improvement is therefore a mechanism story rather than a claim that one monolithic kernel became almost twice as fast. Several bottlenecks were reduced, and their combined value depends on how much each contributes to a particular workload.

What the 48.8% TPOT Claim Does Not Establish

The strongest benchmark is a credible lead for testing, but it is not a portable performance guarantee.

The reported reduction applies to Tencent’s Hy3 evaluation and supported hardware conditions. Neither the number nor the current documentation establishes the same result for every SGLang model.

Hy3 combines model-specific attention behavior, sparse routing, FP8 execution, and a particular expert structure. A dense model without MoE layers cannot benefit from the Fused MoE or Router GEMM paths.

A model using different rotary embeddings, normalization order, head dimensions, quantization scales, or cache layouts may require integration work. It may also activate only part of HPC-Ops.

Hardware creates another boundary. The HPC-Ops requirements currently center on NVIDIA SM90 GPUs and CUDA 12.8 or later. The strongest published results focus on H20.

That leaves NVIDIA Ampere systems, Blackwell configurations, AMD accelerators, and other hardware outside the demonstrated scope. SGLang supports a broader range of environments than this backend presently targets.

Even on H20, traffic shape can change the outcome. Dynamic Attention offers the clearest advantage when a batch contains uneven sequence lengths. Uniform short requests give the scheduler less imbalance to correct.

Fused MoE also depends on batch size. The vLLM team reported its largest gains in small and medium batches, where conventional launches and memory passes consume a larger share of total time.

At high concurrency, larger matrix operations may use the GPU efficiently enough to narrow the difference. Communication may also dominate when expert parallelism spans several devices or nodes.

The 48.8% figure therefore needs context from the full benchmark matrix. Readers should look for prompt lengths, output lengths, concurrency, tensor parallel size, expert parallel size, cache precision, GPU clocks, and baseline configuration.

Baseline quality is especially important. A default backend can be a reasonable comparison for new users, but it may not represent the best tuned alternative.

SGLang’s modular expert-parallel architecture permits custom kernels and specialized backends. FlashInfer, DeepGEMM, Triton, CUTLASS, and native SGLang kernels continue to evolve, so comparative results can shift quickly.

The integration date also matters. Main-branch code receives rapid changes before a stable release reaches broader deployments. A merged backend may still need packaging, documentation, compatibility testing, and operational safeguards.

Numerical accuracy remains another area for independent verification. Router GEMM approximates FP32-weight computation using two BF16 components. That design seeks FP32-level accuracy, but downstream users should test model-level behavior.

Useful checks include output agreement under deterministic decoding, routing-index overlap, perplexity on representative data, and task-level evaluation. A small numerical difference may be harmless, or it may redirect tokens to different experts.

Production reliability extends beyond numerical matching. Kernels must handle boundary shapes, memory pressure, cancellation, continuous batching, prefix caching, and changing sequence lengths without rare failures.

Open-source review helps identify these problems, but visibility is not the same as maturity. The SGLang releases show how frequently model support, kernels, quantization paths, and scheduling behavior change.

Tencent says HPC-Ops already supports large-scale inference within its own environment. That experience is meaningful, yet external deployments often expose combinations that an internal model fleet never uses.

There is also a maintenance question. Upstream code reduces the burden of carrying a fork, but the external HPC-Ops package and its compatibility matrix still need active ownership.

Users should watch how quickly Tencent and SGLang respond to regression reports. They should also check whether continuous integration covers the supported GPU, precision, and model combinations.

None of these limitations invalidates the result. They define what the result actually says.

Tencent and SGLang have presented a strong model-specific benchmark for an upstream backend on supported Hopper hardware. Broader claims require broader reproduction.

The right response is neither automatic adoption nor dismissal. It is a controlled test against the best existing backend under the organization’s own service-level objectives.

Three Signals Will Decide Whether Tencent HPC Changes the Default

The next stage is about adoption, reproducibility, and expansion beyond one model and one GPU family.

The first signal is independent Hy3 benchmarking inside SGLang. External operators should reproduce the reported TPOT improvement with published commands and complete configuration details.

A useful reproduction would include median and P99 TPOT, time to first token, output throughput, memory use, and request failures. It should test both mixed-length and uniform workloads.

Confirmation near the reported range would strengthen Tencent’s claim that combined operator optimization materially changes user-visible decoding latency. Much smaller gains would suggest the maximum result depends heavily on one workload shape.

The second signal is support beyond Hy3 and H20. HPC-Ops already publishes benchmark coverage for DeepSeek-V3, Hunyuan models, and Qwen3-235B at the operator level.

Framework-level integration is harder. Each model can differ in attention layout, normalization order, quantization, expert routing, and distributed execution.

Support for another widely deployed MoE model would show that the backend provides reusable infrastructure rather than a narrowly fitted Hy3 path. Blackwell support would also test whether the project can adapt beyond its original Hopper target.

The project’s roadmap lists broader quantization, future hardware, megakernels, and low-precision communication. A megakernel combines several consecutive operations to reduce launch overhead and intermediate memory traffic.

If these items arrive with SGLang integration and public benchmarks, Tencent HPC becomes a broader competitor among inference backends. If progress remains confined to a few H20 configurations, it stays valuable but specialized.

The third signal is whether SGLang users select the backend in real deployments. Repository stars and isolated microbenchmarks offer limited evidence of operational adoption.

More useful signs include deployment recipes, framework documentation, bug reports from external fleets, compatibility tests, and sustained maintainer activity. Stable release inclusion matters more than main-branch presence alone.

Adoption would pressure competing kernels to improve mixed-length scheduling, router precision, and MoE fusion. SGLang maintainers would then face a productive challenge: selecting the fastest backend without creating confusing or fragile defaults.

Failure reports would be equally informative. They would reveal unsupported shapes, numerical edge cases, packaging friction, or operational constraints hidden by controlled benchmarks.

For developers, the immediate question is practical. Does the new backend lower latency for the exact model, GPU, and traffic pattern they operate?

For infrastructure buyers, the question is economic without requiring a public price comparison. Lower TPOT can increase usable capacity and improve interactive response speed, but only if reliability and output quality remain stable.

For AI product teams, the effect can reach beyond a benchmark dashboard. Faster token delivery shortens coding sessions, agent loops, document analysis, and other workflows that perform several model calls per task.

Tencent HPC has earned serious evaluation because its operators now sit inside SGLang’s upstream path. The integration removes one barrier between an optimized research result and a production test.

The 48.8% figure should start that test, not end it. Teams running Hy3 on supported Hopper hardware can benchmark the backend against their current SGLang configuration now.

Everyone else should watch the three signals: independent reproduction, broader model and hardware support, and evidence of sustained deployment. Together, they will determine whether Tencent HPC becomes a common inference option or remains a specialized Hy3 advantage.