Xiaohongshu and Peking University Release UltraEP, Challenging Static MoE Load Balancing

Xiaohongshu and Peking University have released UltraEP research that challenges a basic assumption behind large-scale mixture-of-experts infrastructure. Instead of predicting which experts will become busy, UltraEP MoE load balancing reacts after each layer’s router reveals the actual demand.

That distinction matters because expert demand can shift between microbatches, layers, and prompt domains. A placement plan based on recent history can become stale before the system uses it. UltraEP responds by creating temporary replicas of overloaded experts and rerouting tokens within the current operation.

The authors report that UltraEP reached 94.6% of an artificially balanced training ideal across three large MoE models. They also report 42% higher average training throughput than Megatron-LM and 1.56 times SGLang’s prefill throughput.

Those results come from the team’s own experiments, not independent replication. The UltraEP paper was submitted in June 2026 and lists researchers from Peking University, Xiaohongshu, Shanghai AI Laboratory, and independent contributors.

The paper provides substantial implementation and evaluation detail. However, it does not identify a public UltraEP source repository. Describing the work as fully open-source therefore requires caution until the runtime code and its license become publicly verifiable.

The more important story is architectural. UltraEP moves load balancing from a periodic control task into the execution path of every MoE layer. That puts established history-based methods, including EPLB, under direct pressure.

UltraEP MoE Load Balancing Reacts After Routing

UltraEP changes when a distributed MoE system decides how to balance work.

A mixture-of-experts model contains many specialized feed-forward networks called experts. Its router assigns each token to a small subset of those experts.

The design reduces active computation relative to using every parameter for every token. It also creates a distributed systems problem. Popular experts can receive far more tokens than their neighbors during the same layer.

Expert parallelism distributes those experts across GPUs. Tokens travel to the devices hosting their selected experts, usually through all-to-all communication, and return after computation.

A GPU holding a popular expert must process more tokens. Other GPUs finish early and wait at synchronization points. The slowest rank can then set the pace for the entire operation.

That imbalance also affects communication and memory. The overloaded rank receives more token activations, transfers more data, and needs more temporary memory. One routing decision can therefore create three related bottlenecks.

Existing balancers often use recent measurements to decide where redundant expert copies should live. DeepSeek’s EPLB project, for example, calculates balanced expert placement from recorded loads.

This approach works when expert popularity changes gradually. It becomes less reliable when prompt composition, token content, or training dynamics alter demand faster than the rebalancing interval.

The UltraEP researchers measured that problem at large expert-parallel sizes. They report substantial variation between layers, workloads, and adjacent microbatches. DeepSeek-V3 training showed especially visible short-term fluctuations.

UltraEP waits until the gating operation has produced exact routing assignments. It then calculates a balancing plan for that specific layer and microbatch.

The runtime can materialize additional physical copies of hot experts in reserved GPU memory. It also divides the selected tokens among the original expert and its temporary replicas.

This happens without changing the model’s logical routing decision. A token still uses the expert selected by the model. UltraEP changes only which physical copy performs the computation.

That separation is important for training semantics. The authors say replica gradients are aggregated during the backward pass, while the primary parameters remain managed by the training framework.

UltraEP also excludes temporary replicas from checkpoints and optimizer state. They are execution resources rather than new learned experts.

The event is therefore more than another faster kernel. It proposes that expert placement can become an immediate response to real routing, provided the hardware moves weights quickly enough.

Why Large MoE Systems Make Stale Plans Expensive

The pressure grows as models use more experts across larger groups of GPUs.

Fine-grained MoE models can contain hundreds of smaller experts. Large expert-parallel groups may spread those experts across 32 or 64 ranks.

This arrangement gives every rank only a small share of the total expert pool. A shift in one expert’s demand then maps more directly onto the workload of one device.

With fewer experts per rank, there are also fewer unrelated workloads to average out the skew. The system becomes more sensitive to individual hotspots.

NVIDIA’s Megatron Core guide documents several router-balancing options. These include auxiliary losses, sequence-level losses, global losses, and dynamic expert biases.

Such techniques influence the model’s routing distribution. They do not eliminate moment-to-moment variation after the router encounters real tokens.

Training workloads change as the model learns. Sampling introduces additional jitter, and different layers can develop different routing patterns.

Inference prefill is even less predictable. Prefill processes the input prompt before token-by-token generation begins. Prompt length, subject matter, arrival rate, and batch composition can all change together.

A coding batch may concentrate demand differently from a mathematics or long-context batch. Combining unrelated requests can create another pattern seconds later.

UltraEP’s authors argue that history-based expert placement cannot follow these changes closely enough. Their experiments configured EPLB to rebalance every three global training batches and every 50 prefill steps.

The paper reports that a stale placement sometimes made rank imbalance worse. A formerly popular expert could retain replicas after demand moved elsewhere, while a new hotspot remained concentrated.

This is the primary contest behind UltraEP: exact real-time balancing against periodic prediction from historical load.

It is not primarily Xiaohongshu against NVIDIA or SGLang as companies. Megatron-LM and SGLang are integration frameworks and experimental baselines. Both can potentially host new balancing systems.

The pressure instead falls on infrastructure designs that treat expert placement as a slow control-plane decision. UltraEP pulls that decision onto the hot path.

The paper evaluates GLM-4.5-106B training on 128 GPUs. Qwen3-235B and DeepSeek-V3 training used 256 GPUs across four racks.

Qwen3-235B-A22B provides a useful example of the scale involved. The official Qwen repository describes it as a 235-billion-parameter model with 22 billion active parameters per token.

For production testing, the researchers used an internal RefMoE-288B-A16B model under 32-way expert parallelism. They report that UltraEP sustained more than 92% of ideal throughput during that run.

The team also reports a 9.6% average improvement over periods sampled without balancing. Its training-loss curve followed the expected trajectory, according to the paper.

That production test is valuable because it extends beyond short benchmark windows. However, the paper does not disclose the complete workload, cluster economics, or operational failure history.

The result therefore supports production feasibility within the authors’ environment. It does not establish that every large MoE cluster will receive a similar gain.

The Mechanism Depends on Fast Expert Replication

UltraEP works by solving placement and token rerouting together, then hiding most replica movement behind computation.

The runtime starts with the exact load matrix produced after routing. That matrix records how many tokens from each source rank selected each logical expert.

UltraEP then searches for a target workload threshold. The objective is to keep every GPU rank below that threshold while respecting available replica slots.

A quota-driven planner decides which hot experts need copies, where those copies should run, and how many tokens each instance should accept.

This joint decision differs from choosing expert replicas first and distributing tokens later. A replica is created only when the plan assigns useful work to it.

The planner also favors locality. Tokens can consume capacity on a nearby or local expert instance before the system sends residual demand elsewhere.

After aggregate quotas are set, the runtime converts them into per-token destinations. The paper describes this step as a localized lookup over cumulative quotas.

The planner runs entirely on the GPU. That avoids CPU synchronization and host-device metadata transfers inside every layer.

This choice addresses one obvious concern about real-time balancing. An accurate plan offers little value if calculating it stalls the model longer than the imbalance would.

Planning is only half the problem. The target GPU needs the expert’s current weights before it can process reassigned tokens.

Training makes this harder because weights change after optimizer updates. A runtime cannot assume that a replica copied many batches earlier remains current.

UltraEP reserves redundant expert slots on each rank. During a layer, it streams the required expert state into those slots using device-side transfer tasks.

The paper calls this persistent tile streaming. Expert tensors are divided into smaller tiles so transfers can use the rack fabric continuously and overlap with other work.

A single source rank can become a communication hotspot when many GPUs request the same expert. UltraEP addresses this with relay-based fan-out.

The original holder sends chunks to selected relay ranks. Those ranks forward the chunks to other destinations while the transfer continues.

This structure spreads outgoing traffic across devices with available bandwidth. It resembles a streaming distribution tree rather than repeated point-to-point copies from one source.

The researchers report that their communication design replicated expert state 3.1 to 5.5 times faster than the evaluated mainstream communication backends.

UltraEP integrates with DeepSeek’s DeepEP library for token dispatch and combination. DeepEP handles the movement of routed token data, while UltraEP manages dynamic expert replication and balancing.

The standalone UltraEP runtime reportedly contains about 9,600 lines of C++ and Python, including device kernels. Each Megatron-LM and SGLang integration required fewer than 1,000 additional lines.

That modularity matters for adoption. A balancer tied to one model framework would face a much narrower path into production.

The design also preserves framework-managed states for primary experts. Temporary replicas use shared internal buffers, avoiding duplicate optimizer state and permanent checkpoint growth.

These choices explain how UltraEP can rebalance at layer granularity. They also expose its central dependency: the system assumes unusually fast communication inside a rack-scale node.

The Benchmark Gains Come With a Hardware Boundary

UltraEP’s strongest results apply to rack-scale GPU fabrics, not ordinary multi-node clusters.

A rack-scale node extends high-bandwidth, scale-up connectivity beyond one server. The paper’s test environment placed 64 GPUs across 16 servers in each rack.

The authors report that the rack’s scale-up links offered eight to ten times the bandwidth of its scale-out RDMA network.

UltraEP keeps each expert-parallel group inside that faster domain. Data parallelism or pipeline parallelism handles expansion across racks.

This topology gives the runtime enough bandwidth to move expert weights after routing without exposing the full transfer delay.

A conventional Ethernet or lower-bandwidth RDMA cluster may not provide that condition. Copying expert state could cost more than waiting for an overloaded rank.

The paper does not claim universal performance across commodity clusters. Its title and system design explicitly target rack-scale nodes.

That limitation does not invalidate the approach. It defines the market where the approach makes sense.

NVIDIA, AMD, and cloud providers are building denser rack-scale systems with fast accelerator fabrics. UltraEP treats that hardware as a programmable resource for dynamic model execution.

The reported training comparison used Megatron-LM as the no-balancing baseline. Across GLM-4.5-106B, Qwen3-235B, and DeepSeek-V3, UltraEP improved average throughput by 42%.

Other evaluated balancing variants improved throughput by smaller amounts. The paper attributes their weaker results to delayed layouts, limited replica budgets, or less effective rerouting.

UltraEP reached an average 94.6% of the force-balanced training ideal. That ideal modified the router to distribute tokens evenly, so it represents an artificial upper bound rather than deployable model behavior.

For serving prefill, UltraEP reached an average 93.9% of ideal throughput. Individual tests reportedly fell between 90% and 97%.

The researchers replayed routing traces captured from SGLang to keep load conditions consistent across balancing algorithms. They report 1.56 times SGLang’s throughput and 1.29 times EPLB’s throughput.

SGLang already supports several expert-parallel communication and compute backends. Its expert parallelism documentation shows how deployment results depend on hardware, quantization, and backend selection.

UltraEP’s SGLang comparison used version 0.5.9 plus a specified commit. Software in this area evolves quickly, so later framework releases may narrow or alter the measured gap.

The evaluation also focuses on training and serving prefill. It does not present UltraEP as a general solution for autoregressive decode.

Decode processes a smaller number of new tokens per step and often becomes memory-bound. Moving full expert weights during those shorter operations presents a different cost equation.

The benchmark methodology carries another constraint. The three public-model training comparisons resumed late-stage checkpoints and ran for 20 additional global batches.

The researchers selected this window to cover multiple balancing intervals. It does not replace full independent training runs for every baseline.

The authors did conduct longer internal production training and reported preserved convergence. Yet outside researchers cannot reproduce the private model, corpus, or exact cloud environment.

Memory overhead also deserves attention. UltraEP reserves slots for redundant experts, although the paper reports reduced peak token-activation memory after balancing.

Operators must still budget capacity for temporary weights and communication buffers. The appropriate reserve will depend on expert size, precision, workload skew, and topology.

Finally, public availability remains unclear. The paper discloses implementation size and integration details but provides no visible UltraEP repository link.

A public paper is not the same as open-source software. Until code, build instructions, and a license appear, external teams can study the mechanism but cannot directly audit the implementation.

UltraEP’s Results Shift the MoE Infrastructure Debate

The paper suggests that future MoE efficiency gains will come from reacting to workload state, not only improving static kernels.

Distributed MoE optimization has often concentrated on three areas. Teams improve token dispatch, fuse grouped matrix operations, and adjust router behavior during training.

UltraEP adds another layer. It changes the physical execution layout after seeing the router’s actual decision.

This design does not replace communication libraries or fast matrix kernels. It sits above them and decides where each expert computation should occur.

The distinction creates room for cumulative gains. A deployment might combine better routing objectives, optimized all-to-all dispatch, faster expert kernels, and real-time physical replication.

History-based placement will not disappear. It remains cheaper when demand changes slowly or the hardware cannot move expert weights quickly.

Prediction can also prepare the system before exact demand arrives. A hybrid approach might use history for baseline placement and exact load for limited corrections.

UltraEP makes that comparison measurable. Future systems will need to show whether predictive placement remains competitive once exact-load correction becomes practical.

The work also changes how infrastructure teams should interpret utilization. Low average GPU utilization may hide a synchronization problem rather than weak kernels.

If one rank is overloaded while others wait, improving the average kernel speed may leave the critical path largely unchanged. Balancing the slowest rank can produce a larger system-level gain.

That lesson reaches beyond foundation-model training. Large prefill workloads appear in classification, document analysis, recommendation, verification, and batch reasoning services.

A company processing varied enterprise documents could encounter rapidly changing expert demand. Legal text, source code, financial reports, and support conversations may activate different expert combinations.

Such workloads also arrive in uneven batches. A queue can shift from short customer messages to long technical documents without warning.

UltraEP’s reported advantage grows under this turbulence. That makes the system relevant to operators serving heterogeneous prompts, not only teams pretraining frontier models.

Still, the economic case depends on utilization and hardware cost. A percentage throughput gain does not automatically justify a rack-scale migration.

Teams need to compare extra reserved memory, fabric requirements, engineering complexity, and reliability against the value of higher throughput.

They also need operational controls. Dynamic expert replication adds new failure modes around buffer management, synchronization, gradient aggregation, and transfer scheduling.

The paper presents a carefully co-designed runtime, not a configuration flag that any cluster can enable safely.

For Megatron-LM, SGLang, vLLM, and DeepEP contributors, UltraEP creates a concrete implementation target. Framework maintainers can test whether similar planning belongs inside existing runtimes.

For accelerator vendors, the paper highlights the value of peer-memory access and device-initiated communication. Faster links alone are insufficient if software cannot schedule irregular expert transfers efficiently.

For model designers, the results reduce one systems penalty associated with specialized routing. Better execution balancing may allow routing to follow token needs without imposing strict artificial uniformity.

That point is subtle. Router-side balancing losses can push token assignments toward equal expert utilization, but excessive pressure may interfere with expert specialization.

UltraEP attempts to preserve the model’s logical choices and balance only their physical execution. If replicated independently, that separation could become its most important contribution.

What to Watch After the UltraEP Release

Three signals will determine whether UltraEP becomes an infrastructure pattern or remains a compelling research prototype.

The first signal is public code. A verifiable repository should include the runtime, supported hardware, build process, framework patches, tests, and license.

Code access would allow outside teams to inspect the GPU planner and replication pipeline. It would also clarify whether the term “open-source UltraEP” describes software or only publicly released research.

Independent reproduction on Qwen3-235B would strengthen the reported case. Comparable results should disclose GPU type, fabric topology, precision, batch settings, and framework commits.

If independent teams approach the paper’s 94.6% training result, exact-load balancing will gain credibility as a repeatable technique. Large gaps would point toward environment-specific advantages.

The second signal is support beyond one rack architecture. Tests on different rack-scale platforms would reveal how much performance comes from the algorithm and how much comes from one fabric.

Results on smaller clusters would be equally informative. They could establish the bandwidth threshold where dynamic replication stops paying for itself.

Decode support is another part of this signal. UltraEP currently targets training and prefill, where enough computation may exist to hide weight transfers.

A future decode design would need a different replication strategy, smaller transfer units, or stronger reuse across steps. Failure to address decode would keep UltraEP focused on only part of serving.

The third signal is framework adoption. Native or experimental integration into Megatron-LM, SGLang, vLLM, or DeepEP would expose the system to broader workloads.

Framework adoption would also create fairer comparisons against newer kernels and communication backends. The current baselines represent specific versions from the authors’ evaluation.

Production telemetry matters more than one benchmark. Operators should report tail latency, memory headroom, planner overhead, replica churn, and behavior during hardware faults.

They should also measure accuracy and training convergence over complete runs. Physical rerouting should preserve model semantics, but long deployments provide a stronger test than short continuations.

UltraEP MoE load balancing presents a clear claim: exact routing information can outperform historical placement when rack-scale communication makes immediate replication affordable.

The paper’s numbers support that claim inside the authors’ environment. Its 42% training gain and 1.56-times prefill result are large enough to demand attention, but not large enough to skip replication.

Developers should watch for code and reproducible benchmarks before planning deployments. Infrastructure buyers should ask whether their fabric can support expert movement on every layer’s critical path.

Teams evaluating the research can preserve papers, benchmark notes, and deployment decisions in a searchable engineering knowledge base. The immediate question is whether independent results will turn UltraEP from an impressive paper into deployable MoE infrastructure.