Cloudflare Smaller Model Serving Cuts Memory, but Safety Becomes the Test
Cloudflare smaller model serving now fits twice as much Kimi context into GPU memory, despite accepting slightly slower processing at ordinary concurrency. The company also compresses GLM weights and checks shared cache pages before supported decode operations read them. Together, these changes turn GPU memory from a fixed ceiling into a resource Cloudflare can actively manage.
That matters because Moonshot AI’s Kimi models and Z.ai’s GLM models are not ordinary additions to an inference catalog. They combine large parameter counts, long context windows, and mixture-of-experts architectures. A mixture-of-experts model activates selected parts of its network for each token, reducing computation without making its stored weights small.
The conflict is not simply Cloudflare against another inference provider. It is dense utilization against safe isolation on shared hardware. Packing more requests into one GPU improves economics and total throughput, but it also increases the consequences of faulty cache bookkeeping.
Cloudflare says its new configuration preserves model accuracy while increasing capacity. However, most supporting measurements come from Cloudflare’s own evaluation suite and infrastructure. The next test is whether those gains remain stable across workloads, hardware generations, and much larger production volumes.
What Cloudflare changed for Kimi and GLM
Cloudflare combined three memory techniques because no single optimization solves frontier-scale model serving.
The company detailed the changes in an August 3 technical account. It applies FP8 quantization to Kimi’s KV cache, INT4 compression to GLM’s weights, and integrity tags to shared cache pages. Each technique addresses a different constraint in the same inference system.
A KV cache stores the attention keys and values created for tokens the model has already processed. It lets a model continue a conversation without recomputing the entire prompt before every generated token. Long prompts and concurrent requests make that cache grow quickly.
Cloudflare stores Kimi K2.6 cache data in FP8 e4m3 rather than BF16. FP8 uses eight bits for each floating-point value, while BF16 uses sixteen. That conversion halves the cache’s memory footprint.
According to Cloudflare, the available in-memory context rises from about 686,000 tokens to roughly 1.37 million tokens. This is aggregate capacity on the tested deployment, not a new context-window limit for one user. The distinction matters because the optimization primarily increases concurrency.
Cloudflare made a separate change for GLM 5.2. It compressed model weights from FP8 to INT4, a four-bit integer representation. The checkpoint reportedly shrank from 705 GB to 421 GB, or about 40 percent.
An eight-way tensor-parallel deployment splits the model across eight GPUs. Under that configuration, Cloudflare says memory use fell from approximately 88 GB to 52 GB per GPU. The remaining space can hold around 1.18 million KV-cache tokens.
The third change protects the shared cache created by this denser packing. Every physical cache page receives a tag that changes whenever the page is reallocated. The server records the pages and tags that each request expects.
Before supported decode operations read those pages, Cloudflare checks the mappings. A mismatch causes the affected request to stop. The system should therefore fail closed instead of reading data associated with another request.
These techniques sit above Cloudflare’s wider large-model architecture. The company previously described Infire inference as a Rust-based engine designed for its distributed GPU network. It also separates prefill and decode into different resource pools.
Prefill processes an incoming prompt and creates its initial cache state. Decode generates the response one token at a time. Those phases stress GPUs differently, which lets Cloudflare optimize each pool separately.
That separation is essential to the new design. Cloudflare keeps BF16 caches and FP8 weights where computation dominates. It uses smaller representations where memory capacity or bandwidth becomes the limiting resource.
The result is not one universally compressed model configuration. It is a phase-aware serving system that changes formats according to the workload. That creates more operational complexity, but it avoids forcing one compromise across the entire request.
Why Cloudflare smaller caches beat raw speed
The Kimi KV cache quantization wins by admitting more work, not by making every request individually faster.
Cloudflare tested Kimi K2.6 decoding on a disaggregated H200 deployment. With one concurrent request, the BF16 cache delivered 137 tokens per second. The FP8 version delivered 125, making the compressed cache slower at that load.
The pattern continued as concurrency increased. At eight requests, BF16 reached 731 tokens per second, compared with 689 for FP8. At sixteen requests, the measurements were 1,106 and 1,028 tokens per second.
BF16 reached 1,558 tokens per second at 32 concurrent requests. However, it then exhausted available memory. Cloudflare says the FP8 cache continued to 64 requests and delivered 2,192 tokens per second.
That final number is about 41 percent above BF16’s highest measured throughput. Cloudflare also reports approximately 30 percent lower cost per token. The gain comes from doing more work on the same deployment, not accelerating each low-load request.
This distinction prevents an easy but misleading headline. Quantization introduces conversion work when the attention kernel reads cached values. At matched concurrency, Cloudflare’s BF16 results remained several percentage points faster.
Kimi KV cache quantization becomes valuable only after memory limits prevent the larger format from accepting more requests. It exchanges modest per-request efficiency for much greater total capacity. That is a sensible trade when demand stays high enough to use the added space.
It is less valuable when traffic is sparse. A deployment serving only a few simultaneous requests would absorb the conversion overhead without using the extra capacity. Cloudflare’s approach therefore depends on routing enough compatible work toward each decode pool.
This is where global infrastructure becomes strategically relevant. Large providers can aggregate traffic across many customers and keep expensive accelerators occupied. Smaller operators often face bursty demand, leaving them unable to capture the same utilization gains.
Cloudflare already uses session affinity and prefix caching in Workers AI. Prefix caching reuses the computed state of identical prompt beginnings. Its large-model rollout exposed cached-token usage and introduced a session-affinity header for better cache routing.
The newer optimization addresses a different layer. Prefix caching avoids repeated prefill work, while FP8 expands decode capacity. Combining both can reduce duplicated computation and admit more active sequences.
Cloudflare compared accuracy across several evaluations. On GSM8K, BF16 scored 94.24 while FP8 scored 94.09. MMLU results were 89.11 and 89.04, respectively.
The FP8 cache scored 67.49 on ARC-Challenge, compared with 66.72 for BF16. On MMLU-Pro, FP8 recorded 79.29 while BF16 reached 80.29. Tool-call validity measured 92.6 percent for FP8 and 92.2 percent for BF16.
Cloudflare describes these results as indistinguishable. That conclusion is plausible within the reported suite, but the scores do not prove universal equivalence. Small numerical changes can affect rare prompts, long agent traces, or tasks outside the selected evaluations.
Several tests also produce nondeterministic outputs. A tiny score difference can reflect sampling, evaluation noise, or quantization. Readers would need repeated trials and confidence intervals to separate those causes.
The company’s internal mcxams benchmark produced identical results, with both configurations passing 61 of 63 tests. Internal evaluations can reflect production needs well, but outsiders cannot independently inspect their coverage.
Kimi KV cache quantization should therefore be judged as an operational result with encouraging quality evidence. It is not a general finding that every model can safely use FP8 cache values. Attention distributions and numerical sensitivity differ across architectures.
Cloudflare’s real achievement is identifying where the format change pays. Prefill remains compute-bound, so the company leaves its cache in BF16. Decode becomes memory-bound, making the smaller FP8 representation useful at higher concurrency.
That choice supports the central thesis. Better inference does not always mean making one request run faster. At scale, it often means completing more useful work before hardware reaches its memory ceiling.
The real contest is memory per useful token
Frontier-model hosting increasingly depends on memory efficiency rather than headline parameter counts alone.
Kimi K2.6 belongs to a model family that combines large stored weights with long-context and agentic features. Cloudflare’s current model documentation lists a 262,144-token context window, vision inputs, tool calling, and structured outputs.
Those capabilities create overlapping memory demands. The model weights must remain accessible during generation. Each active conversation also builds a growing KV cache, while batching requires the server to track many sequences simultaneously.
A model can fit into GPU memory and still be uneconomical to serve. If its weights leave little space for cache pages, each deployment supports fewer active users. Idle or underfilled batches then waste expensive accelerator capacity.
This is the contest Cloudflare smaller representations are designed to change. The relevant metric becomes useful tokens produced per unit of memory, within acceptable latency and quality limits. Raw speed at one request reveals only part of that system.
The shift also pressures providers that depend mainly on standard inference stacks. If two services use similar hardware and model weights, the provider with better cache management can accept more concurrent work. It can also spread fixed infrastructure costs across more generated tokens.
However, software improvements do not erase hardware differences. H200 GPUs provide substantial high-bandwidth memory, while newer Blackwell systems add different low-precision capabilities. Results from one accelerator configuration will not transfer automatically to another.
Traffic shape matters just as much. Coding agents can submit large prompts, reuse prefixes, call tools, and continue across many turns. Consumer chat sessions may use shorter prompts and less predictable follow-up patterns.
An agentic workload can keep a sequence resident for longer. That raises the value of cache capacity, but it also makes scheduling harder. One unusually long request can occupy memory while many smaller requests wait.
Cloudflare’s disaggregated prefill and decode design responds to this imbalance. Compute-intensive prompt ingestion runs in one pool. Memory-sensitive generation runs in another, allowing each pool to scale and use different numerical formats.
This design also introduces coordination costs. Cache state must move or remain accessible across the phase boundary. Routing decisions must account for available memory, existing prefixes, queue depth, and the expected length of each response.
The SGLang project provides the serving framework used in Cloudflare’s experiments and production traffic. Cloudflare says it works with the project to upstream patches and features. That makes some improvements available beyond one provider.
Open infrastructure can narrow the software gap between large platforms and independent operators. Yet deployment experience still matters. A published kernel or scheduler feature does not automatically provide Cloudflare’s traffic volume, telemetry, or capacity planning.
That difference makes the competitive pressure indirect. Cloudflare is not claiming Kimi or GLM as exclusive models. It is arguing that its infrastructure can operate open-weight frontier models efficiently enough for shared, serverless access.
Model developers also benefit from this arrangement. Moonshot AI and Z.ai can reach users who do not want to provision multi-GPU clusters. Wider hosting support can increase adoption and create more feedback about real workloads.
The provider assumes a harder responsibility in return. It must preserve model behavior while transforming numerical formats. It must also prevent one tenant’s cache state from affecting another tenant’s output.
The strongest operators will therefore optimize three variables together: memory capacity, output quality, and isolation. Improving only two creates an unstable service. Higher density without isolation raises security concerns, while compression without quality testing risks silent regressions.
For buyers, benchmark leadership remains relevant but incomplete. An impressive model is useful only when the serving layer delivers predictable latency and correct tool calls. Long queues can erase the practical value of a stronger model.
Developers should also separate model quality from provider quality. The same Kimi or GLM checkpoint can behave differently across hosts because of quantization, sampling defaults, cache policies, and serving software.
A production evaluation should measure complete tasks, not isolated answers. Useful signals include tool-call validity, timeout rates, long-session consistency, and latency under realistic concurrency. Those metrics reveal whether memory optimization actually improves the application.
GLM weight compression shifts the phase tradeoff
GLM weight compression accelerates decode because smaller weights reduce memory traffic, but it slows the compute-heavy prefill phase.
Cloudflare converted GLM 5.2 weights from FP8 to INT4. During decode, the system repeatedly streams model weights from high-bandwidth GPU memory. Moving fewer bytes can produce each token sooner when memory bandwidth is the bottleneck.
The low-concurrency result was the clearest. At one request, FP8 generated 60 tokens per second, while INT4 reached 92. That represents a reported gain of 55 percent.
At eight concurrent requests, throughput rose from 425 to 513 tokens per second. The gain was 21 percent. At sixteen requests, INT4 produced 825 tokens per second, compared with 683 for FP8.
The improvement remained visible at higher load. INT4 reached 1,267 tokens per second at 32 requests, versus 994 for FP8. At 64 requests, the respective results were 1,933 and 1,672.
GLM weight compression does not create the same benefit during prefill. INT4 weights must be expanded before matrix multiplication. Cloudflare measured approximately 8,660 prefill tokens per second for INT4, compared with 10,160 for FP8.
Using INT4 everywhere would therefore sacrifice prompt-processing throughput. Cloudflare instead keeps FP8 for prefill and assigns INT4 to decode. The split preserves the best measured format for each phase.
This finding echoes Cloudflare’s earlier work on lossless weight compression. That project explored multiple execution paths because batch sizes and matrix shapes change the balance between decompression and computation.
The current GLM approach uses lossy quantization rather than that earlier lossless method. Converting FP8 weights to INT4 maps more values into fewer representable states. Accuracy testing becomes essential because the original weights cannot be reconstructed exactly.
Cloudflare reported differences below 0.8 points across its evaluated benchmarks. MMLU averaged 86.60 for FP8 and 86.54 for INT4. MMLU-Pro exact scores were 80.80 and 80.47.
On ARC-Challenge accuracy, FP8 recorded 64.93 percent and INT4 recorded 64.85 percent. Both formats passed 62 of 63 cases in Cloudflare’s internal mcxams benchmark.
GSM8K showed a somewhat wider difference. FP8 achieved 94.39 percent exact match, while INT4 reached 93.56 percent. Flexible scoring produced 94.24 and 93.48 percent.
Cloudflare says the compressed model’s quality is indistinguishable. The public numbers support a small average difference, but they leave several questions open. The company did not publish results for every agentic or multilingual behavior.
GLM is commonly used for coding, tool use, and multilingual tasks. General academic benchmarks do not capture every failure mode in those settings. A malformed function argument can matter more than a small average accuracy change.
Rare numerical failures are particularly difficult to detect. A benchmark can show stable aggregate quality while a compressed model changes behavior on unusual prompts. Long reasoning traces may amplify an early difference.
That does not make INT4 unsuitable. It means deployment decisions need workload-specific tests and continued monitoring. Providers should compare the exact model configuration users receive, not only a reference checkpoint.
The same caution applies to speed claims. Tokens per second depend on input length, output length, batching, hardware, kernels, and scheduling. Cloudflare’s measurements describe its tested system rather than a universal GLM performance level.
Still, the phase split offers a useful architectural lesson. Quantization should not be treated as one static export decision. A provider can maintain multiple representations and route computation toward the format that fits each stage.
That flexibility has costs. Multiple weight formats consume storage and complicate deployment. Engineers must verify compatibility, select the correct kernels, and prevent configuration drift across GPU pools.
Operational discipline determines whether the complexity pays. If a request reaches the wrong pool or a model revision changes numerical behavior, theoretical throughput gains matter little. Automation and observability become part of the optimization itself.
Cloudflare’s reported results show why providers accept that complexity. A 40 percent smaller checkpoint leaves substantial memory for active sequences. Faster decode then improves latency where users see tokens arrive.
The tradeoff is concrete rather than abstract. GLM weight compression buys decode capacity and speed by adding numerical risk and prefill overhead. Cloudflare manages that trade by isolating the format to the phase where it wins.
Shared KV cache safety becomes a production issue
Higher GPU density raises the value of every cache page, while making faulty ownership tracking more dangerous.
Paged attention divides KV-cache storage into reusable blocks rather than requiring one continuous allocation per request. Continuous batching then adds and removes sequences while a GPU remains busy. Together, these methods reduce wasted memory.
They also create a demanding bookkeeping problem. Physical cache pages are constantly assigned, read, released, and reassigned. The serving system must preserve the correct mapping between every logical sequence and its physical pages.
A stale mapping could make a request read an incorrect page. That might corrupt the response, crash the operation, or expose state associated with another sequence. The exact consequence depends on the failure and surrounding controls.
Cloudflare says very rare mistakes become operationally relevant at its request volume. Its article uses a one-in-a-billion error as an illustrative threshold. That statement describes the need for defense, not a disclosed incident rate.
The company’s integrity mechanism associates a changing tag with each physical page. A request records the tags it expects. Supported decode operations validate those values before reading the shared cache.
A tag mismatch aborts the affected request. This design favors an explicit failure over returning output based on the wrong state. It resembles generation counters used in other memory-management systems.
Cloudflare evaluated the check on a mid-sized production model using two prefill and two decode workers. Tests used 8,192-token inputs and 1,000-token outputs. Reported concurrency ranged from one to eight.
Throughput declined by 0.38 to 0.79 percent across those tests. The p95 latency increase ranged from 0.42 to 0.80 percent. Cloudflare says even the upper confidence bound remained near one percent.
The company runs validation as a separate batch check. It avoided fusing the operation into the attention kernel because GPU thread groups could create a race. A no-op tracker remains available for deployments where checking is disabled.
These results make integrity checks appear inexpensive. However, the evaluation did not cover every model size, sequence shape, or concurrency level. It also focused on supported decode operations, a qualifier that deserves attention.
Readers should not interpret the mechanism as a complete proof of tenant isolation. Cache integrity is one defensive layer within a larger serving system. Routing, memory allocation, kernel correctness, and process isolation remain relevant.
The check can detect a page-generation mismatch that its tracker understands. It cannot automatically identify every numerical error or software defect. A valid tag does not prove the page contents are semantically correct.
There is also a tension between optional deployment and universal protection. Cloudflare says integrity checking is enabled per deployment. Its stated goal is to make the feature inexpensive enough to leave enabled everywhere.
Until that happens, customers cannot assume every model path uses the same protection. Clear documentation about coverage would help developers assess the remaining risk. Independent security testing would provide stronger evidence than performance measurements alone.
The safety case nevertheless reflects an important change in inference engineering. Performance features now require explicit reasoning about cross-request state. Memory optimization can no longer be evaluated only through throughput graphs.
Compression intensifies that need. FP8 Kimi caches let more requests remain active. INT4 GLM weights create more space for cache pages. Both changes increase the amount of shared state that one deployment handles.
The main opponent in this story is therefore unsafe density. The objective is not maximum packing at any cost. It is higher utilization while preserving request boundaries and acceptable model behavior.
Cloudflare’s approach places the safety check near the resource being shared. That can catch allocation mistakes before cached data enters an attention operation. Early termination also limits the propagation of corrupted state.
Aborted requests still affect reliability. If checks begin firing frequently, users will see errors or retries even when isolation works as designed. Operators must track mismatch rates, investigate causes, and prevent retry storms.
Cloudflare has not disclosed a production mismatch rate in the article. That missing metric matters more than synthetic overhead alone. A low-cost check is useful, but its operational value becomes clearer when it reports real detections.
The company also has an incentive to present the new density as safe. Its evidence should be treated as a technical disclosure from the system’s operator. It is informative, but not equivalent to an independent audit.
For developers, the broader lesson is practical. Shared inference hides infrastructure complexity, but it does not remove it. Provider evaluations should include isolation controls, failure handling, and incident transparency alongside latency and model quality.
What to watch as the optimizations spread
Three signals will show whether Cloudflare smaller model serving becomes a durable advantage or remains a specialized configuration.
The first signal is wider FP8 cache deployment. Cloudflare says it is expanding FP8 KV caches across more of its fleet. Coverage across additional models and hardware would show that Kimi KV cache quantization generalizes beyond one measured setup.
The useful evidence would include concurrency, tail latency, and quality data for different sequence lengths. Stable results across those dimensions would strengthen Cloudflare’s memory-efficiency argument. Frequent model-specific exceptions would weaken it.
The second signal is NVFP4 validation on Blackwell GPUs. NVFP4 is Nvidia’s four-bit floating-point format for newer hardware. Cloudflare says it is testing that representation as another route to smaller weights.
A successful rollout could extend GLM weight compression beyond INT4 and change the prefill-decode balance again. It would also test whether Cloudflare can translate its phase-aware strategy across accelerator generations.
The important result is not a single peak-throughput number. Watch for end-to-end latency, quality evaluations, and memory capacity under production batching. Those measurements would show whether lower precision delivers useful application-level gains.
The third signal is universal cache-integrity coverage. Cloudflare wants checks enabled everywhere at negligible cost. Reaching that goal would connect higher density with a consistent default safety control.
Coverage disclosures should identify supported models, operations, and hardware. Detection metrics would add even more value, particularly if Cloudflare explains how often mismatches occur and what causes them.
These signals matter because the company’s current evidence is strong but bounded. It shows meaningful gains on selected models and deployments. It does not establish that every workload benefits from the same formats.
Developers should test hosted Kimi and GLM systems using realistic prompts, tool chains, and concurrency. Track time to first token, generation speed, timeout rates, and complete-task success. Compare behavior after provider-side model updates.
Teams should also preserve their own evaluation history. A searchable engineering knowledge base can connect benchmark results, incidents, configuration changes, and provider announcements. That context helps distinguish a model regression from an infrastructure change.
Cloudflare has moved the frontier-model argument away from simple availability. The harder question is whether a provider can keep large models fast, economical, accurate, and isolated under real concurrency. Watch the three rollout signals, then judge the system by complete workloads rather than one benchmark or throughput chart.
