Cactus Gemma 4 E2B Hybrid Adds Confidence Scores, but Routing Is the Real Test

Cactus has released Gemma 4 E2B Hybrid with a direct challenge to the usual on-device AI tradeoff. Every answer now carries a confidence score between zero and one. Applications can keep confident answers local while sending uncertain requests to a larger cloud model.

That sounds like a minor output change. It is actually a different deployment model for small AI systems.

Most local models either answer every request or depend on a separate router to decide which requests belong in the cloud. Cactus puts its decision signal inside the model checkpoint. The company says that signal comes from internal hidden states, not from confidence language added to the answer.

The distinction matters because a small model does not need to beat a cloud model on every request. It only needs to identify the requests it can handle reliably. A good routing signal can turn that limited model into the first stage of a larger system.

Cactus claims its smallest Gemma hybrid can match Gemini 3.1 Flash-Lite on most tested benchmarks while escalating only part of the workload. Depending on the benchmark and numerical precision, the reported handoff rate ranges from 15 percent to about 90 percent.

The lower figures make an appealing case for local inference. The higher figures expose the central uncertainty. The model's usefulness depends on the task, hardware format, confidence threshold, and cost of a wrong local answer.

The release therefore raises a more important question than whether Gemma 4 runs on a phone. Can one embedded probe make local-to-cloud routing dependable across text, vision, and audio?

Cactus Gemma 4 E2B Hybrid Turns Confidence Into an API Output

The central change is not another small model. It is a model checkpoint that returns a routing signal with every completed answer.

Cactus describes its approach in the open hybrid model repository. The initial rollout centers on Gemma 4 E2B, the smallest model in Google's new Gemma 4 family.

Google positions E2B and E4B models for mobile and edge deployments. Larger Gemma 4 variants target consumer GPUs and workstations, according to the official Gemma 4 model card.

Cactus post-trains the smaller model with a probe that reads its hidden representation. A probe is a lightweight learned component that maps internal model activations to a useful prediction.

Here, the prediction is whether the completed response is likely to be correct. The application receives that result as structured data alongside the answer.

This design avoids asking the model to write phrases such as “I am 85 percent confident.” Verbal confidence can reflect writing style, instruction tuning, or user prompting. It does not necessarily track correctness.

The structured output also avoids parsing a score from generated text. That choice reduces the chance that formatting changes, prompt injection, or an unusual response will break routing logic.

Cactus shows a simple threshold policy in its documentation. When confidence falls below 0.85, the application can discard the local result and ask a larger model.

The threshold is an example, not a universal safety boundary. Each developer must choose one based on error tolerance, traffic, latency, and the consequences of escalation.

A note-taking application might accept more local answers when summarizing informal material. A medical or financial workflow would need stricter testing and additional safeguards.

The model is available through several common inference paths. Cactus provides examples for its own runtime, Hugging Face Transformers, Apple's MLX framework, and a patched llama.cpp server.

In each case, the goal remains consistent. The answer and confidence score should arrive through separate output fields.

The repository is licensed under MIT, although Gemma model use remains subject to Google's terms. The associated checkpoints are collected in the public model collection.

That accessibility gives developers something more useful than a conceptual demo. They can test the checkpoints, inspect integration code, vary thresholds, and measure behavior on their own workloads.

However, the llama.cpp path requires a compiled patch. The Transformers instructions also document specific version and device-loading constraints.

Those details reveal an important practical limit. An embedded probe changes the model interface, so existing inference engines do not automatically understand it.

Cactus has reduced the integration burden, but it has not eliminated it. Production teams still need compatible runtimes, deployment tests, and monitoring around the confidence field.

Why the Confidence Probe Matters More Than Raw Model Size

A small model becomes more valuable when it can abstain selectively, not when it pretends to handle every request equally well.

The underlying idea is called selective prediction. A model answers when its estimated risk is acceptable and abstains when that risk becomes too high.

Researchers have studied this framework long before the Cactus release. The Selective-LAMA study found that confidence-aware evaluation can expose weaknesses hidden by ordinary accuracy scores.

That research also warned against assuming token probability provides the best confidence function. A fluent model can assign high probability to an answer that is still factually wrong.

Google explored another version through ASPIRE, a selective prediction method that modifies models and trains them to produce better selection scores. Its selective prediction research reported stronger AUROC results than several baseline methods.

AUROC measures ranking quality across possible thresholds. A score of 0.5 represents random separation, while 1.0 represents perfect separation between correct and incorrect outputs.

AUROC does not tell developers that a score of 0.85 means an answer is correct 85 percent of the time. That would require calibration analysis.

Instead, AUROC asks whether correct answers generally receive higher scores than incorrect answers. Good ranking allows a system to retain safer requests while escalating riskier ones.

Cactus reports a mean AUROC of 0.814 across 12 text, vision, and audio evaluations. Its token entropy baseline averaged 0.549.

Token entropy measures uncertainty across predicted tokens. It is attractive because many inference systems can calculate it without training another component.

The reported gap suggests the internal probe sees a more useful correctness signal than token-level uncertainty alone. That conclusion remains a company claim until independent evaluations reproduce the tests.

Results varied across the benchmark set. Cactus reported 0.770 on MMLU and 0.771 on MMLU-Pro. It reported 0.888 on ARC-Easy and 0.834 on ARC-Challenge.

The vision results included 0.840 on MMBench, 0.779 on ChartQA, and 0.781 on DocVQA. The corresponding token entropy scores ranged from 0.435 to 0.615.

Audio produced the release's most interesting result. Cactus says the probe received no audio training data, yet recorded AUROC scores from 0.789 to 0.876 across four audio benchmarks.

Those benchmarks included MMAU, GigaSpeech, Earnings-22, and LibriSpeech. Token entropy ranged from 0.323 to 0.517 on the same tasks.

Cactus interprets this transfer as evidence that the probe reads a modality-independent correctness signal. That is plausible, but the available evidence does not fully establish the mechanism.

A probe can exploit correlations that survive across benchmarks without representing correctness in a general sense. Hidden dataset patterns, answer length, decoding behavior, and evaluation rules can all affect performance.

The zero-audio result still deserves attention. If independent tests confirm it, developers may not need a separate confidence system for every input modality.

That would be especially useful for assistants handling documents, images, recorded meetings, and spoken questions. One routing interface could cover several media types.

Teams building these applications also need context, not only model confidence. Local access to prior decisions, notes, and documents can improve the input before routing begins.

A searchable personal knowledge base can supply that context. The confidence probe then addresses a separate issue: whether the generated answer should be trusted locally.

The Real Contest Is Local First Versus Cloud by Default

Cactus is pressuring cloud-first inference by arguing that uncertainty, rather than model size, should determine where each request runs.

A cloud-first system sends every prompt to a remote service. The design provides consistent access to a larger model, but every request depends on connectivity and external infrastructure.

A local-first system keeps processing on the device. It can reduce network exposure, work offline, and respond without a cloud round trip.

Neither route wins universally. Small local models have tighter memory and compute limits, while cloud models can offer broader capabilities and more frequent updates.

Hybrid routing attempts to assign each request to the cheaper or more private path that still produces an acceptable answer. The router becomes the system's control point.

Cactus reports the handoff rate needed for Gemma 4 E2B Hybrid to match Gemini 3.1 Flash-Lite on selected benchmarks. These figures come from the company, not an independent audit.

At full FP16 precision, the reported cloud share was 15 to 20 percent on ChartQA. It reached 30 to 35 percent on MMBench, GigaSpeech, and MMAU.

LibriSpeech required a reported 25 to 30 percent handoff rate. MMLU-Pro was much less favorable, requiring 45 to 55 percent of requests to reach the cloud.

Quantization changed the picture. Quantization reduces numerical precision so a model uses less memory and can run more efficiently on constrained hardware.

At four-bit precision, the reported handoff rate rose to 25 to 30 percent on ChartQA. MMBench and GigaSpeech rose to 40 to 45 percent.

MMLU-Pro required about 90 percent cloud handoff at four bits. Cactus did not report a three-bit result for that benchmark.

This spread prevents a simple claim that the local model handles most requests. It handles most requests only for particular tasks, model formats, and quality targets.

A chart-reading application might keep a large share of its workload local. A demanding knowledge task resembling MMLU-Pro might receive little benefit after quantization.

The comparison target also matters. Matching Gemini 3.1 Flash-Lite is not the same as matching the largest available cloud model.

A production team might choose a stronger fallback because the escalated requests are already the difficult cases. That change could improve quality while increasing latency or operating cost.

Routing also creates two user experiences. A local answer can arrive quickly, while an escalated answer must wait for network and cloud inference.

Developers must decide what appears during that delay. They can show a progress state, stream the cloud replacement, or disclose that a stronger model is checking the response.

Applications must also control what leaves the device. A confidence score does not make sensitive content safe to upload.

If a request contains private meeting notes, customer records, or proprietary code, the application needs a separate data policy. Low confidence should not override privacy restrictions automatically.

One practical policy could combine several signals. The system could check sensitivity first, then connectivity, confidence, latency budget, and available cloud models.

Under that policy, some uncertain requests would remain local but receive an explicit warning. Others could route to a private enterprise endpoint instead of a public service.

This is why the primary contest is architectural. Cactus is not merely comparing one Gemma checkpoint against one Gemini endpoint.

It is asking whether cloud use should become an exception triggered by measured uncertainty. If that model works, applications can treat local inference as their default execution layer.

What the Benchmarks Still Do Not Establish

The reported AUROC results show promising ranking behavior, but they do not establish calibrated trust or production reliability.

The first open issue is calibration. A score between zero and one looks like a probability, but its numerical appearance does not make it one.

A calibrated 0.80 score should correspond to approximately 80 percent correctness among comparable predictions. Cactus primarily reports AUROC, which measures ranking instead.

A router can perform well by rank while still producing misleading absolute scores. Developers should therefore select thresholds on a representative validation set.

The second issue is distribution shift. Benchmark prompts are cleaner and more stable than requests from a deployed assistant.

Real users mix incomplete questions, private terminology, changing facts, attachments, transcription errors, and conflicting instructions. These inputs can change both answer quality and confidence behavior.

The audio transfer result addresses one form of shift, but it does not cover every production condition. New accents, background noise, specialized vocabulary, and long recordings remain relevant tests.

The third issue is selective accuracy. Developers need to know the error rate among answers retained below each cloud budget.

AUROC summarizes performance across all thresholds. It does not directly reveal whether a particular threshold meets a product's required accuracy.

A production evaluation should plot coverage against risk. Coverage is the share of requests answered locally, while risk is the error rate within that retained share.

Teams should also compare the hybrid system with simpler alternatives. Token entropy is one baseline, but it is not the only possible router.

Other options include repeated sampling, semantic consistency, a separate verifier, retrieval confidence, prompt classification, and rules based on task type.

Some methods require more computation. Others work before generation and can avoid spending local compute on a request that will reach the cloud anyway.

Cactus scores completed generations. That timing means the device first pays the cost of producing an answer, then may discard it and repeat the request remotely.

This approach still saves cloud calls, but it does not minimize total computation. A pre-generation router might be faster for obviously difficult prompts.

A combined design could classify requests before generation and apply the probe afterward. Easy requests would stay local, obvious escalations would skip local generation, and ambiguous cases would use both stages.

The fourth issue is benchmark ownership. Cactus published the checkpoints, code, claimed results, and implementation notes, which supports external examination.

However, the repository does not provide the equivalent of a peer-reviewed evaluation with independent replication. The reported numbers should remain attributed to Cactus.

The fifth issue is fallback quality. A stronger cloud model does not guarantee a correct replacement.

If both models share training gaps or interpret the prompt similarly, escalation can repeat the original error. Applications still need source grounding and domain-specific validation.

For document questions, retrieval quality may matter more than model size. A larger model cannot answer from a missing contract clause or an outdated project record.

This makes workflow context an important complement to routing. Systems that blend current documents with prior work can reduce uncertainty before either model responds.

A knowledge blending workflow can combine related local information for that purpose. The routing score then helps decide whether the resulting answer needs stronger inference.

High-stakes applications require another layer. Medical, legal, security, and financial outputs may need human review even when model confidence is high.

A confidence score should support a risk policy, not replace one. The difference becomes essential when the cost of a confident mistake exceeds the cost of escalation.

Why This Release Arrives at the Right Moment

Gemma 4 makes a small multimodal model available, while device hardware and cloud costs make selective execution increasingly relevant.

Small models now handle more than short text completion. Gemma 4 E2B targets text, vision, and audio workloads within a model intended for edge-class deployment.

That broader input range changes the economics of local AI. A single assistant can interpret a photographed chart, transcribe speech, summarize text, and answer follow-up questions.

Yet multimodal capability also creates uneven difficulty. Reading a clean chart differs from understanding noisy audio or solving a specialized knowledge question.

A fixed routing policy based only on modality would miss those differences. Confidence-based routing attempts to make the decision at the individual answer level.

The approach also fits emerging AI agents. Agents often perform many small operations rather than one isolated request.

A meeting workflow might transcribe audio, identify decisions, create tasks, draft a summary, and answer questions about the discussion. Sending every step to a large model can be excessive.

A hybrid system could keep routine extraction and formatting local. It could escalate ambiguous decisions, conflicting statements, or difficult cross-document questions.

However, confidence propagation becomes a new challenge. An early mistake can contaminate later steps even when each later answer appears confident.

Agent developers should therefore store confidence with intermediate outputs. They should also reconsider downstream work when a critical earlier step receives a low score.

The same principle applies to offline use. A device may have no reachable cloud fallback during travel, field work, or a network failure.

In that situation, confidence remains useful even without routing. The application can abstain, warn the user, save the request, or retry when connectivity returns.

Cactus therefore gives developers two related capabilities. It supports automatic handoff when the cloud is available and visible uncertainty when it is not.

This is more concrete than a model writing cautious language. A structured score can drive application behavior, monitoring, and policy enforcement.

Still, developers should avoid presenting the number directly to users without calibration. “Confidence 0.82” sounds precise even when its practical meaning varies across tasks.

A better interface might translate validated score ranges into actions. The application could answer normally, request clarification, check sources, or send the task for review.

That design keeps the uncertainty signal operational. It does not ask users to interpret a machine-learning metric during ordinary work.

The timing also reflects pressure on model providers. If small models can safely retain more requests, cloud vendors face lower inference volume for routine tasks.

At the same time, hybrid systems can increase demand for cloud models on the hardest requests. Providers may respond with specialized fallback endpoints or their own device-cloud routing layers.

Hardware companies also have an incentive to support the pattern. Better runtime integration could expose probe outputs without custom patches and reduce the friction Cactus currently documents.

The release is therefore an early implementation of a larger shift. Model selection is moving from a product-wide decision to a per-request decision.

Three Signals Will Determine Whether Cactus Hybrid Holds Up

Independent replication, real-device coverage curves, and native runtime support will decide whether this becomes an architecture or remains a promising checkpoint.

The first signal is independent benchmark replication. Researchers and developers need to reproduce the reported AUROC results using the released checkpoints.

Replication should preserve Cactus's evaluation settings before testing alternatives. That makes it possible to separate implementation differences from model behavior.

The most important result to verify is cross-modal transfer. Similar audio performance without audio probe training would strengthen the claim of a shared correctness signal.

A large drop would weaken that interpretation. It could indicate sensitivity to decoding settings, dataset preparation, or undocumented evaluation choices.

Independent testing should also examine calibration. Reliability diagrams and expected calibration error would clarify whether the zero-to-one output behaves like a usable probability.

The second signal is a real-device risk-coverage curve. Developers need measurements from phones, laptops, and edge hardware under realistic memory constraints.

Those tests should include latency, energy use, model loading time, local error rates, and cloud handoff frequency. Quantized models deserve particular attention.

Cactus's own figures already show that precision changes routing economics. A four-bit build may fit the device better while sending substantially more requests away.

The right question is not whether local generation is faster in isolation. It is whether the complete hybrid path improves quality, privacy, latency, and resource use together.

Tests should also include changing network conditions. A router that works on office Wi-Fi may feel very different on cellular service or an intermittent connection.

The third signal is native support from inference runtimes. The current examples prove integration is possible, but patches and version constraints limit adoption.

Broader support in llama.cpp, Transformers, MLX, or mobile runtimes would reduce custom engineering. It would also encourage common conventions for confidence output.

Standard interfaces could let developers swap models without rebuilding application logic. They could also support monitoring tools that compare scores, outcomes, and handoff rates.

Without those conventions, every confidence-enabled checkpoint risks becoming a one-off integration. That would slow experimentation and make comparisons harder.

Cactus Gemma 4 E2B Hybrid has already made one useful idea concrete. A local model does not need to answer everything to become the default.

The harder task begins after release. Teams must validate thresholds, protect sensitive inputs, measure retained error rates, and decide what happens when no cloud route exists.

Developers should start with a bounded workflow that has verifiable outcomes. Document extraction, chart questions, or recorded meeting transcription provide clearer evaluation than open-ended conversation.

They can then compare three paths: always local, always cloud, and confidence-routed hybrid. The useful result is not the highest headline score.

It is a policy that meets a defined error limit while retaining enough requests locally to justify the added system complexity.

Watch the next independent evaluations closely. If the probe preserves its ranking quality across devices and private workloads, confidence can become a practical routing primitive.

If performance collapses under distribution shift, the release will still offer a valuable lesson. A structured uncertainty field is only as trustworthy as the evidence behind its threshold.