Why AI Agent Builders Should Focus on Routing First

Building AI agents should start with routing logic instead of model choice. The router decides which layer of models processes each request, and this decision determines cost more than any single model swap.

Recent analysis from investor Tomer Tunguz highlights the same point. Tunguz argues that correct routing moves the majority of work to zero-cost local models or cheaper async batch jobs. Coinbase offers a concrete case. CEO Brian Armstrong reported that improved defaults, routing, and caching cut AI spending in half even as token usage grew.

Most agent traffic never needs a top-tier model. Simple classification, data lookup, and routine decision steps can run on smaller open-source models hosted locally. Only complex reasoning or high-stakes tasks require larger frontier models.

Three-layer routing structure

The recommended design uses three distinct layers. A skill classifier first tags the incoming task. A router then selects the execution path. A model selector finally picks the specific model or inference mode inside that path.

This separation keeps each layer simple and testable. The classifier only needs to identify intent. The router only needs historical performance data and current cost signals. The model selector only needs latency and accuracy targets for the chosen path.

Cost differences that routing exploits

Local inference runs at near-zero marginal cost once the hardware is already owned. Async batch inference costs roughly two orders of magnitude less than real-time calls. Most agent workflows do not require sub-second answers, so the batch path is viable for the majority of volume.

When an agent uses synchronous prediction only for edge cases and schedules the rest for overnight batch evaluation, overall spend drops sharply. The nightly batch job can also retrain routing weights based on actual accuracy results, closing the loop without extra engineering overhead.

After skill distillation into smaller specialized models, 70 to 80 percent of non-coding agent traffic can stay local. The remaining slice still benefits from selective use of larger models only when the router flags genuine need.

Coinbase example shows measurable impact

Coinbase did not publish exact model names or token counts. It did state that better default prompts, smarter routing, and aggressive caching produced a 50 percent spending reduction while usage continued to climb. The result demonstrates that routing improvements compound faster than any single model upgrade.

Similar patterns appear elsewhere. Teams that added a lightweight classifier before every agent call reported the same pattern: most queries never reach the expensive model, yet overall task completion rates stayed flat or improved because the right model was chosen more consistently.

Why model choice alone fails to solve cost

Swapping to a cheaper model across the board usually hurts accuracy on hard tasks. Keeping a single expensive model for everything wastes money on easy tasks. Routing removes the binary choice. It lets the system keep high accuracy where it matters and low cost everywhere else.

The router also captures context that static model selection cannot. It can consider time of day, current queue length, user tier, or even recent accuracy feedback. Those signals change continuously, so a static model list becomes outdated within days.

Implementation realities and limits

Building the router requires initial instrumentation. Every task must be logged with its skill tag, chosen path, latency, cost, and final accuracy. Without that data, the router has nothing to optimize. Teams that skip logging usually end up with brittle rules that drift over time.

Another constraint is the quality of the initial classifier. If the classifier mislabels 15 percent of tasks, downstream routing errors multiply. Investing early effort in a high-precision classifier pays off more than adding another model option.

Data privacy rules may also limit which tasks can leave the local environment. The router must respect those constraints before cost calculations even begin. In regulated settings this requirement can shift the optimal split between local and remote models.

What to watch in the next quarter

Teams shipping agent platforms will publish routing accuracy numbers alongside raw model benchmarks. Those numbers will show whether the three-layer design holds under real traffic.

Coinbase and similar companies may release follow-up disclosures on cost per active user. Sustained reductions would confirm that routing improvements scale beyond early experiments.

Finally, open-source routing frameworks will appear with pre-trained classifiers and default weight-update logic. Adoption rates of those frameworks will indicate how quickly the pattern spreads beyond companies with dedicated infrastructure teams.