Wayfinder Router: Deterministic Routing Between Local and Hosted LLMs

Wayfinder Router routes prompts between local and hosted LLMs by checking structure and wording on the device itself.

The tool reads length, presence of headings, lists, code blocks, and terms such as proof or hard constraint. It assigns a score in microseconds and selects the target model without any external call. Users can adjust the threshold on their own data. The system stays offline by design and works with any OpenAI-compatible endpoint, including Ollama, Anthropic, Groq, and vLLM.

RouteLLM and NotDiamond require a model call for every routing decision. That step adds latency and cost while injecting randomness into the outcome. Wayfinder Router removes the call entirely. The default configuration uses only structural signals because vocabulary clues failed generalization tests during blind evaluation. Lexical cues remain disabled unless the user turns them on after calibration.

Prompt Features That Drive Decisions

Wayfinder Router inspects four structural signals first. Length matters because longer prompts often need more context than a small local model can supply. Headings and lists signal organized requests where accuracy counts. Code blocks raise the score toward a hosted model when precision in syntax or reasoning is required.

The second stage looks for wording that matches hard constraints or mathematical language. When these patterns appear, the score pushes the prompt away from local models. All calculations finish locally and return a route in under one millisecond on ordinary hardware.

Users receive a terminal command and a web demo that runs in dry-run mode without any API keys. Both demos expose the raw score for every tested prompt. Benchmark scripts and a FAQ file ship with the repository.

Comparison With Model-Based Routers

Model-based routers call an auxiliary LLM to decide the target. Each decision consumes tokens and time. The auxiliary call can choose the wrong model because the judge itself is stochastic. Wayfinder Router replaces that judge with fixed rules derived from prompt structure.

The offline route removes three recurring problems at once. No added network round-trip occurs. No extra token cost accrues. No random deviation affects the chosen path. Teams that need predictable behavior in production pipelines can therefore rely on the same routing outcome every time the same prompt shape appears.

Calibration On Private Data

A supplied script lets users label a set of their own prompts as better suited to local or hosted models. The script computes the structural scores for each sample and suggests a threshold that matches the supplied labels. After calibration the chosen threshold applies to new prompts without further human review.

The same process works for any domain. Customer-support teams can favor speed on simple queries while routing legal or financial prompts to stronger models. Engineering groups can keep code-generation tasks on capable endpoints while letting quick documentation questions run locally.

Integration Path

The package registers as an OpenAI-compatible client. Existing code that already targets an OpenAI endpoint can swap the base URL and receive routing without changes to request format. Self-hosting requires only the Python package and a supported endpoint list.

Terminal users can invoke the router directly from the command line for batch processing. The web demo surfaces the same decision logic inside a browser so non-technical stakeholders can test prompts before committing to production flows.

Limits and Future Checks

The current implementation ignores semantic meaning beyond the structural and lexical signals described. Complex domain reasoning may still require manual overrides. The maintainers publish monthly benchmark updates so users can verify that the chosen thresholds continue to match observed outcomes over time.

Three signals will indicate whether the approach remains viable: release of new evaluation datasets that test routing accuracy on private corpora, updates to the supported model list that include faster local variants, and community reports on calibration stability after three months of use.

Developers who already run mixed local and hosted fleets can test the router today through the dry-run demo. The package remains available under an open-source license at the public repository.