Optimizing Agentic Workloads with Per-Request Model Routing
Engineering teams using large language models face steep API costs when routing every prompt to frontier models by default. Analysis of production workloads shows that 60% to 70% of agentic requests consist of short, simple completions capable of running on low-cost alternatives at quality parity. Weave Router provides a Go-based proxy that inspects each incoming request, maps it against intent clusters, and dispatches it to the most cost-effective model. On production Claude Code traffic, this approach achieves 80% to 85% cost reductions while maintaining output quality and preserving developer experience.
The Financial Drain of Fixed-Model Inference Architecture
Most software development teams building AI-assisted tools point their applications at a single frontier model API. Every prompt, from multi-file refactoring to trivial syntax autocompletions, incurs the maximum token rate. Because frontier models carry higher per-token prices than smaller open-source options, this blanket routing strategy leads to significant budget inflation. In agentic workflows, multi-turn interactions continuously re-send system prompts and conversation histories, multiplying the financial impact of using high-cost models for simple tasks.
The underlying issue lies in dispatch logic rather than model capability. A substantial portion of developer interactions requires basic code modifications or structured data extraction rather than advanced reasoning. Routing every action to a top-tier provider creates unnecessary expense without producing measurable quality improvements. To address this inefficiency, engineering organizations require an intermediate gateway that evaluates each prompt before choosing an execution target.
How Weave Router Evaluates and Dispatches Prompts
Weave Router functions as a standalone Go service that sits between client applications and model providers. It acts as a drop-in replacement for Anthropic Messages, OpenAI Chat Completions, and Google Gemini generateContent endpoints. Instead of relying on static keyword lists or basic prompt length heuristics, the system utilizes a small ONNX model to generate vector embeddings for incoming prompts. The service scores these embeddings against a frozen set of intent cluster centroids derived from the Avengers-Pro research line arXiv:2508.12631.
After identifying top candidate clusters, the routing engine selects the lowest-cost model that has historically demonstrated quality parity with frontier models for that specific category. The classification pipeline executes quickly, adding low single-digit milliseconds of overhead to overall request processing. To protect the economics of prompt caching during long agentic sessions, Weave Router pins a session to its initial model choice. The proxy revisits its selection only when switching providers offers end-to-end savings after factoring in cache warm-up overhead on the destination system.
On production Claude Code traffic, Weave Router achieves measured cost reductions of 80% to 85% without quality regression.
Weave Engineering Blog
Model Options and Economic Comparisons
Evaluating inference efficiency requires examining the trade-offs between execution cost, response latency, and generation quality. The table below outlines model metrics recorded across routed benchmarks and provider environments:
Model / Configuration
Quality Score
Relative Cost Profile
Latency / Overhead
Weave Routed Strategy
94 / 100
1/38th of frontier baseline
3 ms proxy overhead
DeepSeek V4 Pro
94 / 100
Low ($)
Under 50 ms classification
Claude Sonnet 5
88 / 100
High ($$$)
Standard provider latency
Claude Opus 4.8
94 / 100
Highest ($$$$)
Standard provider latency
GPT-5 Codex
86 / 100
Moderate ($$)
Standard provider latency
To support different operational requirements, Weave Router offers three primary routing policies:
Quality-first
Routes traffic to frontier models whenever quality scores diverge significantly, enforcing a strict minimum quality floor across all completions.
Balanced (Default)
Identifies a parity band and dispatches prompts to the lowest-cost model inside that quality window, reserving frontier models for complex tasks.
Cost-first
Selects the lowest-priced model that clears a configured threshold for high-volume routine operations, recording detailed token audit logs for every request.
Deploying Weave Router in Local and Production Workflows
Setting up Weave Router requires minimal changes to existing development setups. Teams can utilize a managed service or self-host the Go application within their own cloud infrastructure. In both configurations, developer tools connect via standard environment variables without code modifications.
Fast Installation for Developer CLI Tools
To connect client tools like Claude Code, OpenAI Codex CLI, opencode, or pi to the router, developers run a single command in their terminal:
$ npx -y @workweave/router
The interactive setup detects installed client applications and automatically configures environment variables across Anthropic, OpenAI, and Google formats. Developers generate a bearer key locally that remains on the device unless explicitly exported. For repository-specific configurations, the installer supports scoped flags:
$ npx @workweave/router --scope project
Self-Hosting the Router Stack
Organizations preferring full data control can deploy the open-source Go codebase alongside a local Postgres database. The setup process involves defining upstream provider keys and running the build targets:
# Add upstream provider keys
echo \"OPENROUTER_API_KEY=sk-or-v1-...\" >> .env.local
# Launch Postgres and the router service on port 8080
make full-setup
Once running, the proxy accepts requests across standard API formats. Developers can send an Anthropic-formatted request directly to the local endpoint:
