LLM model routing: the cheapest model that can do the job

LLM model routing means classifying each request before you spend anything on it, then sending it to the model that actually fits — for quality, for latency, and for cost. The alternative, which is what most applications do, is to pick one frontier model and send it everything, including “thanks” and “ok cool”.

The default is a pricing decision you didn’t make

If your app has one OPENAI_MODEL in its environment, you’ve decided that every request deserves your most expensive model. That’s defensible for a coding assistant. It’s indefensible for a product where a large share of traffic is greetings, acknowledgements, and one-line clarifications.

The uncomfortable part is that you can’t see the waste in your bill. It’s a single line item. The trivial requests and the genuinely hard ones are averaged together, so the fix never gets prioritised.

What a classifier actually looks at

Routing needs a fast, cheap read of the request before the expensive call. Four signals get you most of the way:

• Intent — is this a task, a question, a follow-up, or social filler?
• Complexity — does answering require multi-step reasoning, or is it a lookup or a rewrite?
• Context size — a 200-page document rules out short-context models regardless of everything else.
• Cost ceiling — what the user’s plan or the operation’s budget allows.

The classifier itself must be dramatically cheaper than what it’s protecting, or you’ve just added a tax to every request. A small local model, or a heuristic pass with a small model only for ambiguous cases, is the shape that works.

Route, don’t just downgrade

The naive version is a quality ladder: hard requests get the big model, easy ones get the small model. That’s better than nothing but it misses most of the value, because models aren’t ranked on a single axis.

They have shapes. One is strong at long-form drafting, another at code, another at multimodal input, another is simply very fast and very cheap. “Cheapest that can do the job” means matching the shape of the request to the shape of the model — not walking down a price list.

This is why routing beats a static choice even when cost isn’t your concern: you get a better answer on long-form drafting from the model that’s good at long-form drafting.

Fallback chains are the part that pays for itself

Providers degrade. They rate-limit you, they have incidents, they deprecate model versions with more notice than you read.

If your app names one model, every one of those events is an outage. If routing already exists, the fallback is nearly free: when the first choice fails or times out, the run re-routes to the next model in the chain and continues.

Worth designing carefully:

• Fall back on shape, not price. Substituting a short-context model into a long-context request produces a confident, truncated, wrong answer — worse than an error.
• Make it visible. A run that silently completed on a weaker model is a debugging nightmare later. Log which model served each step.
• Cap the retries. A chain that walks every provider on a malformed request burns budget on something that was never going to succeed.

The prompt you shouldn’t send at all

There’s a tier below “cheapest model”, and it’s the one people skip: some requests don’t need a model.

“Thanks”, “ok”, “got it”, “hi” — this class is bigger than it looks. Measured across all 656 conversational requests in our Telegram beta (March–May 2026), 56% never reached a model at all: the local template path answered them in single-digit milliseconds, with no API call, no tokens, and no network round-trip. The remaining 44% routed across eleven models, for a recorded model spend of $1.66 per thousand requests over the whole surface. That’s not routing so much as a filter in front of it, and it’s the single largest saving available because the cost isn’t reduced, it’s zero.

Two honest caveats. First, this only works if the classifier is genuinely reliable on that narrow class — misfiling a real request as filler is a much worse failure than overspending on it, so the bar should be high and the fallback should always be “send it to a model”. Second, your own savings depend entirely on your traffic mix; our 56% came from a chat-heavy beta, and a product where every message is a substantive task will save nothing here. That’s fine — measure your own share before assuming ours.

Where this leaves the bill

Routing doesn’t make one request cheaper. It stops you paying frontier prices for work that never needed them, and it makes provider outages a re-route instead of an incident.

Velaris classifies every prompt before it costs anything and routes it accordingly — see the router in the live demo, or read how it picks a tool once it has a model.