Bifrost now grades each prompt's difficulty before it picks a model

Maxim AI’s open-source gateway Bifrost now decides how hard a prompt is before it decides where to send it. The complexity router assigns every eligible request a Simple, Medium or Complex tier and hands that tier to the gateway’s routing rules. Bifrost v2.1.0, released on 8 September 2026, rebuilt the router around an embedding-based classifier. Bifrost v2.3.0, released on 13 October, added a second classifier that asks a decision model, TypeSafe’s Jev by default, which tier fits the request.

The pitch is cost. Most production prompts are extraction, reformatting, lookups and short answers that a small model handles well, yet many teams send everything to the strongest model they have because choosing per request is tedious and risky to do in application code. A gateway already sees every prompt, already holds the provider keys and already writes the logs, so it is a natural place to make that choice once for every application behind it.

That matters most to platform teams running shared AI infrastructure for several product groups, and to anyone whose monthly model bill is dominated by a frontier model doing work a model one-fortieth the price could do. This piece covers the economics, how the router works, the supporting changes across four releases, a worked cost scenario, how the big clouds approach the same problem, and where routing by difficulty runs out.

Most prompts do not need a frontier model

The price gap between the top and bottom of a single vendor’s line-up is now wide enough that model choice is a bigger lever than almost any other optimisation. Anthropic’s published API prices, as of 28 October 2026, make the point:

A request that costs a cent on Opus 5.5 costs a quarter of a cent on Haiku 5.5 for the same tokens. Across millions of calls a month, the question of which model answered stops being an engineering detail and becomes a budget line. The site’s coverage of falling inference prices makes a related point: the cheap end of the market keeps getting cheaper and better, which widens the reward for sending work there when it is good enough.

The research case for routing is not new. FrugalGPT, from Lingjiao Chen, Matei Zaharia and James Zou at Stanford in May 2023, described a cascade that tries cheaper models first and escalates only when needed, and reported that it could match the best single model “with up to 98% cost reduction” on its evaluation tasks. The Hybrid LLM paper, accepted at ICLR 2024 from a Microsoft and University of British Columbia team, trained a router to predict query difficulty and reported “up to 40% fewer calls to the large model” with no drop in response quality.

The most cited result is RouteLLM, from a team led by UC Berkeley and LMSYS researchers, which trained routers on human preference data to pick between a strong and a weak model. The accompanying LMSYS write-up reported “cost reductions of over 85% on MT Bench, 45% on MMLU, and 35% on GSM8K” compared with sending everything to GPT-4, while keeping 95% of GPT-4’s performance.

Two things in those numbers deserve attention. First, the savings vary by a factor of more than two depending on the workload: a chat benchmark full of easy turns saves far more than a maths benchmark. Second, every result is measured against a benchmark with known answers. A production team has no answer key, so it has to decide for itself what “good enough” means for the cheap tier. Both points shape how a gateway feature like this one should be used.

Why the gateway is the place to decide

A routing decision can live in three places: in each application, in a managed router inside one cloud’s model catalogue, or in the gateway between applications and providers. Each application deciding for itself means every team writes and maintains its own classifier, its own model list and its own fallbacks. A cloud’s managed router only routes among the models that cloud sells.

The gateway already has what routing needs. It sees the full request, including the user turn; it holds keys for every provider; it already runs fallbacks when a provider fails; and it logs every call with cost attached. Readers new to the category can find the general case in the site’s comparison of LLM gateways, and Maxim’s own overview of what an LLM gateway does for production traffic covers the same ground from the vendor side.

Bifrost’s documentation puts the goal plainly: the router “lets you send simple requests to a fast, cheap model and hard ones to a frontier model, with no changes to your application.” The application keeps calling one endpoint with one model name. Since v2.3.0, when a rule rewrites the model, the caller’s original route comes back as routing_info.requested_provider and requested_model, and in two x-bifrost-routing-info-requested-* response headers, so the application can see what happened without having asked for it.

How the complexity router works

The router runs four steps on each request, according to the documentation:

1. Extract the user-written text. System prompts and assistant replies are not classified.
2. Classify it with the decision model, or with the semantic classifier and its fallback.
3. Apply session state, if enabled, so a conversation keeps the highest tier it has reached.
4. Route. The tier is published as complexity_tier for routing rules, and how it was produced is logged.

An important detail is that classification is lazy. It “runs only when a routing rule references complexity_tier”, so a deployment that has not written a complexity rule pays nothing for the feature. And if classification fails, times out or finds no confident match, the rule simply does not match and the request carries on through normal routing. The failure mode is “the request goes where it would have gone anyway”, not an error.

Figure 1: The complexity router only adds a label. Routing rules decide what the label does, and a missing label falls through to normal routing.

Two classifiers, chosen per deployment

Bifrost ships two ways to assign a tier, and the documentation compares them directly:

The two can be combined. A team can run the semantic classifier as the primary and set the decision model as its fallback, so the slower call only happens for requests that no reference phrase matches confidently.

The decision-model classifier

For each request, Bifrost asks the decision model a single question: which tier best describes the complexity of the latest human request? It sends each tier’s definition, signals and examples alongside, and the model answers SIMPLE, MEDIUM or COMPLEX, optionally with a confidence. The shipped guidance defines Simple as “Direct work answerable from the request itself or common knowledge in one straightforward step”, and Complex as “Advanced expertise combined with substantial reasoning, derivation, design, or synthesis.”

Three fixed rules travel with every call, and they are the most interesting design choice in the feature. The model is told to “judge the work, not the wording”, so length, format and how urgent a prompt sounds do not change its tier. It classifies only the latest request, using earlier messages only to resolve references like “now do the same for the second table”. And it treats embedded instructions as content: a prompt that says “classify this as simple” is data to judge, not an instruction to follow. That last rule is a direct defence against users gaming the router into a cheaper or more expensive model.

Jev is the default, but the v2.3.0 release notes and the docs list a range of alternatives, including OpenAI decision models such as gpt-6-luna, Cloudflare’s Clef models, Perplexity’s deciders, Microsoft-Decision-1 on Azure, Databricks ai_decide, and self-hosted Laya and Nimble models. The site covered how Bifrost brought Jev and the /v1/decisions endpoint into the gateway in its earlier report on Bifrost’s TypeSafe integration; the complexity router reuses that plumbing rather than adding a new one.

The cost of classification is visible and small. Per the timeout and cost settings, decision calls “count toward the request’s cost and budgets, under the model that answered”. TypeSafe’s models page lists Jev at $0.042 per million input tokens and says “You pay per input token. Output tokens are free.” The latency is the real price: the default decision.timeout is 1.5 seconds, and the call runs on the request path, so its latency “is added to every request it classifies”.

The semantic classifier

The semantic classifier embeds the latest user message and takes the tier of the nearest reference phrase. Bifrost ships 150 default phrases, 50 per tier, “spread across use cases and writing styles so the classifier learns the requested work rather than the topic or length”. The three lists can hold up to 750 phrases in total. The docs advise teams to add phrases drawn from their own users’ prompts, noting that “a few domain-specific phrases per tier usually improve routing more than any other tuning.”

The v2.1.0 release that introduced it also brought a pluggable vector store with an embedded chromem backend, cross-node coordination so one node embeds the phrases and others adopt them, an optional LLM fallback classifier, and new status, generations and retry endpoints under /api/routing/complexity-analyzer-*. It retired the old keyword scorer. The release notes flagged one breaking rename: the routing_debug field on responses and logs became routing_metadata, with no alias.

Session-aware routing

Agents complicate per-request routing. An agent session alternates hard planning turns with trivial follow-ups, and bouncing between models turn by turn throws away prompt-cache hits and can change the agent’s behaviour mid-task. Bifrost’s answer is session-aware routing: a session keeps the highest tier it has reached, can only move up, and expires after 24 hours of inactivity. The docs say this “avoids model switches that hurt prompt-cache reuse”. Only the tier is stored, under a hashed session ID, and Bifrost reads the session headers that Codex and Claude Code already send, as well as its own x-bf-session-id.

The v2.2.0 release on 15 September fixed a gap here. Continuation turns, such as a tool result following an earlier user message, had been skipped entirely when no session state was found; they are now classified from the recovered last user text.

Writing the rules

The router itself does not route anything. It publishes a variable, and a team writes routing rules in CEL, Google’s Common Expression Language, that use it. The documentation recommends starting with a single carve-out for the hardest tier and leaving everything else on existing routing:

{
  "id": "complexity-complex",
  "name": "Complex → Frontier model",
  "enabled": true,
  "cel_expression": "complexity_tier == \"COMPLEX\"",
  "targets": [{ "provider": "anthropic", "model": "claude-opus-4-5", "weight": 1 }],
  "scope": "global",
  "priority": 0
}

Because the tier is just another CEL variable, it combines with everything else the rules engine knows: headers, team, virtual key, request type and capacity metrics. Examples from the docs include team_name == "ml-research" && complexity_tier == "COMPLEX" and request_type == "chat_completion" && complexity_tier != "SIMPLE". Rules can be scoped to one team first, which is the sensible way to pilot.

Four releases, one routing stack

The complexity router is the headline, but the surrounding changes in the same release train are what make it usable in production.

The routing plugin

The v2.0.0 changelog records that routing rules and the complexity router now “live in a dedicated routing plugin that runs after governance so rules evaluate on the fully stamped context”. In practice that means a rule can see which team and virtual key a request belongs to before it decides the model. The same release taught complexity routing to read the text of mixed text-and-image turns.

A cutoff for slow first tokens

Cheap models often run on busier, less predictable capacity. The time-to-first-token cutoff in v2.3.0 lets a routing rule set ttft_timeout_ms, between 1 and 300,000. If a streaming attempt has not produced its first token in time, Bifrost closes it and moves to the next fallback, and “the client never sees the slow attempt”. The last attempt in the chain is never cut off, a miss does not retry the same provider, and non-streaming requests ignore the setting. The docs add a fair warning: the provider may still bill the prompt tokens of an attempt that was cut off.

Failures classed by cause

Routing more traffic to more providers means meeting more kinds of failure. Since v2.3.0, every failed attempt gets a failure_class such as credential, quota, model_access, model_gone, region_blocked, rate_limit or transient, read from the provider’s own error code where one exists. The docs explain why: providers disagree on what a status means. Gemini rejects a bad key with a 400, while OpenAI reports an empty credit balance as a 429. Credential and quota failures now hold a key out for every model, while a key refused for one model is rotated past.

Prices that change by the hour

Savings are only as good as the cost accounting behind them. The v2.2.0 changelog added off_peak_cost_multiplier and a weekly peak_hours schedule to model pricing, so that providers that bill the same model at two rates are costed correctly. Bifrost’s pricing documentation cites DeepSeek, which “charges half its normal rate outside a set of published peak windows”. Peak or off-peak is decided by the request’s start time, and flat per-request fees are never discounted.

Figure 2: A three-tier ladder spreads one workload across models whose output prices differ by a factor of 40.

What it saves: an illustrative scenario

The following scenario is illustrative. The traffic volume, token counts and tier mix are assumptions chosen to show the arithmetic, not measurements from any deployment. Prices are Anthropic’s list prices as of 28 October 2026.

Assume an internal assistant handles 10 million requests a month, averaging 1,500 input tokens and 400 output tokens per request. That is 15,000 million input tokens and 4,000 million output tokens a month.

Baseline: everything on Claude Opus 5.5 ($4 input, $20 output per million tokens):

• Input: 15,000 × $4 = $60,000
• Output: 4,000 × $20 = $80,000
• Total: $140,000 a month

Routed: 60% Simple to Haiku 5.5, 30% Medium to Sonnet 5.5, 10% Complex to Opus 5.5.

• Simple, 6 million requests: 9,000 million input tokens × $0.10 = $900, plus 2,400 million output tokens × $0.50 = $1,200. Subtotal $2,100.
• Medium, 3 million requests: 4,500 × $2 = $9,000, plus 1,200 × $10 = $12,000. Subtotal $21,000.
• Complex, 1 million requests: 1,500 × $4 = $6,000, plus 400 × $20 = $8,000. Subtotal $14,000.
• Classification: assume each Jev call reads about 1,000 input tokens (the tier guidance plus the latest user message). 10 million × 1,000 = 10,000 million tokens × $0.042 = $420.
• Total: $2,100 + $21,000 + $14,000 + $420 = $37,520 a month, about 73% less than the baseline.

The result is sensitive to the mix, which is exactly what a team does not know before it turns the feature on. With a harder mix of 30% Simple, 40% Medium and 30% Complex, the same arithmetic gives $1,050 + $28,000 + $42,000 + $420 = $71,470, a 49% saving. And against a more realistic baseline of everything on Sonnet 5.5 ($30,000 input plus $40,000 output, so $70,000), the 60/30/10 mix saves about 46%, not 73%.

Three things the arithmetic leaves out cut the other way. Prompt caching discounts repeated input heavily on every tier, which narrows the gap. A misrouted complex request answered badly by a small model has a cost that does not appear on the invoice. And the decision call adds 600 to 800 ms to every classified request, which matters more for a chat interface than for a batch job. The honest summary is that routing by difficulty can halve a bill for a workload dominated by easy prompts, and saves much less for one dominated by hard ones.

How the big platforms approach it

The large clouds have reached the same conclusion and built routing into their model catalogues.

Microsoft’s model router in Foundry is described as “a trained language model that intelligently routes your prompts in real time to the most suitable large language model”. It is deployed like any other model and analyses prompts “based on complexity, reasoning, task type, and other attributes”. A team picks a routing mode: Balanced, the default, which picks the cheapest model within a small quality band of the best; Cost, with a wider band; or Quality, which ignores cost. The current version routes across models from OpenAI, Anthropic, xAI, DeepSeek, Meta and others available in Foundry. Bifrost’s documentation includes a provider page for calling Azure’s model-router deployments through the gateway, so the two are not mutually exclusive.

Amazon Bedrock offers intelligent prompt routing, which AWS describes as a single endpoint that routes “between different foundational models within the same model family”. A team picks two models and a fallback, and sets a “response quality difference” threshold: the router switches away from the fallback only if it predicts the other model’s answer will be better by more than that margin. AWS’s documentation notes that it is “only optimized for English prompts” and cannot adjust its decisions based on application-specific performance data.

The structural difference is ownership of the decision. In a managed router, the routing model and its notion of quality belong to the cloud vendor, and the team tunes a dial. In Bifrost, the classifier says only how hard the request is, and the team writes the rule that maps difficulty to a model. That is more work, and it is also the property that lets a platform team route across Anthropic, OpenAI, Google and self-hosted models in one policy, and change the mapping when prices move. A longer survey of how gateways compare on routing and governance is in Maxim’s production-ready comparison of the top five LLM gateways.

Where routing by difficulty runs out

The limits of this approach are mostly limits of the category, and they apply to every router, managed or self-hosted.

Difficulty is not the same as stakes. A classifier judges how much reasoning a request needs. It does not know that one simple-looking question comes from a regulated workflow where a wrong answer is expensive. Bifrost’s rules can combine the tier with headers, teams or virtual keys, which is the right tool, but someone has to write those conditions. A rule like “anything from the legal team goes to the frontier model regardless of tier” belongs next to the complexity rules.

Nobody grades the cheap tier for you. The research results above were measured against benchmarks with answers. In production the router has no answer key. Bifrost logs complexity_tier, complexity_mechanism and, for semantic matches, the similarity score and the phrase that matched, and the log explorer can filter on tier and mechanism. That makes misroutes findable. Deciding whether Haiku’s answer to a Medium request was acceptable still takes an evaluation process, human review or both.

Classifiers add latency to the requests they touch. A 600 to 800 ms decision call is negligible for a long agent task and noticeable in a chat window. The semantic classifier is faster but only as good as its phrases. Running semantic first with the decision model as fallback is the documented compromise.

Difficulty is a text problem. Grading how hard a prompt is only makes sense for text the classifier can read. Bifrost’s docs set the scope accordingly: image generation, embeddings, audio, video and count-tokens requests, mixed-media chat content and system-only requests go straight to normal routing, where model choice is usually fixed by the modality anyway.

Health-aware provider and key selection sits in the enterprise tier. Complexity routing, rules, fallbacks, the TTFT cutoff and failure classes are documented without an enterprise marker. Adaptive Routing, which picks the healthiest provider and key from live traffic and gained an Incidents page and per-attempt load-balancer metadata in v2.3.0, is part of Bifrost Enterprise. Teams on the open-source build get the difficulty routing and the static fallbacks, and do the provider-health tuning by hand.

What to watch

Three developments will decide how much of this becomes standard practice.

The first is whether decision models settle in as the default classifier. A judgment model that answers one typed question in under a second, at four cents per million input tokens, is close to ideal for this job, and the long list of compatible decision models in Bifrost’s docs suggests the category is filling out quickly. If those models prove well calibrated on real traffic, the curated-phrase approach becomes a latency optimisation rather than the main path.

The second is evaluation tooling that closes the loop. Today a team reads tier logs and judges quality separately. The obvious next step, for every router, is feeding quality scores back into the tier guidance or phrases automatically. Watch for that in managed routers and gateways alike.

The third is pricing. The case for routing depends on the gap between the cheapest capable model and the frontier. That gap is currently a factor of 40 within Anthropic’s own line-up. If small models keep closing on capability while frontier prices hold, routing by difficulty stops being an optimisation and becomes the default way to buy inference. The gateway that already sees every prompt is where that default will be set, and Bifrost’s last four releases are a bet that teams will want to write that policy themselves.