Routing and failover

The same model is usually served by several providers. When a request does not name one, the platform produces an ordered sequence of candidates and tries them in turn until one succeeds. This page explains how that sequence is built and where you can influence it.

Routing has two steps#

The two never interfere: ordering can never widen the pool, and admission does not decide order. That distinction explains most cases of "why didn't this go to the provider I wanted".

Admission: who enters the pool#

A model-provider price enters the candidate pool only if all of the following hold:

• The model is in the key's model allowlist (an empty allowlist means no restriction)
• The provider is in the key's and account's pinned provider list (an empty list means no restriction)
• Its supply tier is within the effective supply-tier scope
• Its context limit can fit this request
• It supports the protocol used for this request
• The provider and that price are in a routable state

When the pool is empty the request returns an error rather than falling back to a provider outside the scope. This is deliberate: silently widening a constraint means the limit you believe is in force is not.

⚠️ Context capacity is compared using an estimate, and requests are admitted when the estimate falls short. Requests extremely close to the limit may therefore still be rejected upstream.

Ordering: who is tried first#

Sorting strategies#

Tie tolerance decides how "hard" a strategy is#

A strategy expresses a preference, not a guarantee. Ordering first computes a total order by the metric, then groups into bands by tolerance — candidates within the tolerance of each other form one band, and within a band traffic is again split by weighted random.

The tolerances are not conservative padding; they match the real noise level of each metric:

• 20% for latency and throughput — measurement noise is already of that magnitude, and ordering on a 5% difference is false precision. This also explains a common question: after selecting "latency first", traffic is still spread out, because providers performing similarly are judged to be in the same band.
• 2% for price — price is an exact value, and the tolerance only absorbs rounding in promotional multipliers.
• 1 percentage point for reliability — success rate is already a ratio, and a relative tolerance would be unduly amplified at the low end (0.99 versus 0.98 differs by 1%, 0.10 versus 0.09 by 11%, yet both differ by one percentage point).

price compares a blended rate#

The price strategy orders by a blended unit price of input × 3 + output × 1, not by any single item. A provider with cheap output but expensive input may therefore not come first.

Ordering uses the listed price you actually pay, including promotions and account-specific discounts — not our wholesale cost.

What statistical strategies do without enough data#

latency, throughput, and reliability rely on measurements from the last 24 hours, and candidates with too few samples are not judged:

• Some candidates lack samples — they are inserted at the median rather than placed last. No data does not mean poor performance, and placing them last would make samples even harder to obtain, creating a self-fulfilling judgment
• All candidates lack samples — the whole pass degrades to balanced. This is normal for a newly launched model or provider, and resolves itself as data accumulates

All three exclude client errors from the denominator: failures caused by a malformed request are not counted against the provider.

Session affinity: stable ordering within a session#

If a multi-turn conversation switches provider on every turn, the upstream prompt cache will never hit — input tokens are billed at full price, where a cache hit typically costs about a tenth of that, and time to first token is higher too.

The random source behind weighted selection is therefore derived from a session key: the same session produces the same candidate order on every turn, while different sessions remain uncorrelated so traffic overall is still split by weight.

The session key is determined in this order:

Pass prompt_cache_key explicitly. Automatic derivation depends on the start of the conversation staying unchanged, whereas an explicit key does not — it is more reliable for long sessions and branching conversations. This is the highest-value single change for cost optimization.

Preferred providers#

Beyond the sorting strategy, a set of preferred providers can be named. The pool is split into two groups — preferred first, the rest second — and each group is ordered by the sorting strategy.

Preferred providers do not change the pool, so they can never make a model uncallable: if all preferred providers are unavailable, the request still reaches the others.

When the preferred list matches none of the candidates, or all of them, grouping is dropped and ordering falls back to a single pass.

Circuit breaking wins over everything#

Links that fail repeatedly are tripped and moved to the end of the sequence after ordering. In other words, circuit breaking can override your preference: a preference says "try this one first under normal conditions", while a trip says "this one just failed repeatedly".

If every link is cooling down, the request returns an error stating that explicitly.

Failover#

Whether a failed attempt moves to the next candidate depends on the error type:

• Transferable: connection failures, timeouts, upstream 429, upstream 5xx
• Not transferable: parameter errors, authentication failures, content policy refusals, context overflow, and other client-attributed errors

Non-transferable errors return immediately. Another provider would give the same result, and continuing only adds latency. For which error types trigger failover, see the error table in the API reference.

Attempts that fail during a transfer are not billed; the upstream cost is absorbed by the platform. The full attempt chain is recorded under "routing attempts" in the call log, showing which providers were tried and why each failed.

Only upstream-attributed failures count toward circuit breaking. Client errors never remove a link — otherwise a script with a malformed request could take down every healthy link.

Four configuration layers#

Sorting strategy and candidate scope can be set in four places, the closer to the request, the more specific:

The workspace layer exists only for organization workspaces. A personal workspace has nothing above it but the account settings.

The two kinds of configuration merge differently, and that is deliberate:

The preference axis replaces wholesale rather than merging field by field. Otherwise you get "this key overrode half the configuration while the other half drifts with the global default", when the administrator believed they had only changed a default.

The constraint axis intersects so that a floor set at the account level cannot be bypassed by a key.

Intersecting supply-tier scopes#

The three values are not a total order — "official quality only" and "value only" do not contain each other, and their intersection is empty. Intersection is applied layer by layer (account ∩ workspace ∩ key); the table below holds for any two adjacent layers:

Conflicting combinations are rejected on save, never left to runtime. Narrowing an upper layer is likewise rejected — and names the affected workspaces or keys — when it would leave something below with no candidate at all.

Once an upper layer pins it, the layer below shows the result#

When an upper layer has already fixed the supply-tier scope, the layer below no longer offers a dropdown: every legal option would produce the same effective value. The interface shows that value directly and states which layer fixed it.

Pinned providers work the same way — once an upper layer names them, only those remain as candidates below.

Request-level suffixes#

Suffixes follow the model ID, separated by a colon.

Pinning a provider#

{ "model": "anthropic/claude-opus-5:anthropic" }

Pinning reduces the pool to that provider alone; if it is unavailable the request fails outright with no switching. Pin when you need determinism; leave routing automatic when you need availability.

If the pinned provider is not in the key's allowed list, the request returns invalid_request — retrying will not change the outcome.

Setting the sorting strategy#

{ "model": "anthropic/claude-opus-5:@price" }

@ is the fixed prefix for strategy suffixes, distinguishing them from provider identifiers.

⚠️ Strategy suffixes are locked by the key by default. To let callers set a strategy per request, enable "allow request override" in the key's settings. Without it, using the suffix returns an error rather than being silently ignored.

⚠️ The two suffixes cannot be combined. Pinning a provider leaves a single candidate, so ordering is meaningless and such requests return an error.

A request suffix can only change ordering, never widen scope. Supply-tier scope and pinned providers are decided by the configuration layers and cannot be overridden by a request.

When the model ID itself contains a colon#

Some canonical names and aliases contain colons, such as llama3.1:70b or Bedrock-style ...-v2:0. Resolution looks up the complete string in the alias table first and treats a hit as a full model name; only a miss falls back to splitting on the last colon. These names can be used as-is, with no escaping.

Common questions#

Why didn't the request go to the cheapest provider#

In order of likelihood: that provider is outside the effective scope (supply tier or pinned provider list); its context limit cannot fit the request; it does not support the protocol in use; it is cooling down after circuit breaking; or the current sorting strategy is not price. "Routing attempts" in the call log shows which providers were actually tried.

I changed the account default strategy — why did some keys not change#

Sorting strategy is on the preference axis, where the most specific setting wins. Keys that have their own strategy do not follow the account default; change them individually, or set them back to "follow workspace default".

Does failover charge twice#

No. Attempts that fail during a transfer are not billed; only the successful one incurs a charge.

What happens if the upstream fails mid-stream#

Content already sent to the client is not rolled back. The request is billed for the usage already produced and marked as upstream-truncated in the log.

Related#

• Providers and supply tiers — tier definitions and how metrics are measured
• API keys — configuring routing and provider scope on a key
• Errors and retries — which error types trigger failover