Intelligent Routing

Overview

Intelligent Routing (also known as model routing) decides which model a request finally uses before it is sent upstream. Callers keep writing a model name as usual; the gateway rewrites the model according to rules the administrator configures, and switches to a backup model automatically when the primary model fails.

Typical uses:

Scenario	Description
Auto tiering	Callers always send one virtual model name (such as og-auto), and the gateway picks a simple, medium, or complex tier model by request complexity: cheaper models for simple questions, stronger models for complex tasks
Model replacement and canary	Move all or a percentage of a model's requests to another model, with no code change on the caller side
Failover	When a model's call ultimately fails, switch to backup models in order, reducing failures caused by a single unavailable model
Conditional routing	Send matching requests to a specific model, for example long inputs to a long-context model, requests with images to a vision model, tool-calling requests to a model good at function calling, or a given tenant or client (such as Claude Code) to a dedicated model

Intelligent Routing chooses which model to use based on the request content, such as task complexity; provider routing chooses a provider for the same model based on runtime signals such as latency, price, and availability, which is traffic management. They are two different layers:

	Intelligent Routing (this page)	Provider routing
Decides	Which model to use	Which provider serves the same model
On failure	Retries with a different model	Retries among providers of the same model
Billing	Billed by the rewritten model	Does not change the billed model
Order	First	Second: after Intelligent Routing picks a model, every request still goes through provider routing

How It Works

Concept	Description
Rule	A set of match conditions plus an action. Rules are evaluated by priority, and the first matching rule wins
Target model	The model used once a rule matches. Several targets with weights are picked at random by proportion
Backup model	Models tried in order when the primary model ultimately fails
Virtual model	A model name that exists only in the gateway (such as og-auto); it does not need to exist at, or be priced by, any provider
Complexity tier	SIMPLE, MEDIUM, or COMPLEX, judged by the gateway from the request content

Requests that match no rule are forwarded unchanged. If a rule itself errors (for example, a malformed condition), only that rule is skipped; the request is never blocked.

Prerequisites

• The administrator has enabled the Intelligent Routing plugin on the target Gate instance and configured routing rules.
• Every target model and backup model used in the rules has been onboarded and priced in Providers & Models, and the caller's group has an available provider for it.
• If the caller's API Key restricts available models, the list must include those target and backup models; see Billing and Permissions.

Supported Endpoints

Endpoint type	Path	Where the model comes from
OpenAI format	POST /v1/chat/completions	Request body model, including streaming
OpenAI Responses format	POST /v1/responses	Request body model, including streaming
Anthropic format	POST /v1/messages	Request body model, including streaming
Legacy completions	POST /v1/completions	Request body model, including streaming
Gemini native format	POST /v1beta/models/{model}:generateContent, :streamGenerateContent	The model in the URL; model in the request body is ignored

Administrator Setup

The management UI is a preview

The scenario setup, dry-run, and routing monitor UI described in this section is currently a preview, available under Plugins → Intelligent Routing in the OneGateway console. Administrators can also edit Gate's rule definition file directly, or use Gate's rule management API (Chinese) to configure and dry-run rules.

Common uses can be configured as scenarios by filling in model names. The effects below describe the rules a scenario generates; when you write the rule file yourself, behavior follows the rules you write:

Scenario	What you fill in	Effect
Virtual model · tiered by complexity	A virtual model name and the model for each of the simple, medium, and complex tiers	Besides the three tier rules, a catch-all rule is generated: when complexity cannot be determined (for example, an image-only request), the medium tier model is used
Virtual model · fixed mapping	A virtual model name and the model it maps to; you can add several models with shares	Callers send the virtual model name (such as og-coding), and the gateway always rewrites it to the given real model
Replacement and split	The original model and the model to replace it with; you can add several models with shares	One model replaces all traffic; several models split traffic by share for a canary or load balancing, and the original model can be included to keep part of the traffic
Failover	The primary model and backup models in order	The model is not changed; switching happens only on failure
Model alias	Some alias names and the model they map to	Aliases are first rewritten to the target model, then the target model's other scenarios apply
Conditional routing	Which models it applies to (wildcards allowed; empty means all models), the conditions, and the model to switch to	Conditions can be combined: long context (estimated tokens reach a threshold), contains images, has tools, tenant, request header (such as recognizing a client by user-agent), reasoning effort, protocol, streaming, complexity tier

Every scenario except failover can optionally carry backup models; the three tiers of auto tiering can share one list or each have their own. When needed, a single scenario can also customize its switching conditions (which status codes and which additional error codes switch, and the maximum attempts), while other scenarios keep the global settings; see Automatic Switching on Failure.

Virtual models are managed on their own page, and each card also lists the conditional routing that applies to it (for example, requests with images going to a vision model), which you can add there directly. Other scenarios and hand-written rules share one list on the rules page, in matching order, and the first one that matches wins: conditional routing is placed before other rules for the same model by default, and you use the up and down buttons at the start of a row to reorder, with priorities assigned automatically on save. A scenario placed after one that takes all of the same requests can never match, and the UI flags it.

Each scenario consists of one or more rules carrying a scenario tag (scenario, used only to group rules in the UI; it does not affect routing). Rules edited by hand in the raw file so that they no longer fit the scenario template are shown as custom rules and can only be changed in the raw file.

Before saving, you can dry-run a sample request to see which rule matches, which model it becomes, and which tier it is judged as. A dry run does not forward the request to the target model and incurs no generation cost for it; however, with LLM or embedding classification enabled, the configured classification service is still called, which may incur platform-side cost, and the sample text is sent to that service.

Rules take effect as soon as they are saved, with no gateway restart. For more complex conditions (combining request headers, tenant, protocol, input length, and so on, or chained rules), write them in the raw rule file; see the technical reference (Chinese) for fields and examples.

Everyday and platform configuration

The admin UI splits configuration into two levels and keeps them apart:

Level	For	Includes
Everyday	Organization (company) administrators who understand AI tokens and are comfortable with the platform	Virtual models, replacement and traffic split, failover; conditional routing built from long context, images, tools, and client; enabling, ordering, and backup models of scenarios; picking one of the prepared classifiers; dry runs
Platform	OneGateway platform administrators	Classifiers, the global switching policy, the allowed model range, the plugin switch and runtime settings; per-scenario switching conditions; model aliases; routing by tenant or reasoning effort; custom rules and raw editing; alert thresholds

The viewpoint button at the top right of the admin UI switches between the organization administrator view and the platform administrator view: the organization view shows only everyday configuration, while the platform view shows everything, with platform configuration in the "Platform" menu group or in collapsed sections labeled "Platform" on everyday pages. Mistakes in platform configuration affect every request, so platform administrators should maintain it.

Alerts

Alerts appear at the top of the page in these cases, with thresholds set in the platform settings:

• A scenario's failure rate from rate limiting or upstream faults over the last hour is high; consider adding backup models.
• The share of requests switched to backup models is high; the primary model may be unstable.
• Among requests that need a complexity tier, too many get none; platform administrators should check the classifier.

Caller-side failures such as invalid keys, insufficient quota, or unpriced models cannot be fixed by switching models, so they are not counted in failure rates and are shown separately on the overview.

How Complexity Is Judged

Complexity is judged by a classifier that the platform configures, in one of these ways:

Method	How it works
Local scoring	The request is scored locally from signals such as length, code terms, reasoning phrases, technical terms, and greetings or simple questions, then picks the tier from the score; no model is called, so it adds almost no latency and no cost
Local first	Score locally first; simple requests with clear signals, such as greetings and short questions, are decided right away, and only uncertain ones go to LLM classification
LLM classification	The user's request is handed to a small platform-provided model, which picks the tier
Embedding exemplar matching	The request is compared with preset example sentences for each tier and joins the closest tier

LLM classification and embedding exemplar matching use the platform's own model and credentials, never the caller's API Key or model, and each judgment adds one call to the classification model. Local first calls it only when the local score is uncertain. The administrator can set another method as a fallback that is used when the primary one fails; a local scoring fallback always produces a tier.

Local scoring reads only the wording, so it is coarser than a model: on its own it got about half of a labeled test set right. Local first is recommended instead, saving about 40% of classification calls while staying close to LLM accuracy. It is a routing hint only: callers can influence the tier through their wording.

Administrators can also configure two independent LLM classification models. A successful primary result never calls the backup. Only a timeout, request failure, invalid tier, or full concurrency limit triggers the backup. Every new classification starts with the primary again, so recovery automatically restores primary use. If both fail, the request gets no tier. Backup tiers apply only to the current request and are not saved in sessions. Separate providers, endpoints, and credentials can reduce shared failures.

Classification runs before the business model call, so its wait adds directly to streaming time to first byte and non-streaming response time. Every new classification tries the primary first; a consistently timing-out primary makes each request wait for that timeout before using the backup. Administrators should choose a short primary timeout based on measured latency and budget for credential validation and both classification attempts.

No complexity judging without a classifier

If the administrator has not configured a classifier, or the classification model is temporarily unavailable or times out with no fallback, the request gets no tier: rules that split by tier do not match, and the request is handled by the rules after them. Auto tiering scenarios generate a catch-all rule that uses the medium tier model in this case.

By default only the most recent message written by the user is considered; administrators can widen this to the last several messages with message_window. System prompts, assistant replies, and tool results are always excluded, and tags that tools such as Claude Code add automatically are stripped.

Different virtual models can use different classifiers: administrators can create additional "named classifiers" (for example, local scoring for a chat entry point and an LLM for a coding entry point) and have a virtual model use one of them. Virtual models without a choice use the default classifier. Session tiers are kept separately per virtual model's classifier, so one never affects another.

Differences, parameters, and accuracy of each method are covered in complexity tiers (Chinese).

When complexity is judged

Rules are evaluated in priority order, and complexity is judged the first time a condition needs it, at most once per request for each classifier; if chained rules use different named classifiers, each one judges once and the costs add up. If a rule unrelated to complexity matches before that, no judging happens and there is no overhead; but once judging has happened, its cost is incurred even if the rule that finally matches does not depend on complexity.

Session Tiers

When enabled, the tier within one session only goes up, never down: once an earlier turn of a long conversation is judged complex, later short follow-ups keep using the complex tier model, avoiding back-and-forth model switches that invalidate the provider's context cache. A tier is recorded for the session only when the request succeeds and the tier comes from the configured classifier; a fallback judgment used because the classifier failed applies to that request only. Judging starts over after the session has been idle past its expiry, or after the administrator changes the primary judging method (a different classification model, prompt, or exemplars); adding, removing, or tuning only the fallback method leaves existing sessions untouched. Sessions are identified by the hcd-og-session-id request header, and Claude Code's built-in session header is recognized too; different API Keys never share a session even with the same session ID.

Quick Verification

Make sure an auto tiering scenario is configured (using the virtual model og-auto as an example). Replace the address and API Key with real values and send a simple request:

curl -sS -i 'https://your-gateway.example.com/v1/chat/completions' \
  -H 'Authorization: Bearer <ONEGATEWAY_API_KEY>' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "og-auto",
    "messages": [{"role": "user", "content": "Hi, how is the weather today?"}]
  }'

Then keep everything else the same, swap content, and verify in turn:

Check	content	Expected response headers
Simple request	Hi, how is the weather today?	og-route: ROUTED, og-route-tier: SIMPLE, og-route-model is the simple tier model
Medium request	Help me analyze the error in this log	og-route-tier: MEDIUM, og-route-model is the medium tier model
Complex request	A long prompt asking to design a system architecture and analyze trade-offs	og-route-tier: COMPLEX, og-route-model is the complex tier model

Streaming requests ("stream": true) return the same headers, visible before the first piece of content. Use og-route-model to see which model Intelligent Routing chose. The model in the response body is the model name reported by the provider, which may differ from og-route-model when the provider maps models or uses versioned names.

Billing and Permissions

Billing follows the routed model: whichever model og-auto is rewritten to (that is, og-route-model) is charged at that model's platform price, and the provider side sees and records the rewritten model. The virtual model name itself needs no price. The billing details in the request logs are authoritative for the actual cost.

API Keys with a model allowlist

The provider side checks an API Key's model allowlist against the rewritten model. Adding only og-auto to the allowlist does not work: every target and backup model the rules may use must be included, otherwise the request returns 403, and a 403 does not trigger a switch.

When switching to a backup model, the failed attempt is not charged; only the final successful attempt is billed.

Automatic Switching on Failure

When the matching rule has backup models, the gateway tries them in order after the primary model ultimately fails (provider routing has already retried and still failed).

Primary model returns	Switch?	Why
503 no available provider, 429 rate limited, 500 / 502 / 503, 529 provider overloaded, connection failure	Switch	The model is temporarily unavailable; another model may succeed
504 / 524 timeout	No by default	Switching would multiply the wait time; the administrator can enable it
400 unpriced, 403 no access, insufficient quota	No	Another model would not help, and switching could bypass permissions
Sensitive-word rejection	No	The same content would be rejected by other models too

• At most 3 attempts by default (including the first).
• These are the global defaults. A single scenario can customize its switching conditions, for example counting 504 as switchable for a long-context scenario, or switching on a context-length error code. Only that scenario is affected; insufficient quota, unpriced models, and sensitive-word rejections never switch.
• Streaming requests switch only before output starts; once content has started streaming, an upstream interruption does not trigger another model.
• If all backup models also fail, the error from the last attempt is returned.
• Requests that switched models are not written to the Semantic Cache.

Observing Results and Troubleshooting

Start with the response headers:

Header	Meaning
og-route	PASS no rule matched; ROUTED a rule matched; FALLBACK a switch happened (see the status code for the outcome); BYPASS the caller skipped routing; ERROR routing itself errored and the request was forwarded with the original model
og-route-rule	ID of the matching rule (only the ID, never the conditions)
og-route-model	The model sent to the provider after routing (the last attempted model if a switch happened), which is also the billed model
og-route-requested-model	The model the caller originally requested; present only when the model was rewritten
og-route-attempts	Number of upstream attempts (including the first); greater than 1 means a switch happened
og-route-tier	Complexity tier; present only when complexity was judged

With log storage enabled, administrators can view the routing outcome distribution, complexity tier distribution, and the "requested model → actual model" flow by time window in the routing monitor, and drill into individual requests.

The model in request logs is the routed model

The models in request logs and the dashboard are the models actually called after routing; virtual model names such as og-auto never appear there. To check the model originally requested, use the routing monitor or the og-route-requested-model response header.

Troubleshooting by Symptom

Symptom	Check first
No og-route response header	Whether the plugin is enabled, whether the endpoint is supported, and whether an intermediate proxy keeps response headers
og-route: PASS and the model is unchanged	Whether the model name exactly matches the rule (gpt-4o does not match gpt-4o-mini), whether the rule is enabled, and whether it was saved successfully
403 returned	Whether the API Key's model allowlist includes the model shown in og-route-model
400 unpriced returned	Whether the target or backup model is priced
Frequent FALLBACK	Whether the primary model has an available provider in the current group; a group misconfiguration looks like "always switching"
Tier differs from expectation	Use a dry run to see the basis; adjust the classification model's guidance or exemplar sentences, or switch to the other method
No og-route-tier, requests always take the catch-all rule	Whether a classification model is configured and reachable, and whether the caller's API Key is valid (by default the classification model is called only for API Keys that pass validation; the primary and fallback methods each follow the administrator's own validation setting)
Rule changed but not in effect	Whether saving reported an error; a broken rule keeps the gateway on the previous valid rules

Known Limits

• Intelligent Routing only decides the model; it does not choose providers or API Keys, nor route by quota or rate-limit usage.
• No time-to-first-token switching: a streaming request that has started output is not moved to another model.
• Backup model compatibility is the administrator's responsibility: switching only changes the model name and does not adapt the request to the backup model's context length, tool calling, or other capabilities.
• Session tiers are kept by the service that handles the request. When several copies of the service are deployed they do not share session tiers, so a request routed to a different copy is judged again.

Per-Request Control

Request header	Effect
hcd-og-route: off	Skip routing for this request and forward the original model. Ignored by default; the administrator must enable "allow callers to skip routing". Routing may carry cost or compliance policy, so evaluate before enabling it
hcd-og-session-id	Session ID used for session tiers
hcd-og-tenant-id	Used by tenant-based rules. Callers can set this header freely, so it is only a traffic-splitting device, never a security or billing boundary

Relationship with Other Capabilities

• Guardrails: requests pass Guardrails before routing, so blocked requests incur no complexity judging, and the judging sees already-redacted text.
• Semantic Cache: routing runs before the Semantic Cache lookup; the cache is separated by the rewritten model, so answers from different models never hit each other, and requests that switched models are not cached.
• Call logs: logging wraps routing, so rewritten and switched requests are recorded with the requested model, actual model, matching rule, attempts, and complexity tier.

FAQ

What do callers need to change?

Only change the model name in the request to the virtual model name the administrator provides (such as og-auto). Endpoints, request formats, and SDKs stay the same. Replacement or failover rules on existing models are completely invisible to callers.

Why is og-auto missing from the available models list?

Virtual models exist only in the gateway's routing rules and map to no provider, so they do not appear in Available Models. You can use them directly in requests; ask the administrator which virtual model names are available.

How much latency does Intelligent Routing add?

Rule matching runs locally in the gateway, so the overhead is usually negligible. When a complexity tier is needed, local scoring adds almost no latency; the other methods add one call to the classification model per judgment (local first calls it only when uncertain), whose latency depends on where the classification service runs and the network (tens to about a hundred milliseconds in a test with an embedding model on the internal network); a caller credential check may also run first (its result is cached). On timeout the request gets no tier and is handled by the later rules. When a switch happens, total time includes the failed attempt.

Where is my content sent for complexity judging?

Text is sent only when a complexity tier is needed and a classification model is called; local scoring sends nothing, and local first sends only the requests it is uncertain about. The user message text being judged (the latest one by default; administrators can widen it to the last several with message_window) is sent to the configured classification model (which can be a model the platform hosts itself). System prompts, assistant replies, and tool results are not sent. Classification calls are paid by the platform and do not count against the caller's quota.

Why does the same question sometimes use model A and sometimes model B?

Three possible reasons: the rule splits traffic by percentage, complexity judging is sensitive to wording, or the primary model failed and a backup model was used. The og-route-rule, og-route-tier, and og-route-attempts headers tell them apart.

Quick Answers

Question	Answer
Do rule changes need a gateway restart?	No, they take effect as soon as they are saved
Can a broken rule make requests fail?	No, the gateway keeps the previous valid rules; a single rule whose condition errors is just skipped
Are model names case-sensitive?	Rule matching is case-insensitive but must match the whole string
Can routing apply to only some tenants?	Yes, but the tenant header can be forged by callers, so it is only a traffic-splitting device; see Per-Request Control
Fields and advanced configuration	See the Intelligent Routing technical reference (currently available in Chinese only)