---
head:
- - meta
- name: description
content: Learn how OneGateway Intelligent Routing works — use a virtual model to pick a model by request complexity, roll out a replacement model by percentage, fail over to backup models automatically — plus billing, permissions, and troubleshooting.
- - meta
- name: keywords
content: OneGateway, Intelligent Routing, model routing, virtual model, og-auto, complexity tiers, canary, fallback, failover
---
# Intelligent Routing
## Overview
Intelligent Routing (also known as model routing) decides which model a request finally uses **before it is sent upstream**. Callers keep writing a model name as usual; the gateway rewrites the model according to rules the administrator configures, and switches to a backup model automatically when the primary model fails.
Typical uses:
| Scenario | Description |
| --- | --- |
| Auto tiering | Callers always send one virtual model name (such as `og-auto`), and the gateway picks a simple, medium, or complex tier model by request complexity: cheaper models for simple questions, stronger models for complex tasks |
| Model replacement and canary | Move all or a percentage of a model's requests to another model, with no code change on the caller side |
| Failover | When a model's call ultimately fails, switch to backup models in order, reducing failures caused by a single unavailable model |
| Conditional routing | Send matching requests to a specific model, for example long inputs to a long-context model, requests with images to a vision model, tool-calling requests to a model good at function calling, or a given tenant or client (such as Claude Code) to a dedicated model |
Intelligent Routing chooses **which model** to use based on the request content, such as task complexity; [provider routing](/en/admin/providers-and-models#model-routing) chooses a provider for the same model based on runtime signals such as latency, price, and availability, which is traffic management. They are two different layers:
| | Intelligent Routing (this page) | Provider routing |
| --- | --- | --- |
| Decides | **Which model** to use | **Which provider** serves the same model |
| On failure | Retries with a different model | Retries among providers of the same model |
| Billing | Billed by the rewritten model | Does not change the billed model |
| Order | First | Second: after Intelligent Routing picks a model, every request still goes through provider routing |
## How It Works
```mermaid
flowchart LR
C["Caller
model: og-auto"] --> R["Intelligent Routing
picks a model by rules"]
R -->|"model: target"| P["Provider routing
picks a provider and calls it"]
P -->|success| OK["Returned to caller"]
P -->|final failure| F{"Error is switchable
backup left and under attempt limit?"}
F -->|yes| N["Switch to the next backup model"]
N --> P
F -->|no| ERR["Last error returned"]
```
| Concept | Description |
| --- | --- |
| Rule | A set of match conditions plus an action. Rules are evaluated by priority, and **the first matching rule wins** |
| Target model | The model used once a rule matches. Several targets with weights are picked at random by proportion |
| Backup model | Models tried in order when the primary model ultimately fails |
| Virtual model | A model name that exists only in the gateway (such as `og-auto`); it does not need to exist at, or be priced by, any provider |
| Complexity tier | `SIMPLE`, `MEDIUM`, or `COMPLEX`, judged by the gateway from the request content |
Requests that match no rule are forwarded unchanged. If a rule itself errors (for example, a malformed condition), only that rule is skipped; the request is never blocked.
## Prerequisites
- The administrator has enabled the Intelligent Routing plugin on the target Gate instance and configured routing rules.
- Every target model and backup model used in the rules has been onboarded and priced in [Providers & Models](/en/admin/providers-and-models), and the caller's group has an available provider for it.
- If the caller's API Key restricts available models, the list must include those target and backup models; see [Billing and Permissions](#billing-and-permissions).
### Supported Endpoints
| Endpoint type | Path | Where the model comes from |
| --- | --- | --- |
| OpenAI format | `POST /v1/chat/completions` | Request body `model`, including streaming |
| OpenAI Responses format | `POST /v1/responses` | Request body `model`, including streaming |
| Anthropic format | `POST /v1/messages` | Request body `model`, including streaming |
| Legacy completions | `POST /v1/completions` | Request body `model`, including streaming |
| Gemini native format | `POST /v1beta/models/{model}:generateContent`, `:streamGenerateContent` | **The model in the URL**; `model` in the request body is ignored |
## Administrator Setup
::: warning The management UI is a preview
The scenario setup, dry-run, and routing monitor UI described in this section is currently a **preview**, available under **Plugins → Intelligent Routing** in the OneGateway console. Administrators can also edit Gate's rule definition file directly, or use Gate's [rule management API](/reference/intelligent-routing#规则管理接口) (Chinese) to configure and dry-run rules.
:::
Common uses can be configured as scenarios by filling in model names. The effects below describe the rules a scenario generates; when you write the rule file yourself, behavior follows the rules you write:
| Scenario | What you fill in | Effect |
| --- | --- | --- |
| Virtual model · tiered by complexity | A virtual model name and the model for each of the simple, medium, and complex tiers | Besides the three tier rules, a catch-all rule is generated: when complexity cannot be determined (for example, an image-only request), the medium tier model is used |
| Virtual model · fixed mapping | A virtual model name and the model it maps to; you can add several models with shares | Callers send the virtual model name (such as `og-coding`), and the gateway always rewrites it to the given real model |
| Replacement and split | The original model and the model to replace it with; you can add several models with shares | One model replaces all traffic; several models split traffic by share for a canary or load balancing, and the original model can be included to keep part of the traffic |
| Failover | The primary model and backup models in order | The model is not changed; switching happens only on failure |
| Model alias | Some alias names and the model they map to | Aliases are first rewritten to the target model, then the target model's other scenarios apply |
| Conditional routing | Which models it applies to (wildcards allowed; empty means all models), the conditions, and the model to switch to | Conditions can be combined: long context (estimated tokens reach a threshold), contains images, has tools, tenant, request header (such as recognizing a client by `user-agent`), reasoning effort, protocol, streaming, complexity tier |
Every scenario except failover can **optionally** carry backup models; the three tiers of auto tiering can share one list or each have their own. When needed, a single scenario can also customize its switching conditions (which status codes and which additional error codes switch, and the maximum attempts), while other scenarios keep the global settings; see [Automatic Switching on Failure](#automatic-switching-on-failure).
Virtual models are managed on their own page, and each card also lists the conditional routing that applies to it (for example, requests with images going to a vision model), which you can add there directly. Other scenarios and hand-written rules share one list on the rules page, in **matching order**, and the first one that matches wins: conditional routing is placed before other rules for the same model by default, and you use the up and down buttons at the start of a row to reorder, with priorities assigned automatically on save. A scenario placed after one that takes all of the same requests can never match, and the UI flags it.
Each scenario consists of one or more rules carrying a scenario tag (`scenario`, used only to group rules in the UI; it does not affect routing). Rules edited by hand in the raw file so that they no longer fit the scenario template are shown as custom rules and can only be changed in the raw file.
Before saving, you can **dry-run** a sample request to see which rule matches, which model it becomes, and which tier it is judged as. A dry run does not forward the request to the target model and incurs no generation cost for it; however, with LLM or embedding classification enabled, the configured classification service is still called, which may incur platform-side cost, and the sample text is sent to that service.
Rules take effect as soon as they are saved, with no gateway restart. For more complex conditions (combining request headers, tenant, protocol, input length, and so on, or chained rules), write them in the raw rule file; see the [technical reference](/reference/intelligent-routing#规则定义文件) (Chinese) for fields and examples.
### Everyday and platform configuration
The admin UI splits configuration into two levels and keeps them apart:
| Level | For | Includes |
| --- | --- | --- |
| Everyday | Organization (company) administrators who understand AI tokens and are comfortable with the platform | Virtual models, replacement and traffic split, failover; conditional routing built from long context, images, tools, and client; enabling, ordering, and backup models of scenarios; picking one of the prepared classifiers; dry runs |
| Platform | OneGateway platform administrators | Classifiers, the global switching policy, the allowed model range, the plugin switch and runtime settings; per-scenario switching conditions; model aliases; routing by tenant or reasoning effort; custom rules and raw editing; alert thresholds |
The viewpoint button at the top right of the admin UI switches between the organization administrator view and the platform administrator view: the organization view shows only everyday configuration, while the platform view shows everything, with platform configuration in the "Platform" menu group or in collapsed sections labeled "Platform" on everyday pages. Mistakes in platform configuration affect every request, so platform administrators should maintain it.
### Alerts
Alerts appear at the top of the page in these cases, with thresholds set in the platform settings:
- A scenario's failure rate from rate limiting or upstream faults over the last hour is high; consider adding backup models.
- The share of requests switched to backup models is high; the primary model may be unstable.
- Among requests that need a complexity tier, too many get none; platform administrators should check the classifier.
Caller-side failures such as invalid keys, insufficient quota, or unpriced models cannot be fixed by switching models, so they are not counted in failure rates and are shown separately on the overview.
### How Complexity Is Judged
Complexity is judged by a **classifier** that the platform configures, in one of these ways:
| Method | How it works |
| --- | --- |
| Local scoring | The request is scored locally from signals such as length, code terms, reasoning phrases, technical terms, and greetings or simple questions, then picks the tier from the score; no model is called, so it adds almost no latency and no cost |
| Local first | Score locally first; simple requests with clear signals, such as greetings and short questions, are decided right away, and only uncertain ones go to LLM classification |
| LLM classification | The user's request is handed to a small platform-provided model, which picks the tier |
| Embedding exemplar matching | The request is compared with preset example sentences for each tier and joins the closest tier |
LLM classification and embedding exemplar matching use the platform's own model and credentials, never the caller's API Key or model, and each judgment adds one call to the classification model. Local first calls it only when the local score is uncertain. The administrator can set another method as a fallback that is used when the primary one fails; a local scoring fallback always produces a tier.
Local scoring reads only the wording, so it is coarser than a model: on its own it got about half of a labeled test set right. Local first is recommended instead, saving about 40% of classification calls while staying close to LLM accuracy. It is a routing hint only: callers can influence the tier through their wording.
Administrators can also configure two independent LLM classification models. A successful primary result never calls the backup. Only a timeout, request failure, invalid tier, or full concurrency limit triggers the backup. Every new classification starts with the primary again, so recovery automatically restores primary use. If both fail, the request gets no tier. Backup tiers apply only to the current request and are not saved in sessions. Separate providers, endpoints, and credentials can reduce shared failures.
Classification runs before the business model call, so its wait adds directly to streaming time to first byte and non-streaming response time. Every new classification tries the primary first; a consistently timing-out primary makes each request wait for that timeout before using the backup. Administrators should choose a short primary timeout based on measured latency and budget for credential validation and both classification attempts.
::: warning No complexity judging without a classifier
If the administrator has not configured a classifier, or the classification model is temporarily unavailable or times out with no fallback, the request gets no tier: rules that split by tier do not match, and the request is handled by the rules after them. Auto tiering scenarios generate a catch-all rule that uses the medium tier model in this case.
:::
By default only **the most recent message written by the user** is considered; administrators can widen this to the last several messages with `message_window`. System prompts, assistant replies, and tool results are always excluded, and tags that tools such as Claude Code add automatically are stripped.
Different virtual models can use different classifiers: administrators can create additional "named classifiers" (for example, local scoring for a chat entry point and an LLM for a coding entry point) and have a virtual model use one of them. Virtual models without a choice use the default classifier. Session tiers are kept separately per virtual model's classifier, so one never affects another.
Differences, parameters, and accuracy of each method are covered in [complexity tiers](/reference/intelligent-routing#复杂度分级) (Chinese).
::: tip When complexity is judged
Rules are evaluated in priority order, and complexity is judged **the first time a condition needs it**, at most once per request for each classifier; if chained rules use different named classifiers, each one judges once and the costs add up. If a rule unrelated to complexity matches before that, no judging happens and there is no overhead; but once judging has happened, its cost is incurred even if the rule that finally matches does not depend on complexity.
:::
### Session Tiers
When enabled, the tier within one session **only goes up, never down**: once an earlier turn of a long conversation is judged complex, later short follow-ups keep using the complex tier model, avoiding back-and-forth model switches that invalidate the provider's context cache. A tier is recorded for the session only when the request succeeds and the tier comes from the configured classifier; a fallback judgment used because the classifier failed applies to that request only. Judging starts over after the session has been idle past its expiry, or after the administrator changes the primary judging method (a different classification model, prompt, or exemplars); adding, removing, or tuning only the fallback method leaves existing sessions untouched. Sessions are identified by the `hcd-og-session-id` request header, and Claude Code's built-in session header is recognized too; different API Keys never share a session even with the same session ID.
## Quick Verification
Make sure an auto tiering scenario is configured (using the virtual model `og-auto` as an example). Replace the address and API Key with real values and send a simple request:
```bash
curl -sS -i 'https://your-gateway.example.com/v1/chat/completions' \
-H 'Authorization: Bearer ' \
-H 'Content-Type: application/json' \
-d '{
"model": "og-auto",
"messages": [{"role": "user", "content": "Hi, how is the weather today?"}]
}'
```
Then keep everything else the same, swap `content`, and verify in turn:
| Check | `content` | Expected response headers |
| --- | --- | --- |
| Simple request | `Hi, how is the weather today?` | `og-route: ROUTED`, `og-route-tier: SIMPLE`, `og-route-model` is the simple tier model |
| Medium request | `Help me analyze the error in this log` | `og-route-tier: MEDIUM`, `og-route-model` is the medium tier model |
| Complex request | A long prompt asking to design a system architecture and analyze trade-offs | `og-route-tier: COMPLEX`, `og-route-model` is the complex tier model |
Streaming requests (`"stream": true`) return the same headers, visible before the first piece of content. Use `og-route-model` to see which model Intelligent Routing chose. The `model` in the response body is the model name reported by the provider, which may differ from `og-route-model` when the provider maps models or uses versioned names.
## Billing and Permissions
**Billing follows the routed model**: whichever model `og-auto` is rewritten to (that is, `og-route-model`) is charged at that model's platform price, and the provider side sees and records the rewritten model. The virtual model name itself needs no price. The billing details in the [request logs](/en/user/usage) are authoritative for the actual cost.
::: warning API Keys with a model allowlist
The provider side checks an API Key's model allowlist against the **rewritten model**. Adding only `og-auto` to the allowlist does not work: every target and backup model the rules may use must be included, otherwise the request returns 403, and a 403 **does not trigger a switch**.
:::
When switching to a backup model, the failed attempt is not charged; only the final successful attempt is billed.
## Automatic Switching on Failure
When the matching rule has backup models, the gateway tries them in order after the primary model **ultimately fails** (provider routing has already retried and still failed).
| Primary model returns | Switch? | Why |
| --- | --- | --- |
| 503 no available provider, 429 rate limited, 500 / 502 / 503, 529 provider overloaded, connection failure | Switch | The model is temporarily unavailable; another model may succeed |
| 504 / 524 timeout | No by default | Switching would multiply the wait time; the administrator can enable it |
| 400 unpriced, 403 no access, insufficient quota | No | Another model would not help, and switching could bypass permissions |
| Sensitive-word rejection | No | The same content would be rejected by other models too |
- At most 3 attempts by default (including the first).
- These are the global defaults. A single scenario can customize its switching conditions, for example counting 504 as switchable for a long-context scenario, or switching on a context-length error code. Only that scenario is affected; insufficient quota, unpriced models, and sensitive-word rejections never switch.
- **Streaming requests switch only before output starts**; once content has started streaming, an upstream interruption does not trigger another model.
- If all backup models also fail, the error from the last attempt is returned.
- Requests that switched models are not written to the [Semantic Cache](/en/guide/semantic-cache).
## Observing Results and Troubleshooting
Start with the response headers:
| Header | Meaning |
| --- | --- |
| `og-route` | `PASS` no rule matched; `ROUTED` a rule matched; `FALLBACK` a switch happened (see the status code for the outcome); `BYPASS` the caller skipped routing; `ERROR` routing itself errored and the request was forwarded with the original model |
| `og-route-rule` | ID of the matching rule (only the ID, never the conditions) |
| `og-route-model` | The model sent to the provider after routing (the last attempted model if a switch happened), which is also the billed model |
| `og-route-requested-model` | The model the caller originally requested; present only when the model was rewritten |
| `og-route-attempts` | Number of upstream attempts (including the first); greater than 1 means a switch happened |
| `og-route-tier` | Complexity tier; present only when complexity was judged |
With log storage enabled, administrators can view the routing outcome distribution, complexity tier distribution, and the "requested model → actual model" flow by time window in the routing monitor, and drill into individual requests.
::: info The model in request logs is the routed model
The models in [request logs](/en/user/usage) and the [dashboard](/en/user/dashboard) are the **models actually called after routing**; virtual model names such as `og-auto` never appear there. To check the model originally requested, use the routing monitor or the `og-route-requested-model` response header.
:::
### Troubleshooting by Symptom
| Symptom | Check first |
| --- | --- |
| No `og-route` response header | Whether the plugin is enabled, whether the endpoint is supported, and whether an intermediate proxy keeps response headers |
| `og-route: PASS` and the model is unchanged | Whether the model name exactly matches the rule (`gpt-4o` does not match `gpt-4o-mini`), whether the rule is enabled, and whether it was saved successfully |
| 403 returned | Whether the API Key's model allowlist includes the model shown in `og-route-model` |
| 400 unpriced returned | Whether the target or backup model is priced |
| Frequent `FALLBACK` | Whether the primary model has an available provider in the current group; a group misconfiguration looks like "always switching" |
| Tier differs from expectation | Use a dry run to see the basis; adjust the classification model's guidance or exemplar sentences, or switch to the other method |
| No `og-route-tier`, requests always take the catch-all rule | Whether a classification model is configured and reachable, and whether the caller's API Key is valid (by default the classification model is called only for API Keys that pass validation; the primary and fallback methods each follow the administrator's own validation setting) |
| Rule changed but not in effect | Whether saving reported an error; a broken rule keeps the gateway on the previous valid rules |
### Known Limits
- Intelligent Routing only decides the model; it does not choose providers or API Keys, nor route by quota or rate-limit usage.
- No time-to-first-token switching: a streaming request that has started output is not moved to another model.
- Backup model compatibility is the administrator's responsibility: switching only changes the model name and does not adapt the request to the backup model's context length, tool calling, or other capabilities.
- Session tiers are kept by the service that handles the request. When several copies of the service are deployed they do not share session tiers, so a request routed to a different copy is judged again.
## Per-Request Control
| Request header | Effect |
| --- | --- |
| `hcd-og-route: off` | Skip routing for this request and forward the original model. **Ignored by default**; the administrator must enable "allow callers to skip routing". Routing may carry cost or compliance policy, so evaluate before enabling it |
| `hcd-og-session-id` | Session ID used for [session tiers](#session-tiers) |
| `hcd-og-tenant-id` | Used by tenant-based rules. Callers can set this header freely, so it is **only a traffic-splitting device, never a security or billing boundary** |
## Relationship with Other Capabilities
- **Guardrails**: requests pass [Guardrails](/en/guide/guardrails) before routing, so blocked requests incur no complexity judging, and the judging sees already-redacted text.
- **Semantic Cache**: routing runs before the [Semantic Cache](/en/guide/semantic-cache) lookup; the cache is separated by the rewritten model, so answers from different models never hit each other, and requests that switched models are not cached.
- **Call logs**: logging wraps routing, so rewritten and switched requests are recorded with the requested model, actual model, matching rule, attempts, and complexity tier.
## FAQ
### What do callers need to change?
Only change the model name in the request to the virtual model name the administrator provides (such as `og-auto`). Endpoints, request formats, and SDKs stay the same. Replacement or failover rules on existing models are completely invisible to callers.
### Why is `og-auto` missing from the available models list?
Virtual models exist only in the gateway's routing rules and map to no provider, so they do not appear in [Available Models](/en/user/models). You can use them directly in requests; ask the administrator which virtual model names are available.
### How much latency does Intelligent Routing add?
Rule matching runs locally in the gateway, so the overhead is usually negligible. When a complexity tier is needed, local scoring adds almost no latency; the other methods add one call to the classification model per judgment (local first calls it only when uncertain), whose latency depends on where the classification service runs and the network (tens to about a hundred milliseconds in a test with an embedding model on the internal network); a caller credential check may also run first (its result is cached). On timeout the request gets no tier and is handled by the later rules. When a switch happens, total time includes the failed attempt.
### Where is my content sent for complexity judging?
Text is sent only when a complexity tier is needed and a classification model is called; local scoring sends nothing, and local first sends only the requests it is uncertain about. The user message text being judged (the latest one by default; administrators can widen it to the last several with `message_window`) is sent to the configured classification model (which can be a model the platform hosts itself). System prompts, assistant replies, and tool results are not sent. Classification calls are paid by the platform and do not count against the caller's quota.
### Why does the same question sometimes use model A and sometimes model B?
Three possible reasons: the rule splits traffic by percentage, complexity judging is sensitive to wording, or the primary model failed and a backup model was used. The `og-route-rule`, `og-route-tier`, and `og-route-attempts` headers tell them apart.
### Quick Answers
| Question | Answer |
| --- | --- |
| Do rule changes need a gateway restart? | No, they take effect as soon as they are saved |
| Can a broken rule make requests fail? | No, the gateway keeps the previous valid rules; a single rule whose condition errors is just skipped |
| Are model names case-sensitive? | Rule matching is case-insensitive but must match the whole string |
| Can routing apply to only some tenants? | Yes, but the tenant header can be forged by callers, so it is only a traffic-splitting device; see [Per-Request Control](#per-request-control) |
| Fields and advanced configuration | See the [Intelligent Routing technical reference](/reference/intelligent-routing) (currently available in Chinese only) |