Model routing

Choosing the model for each kind of work

Model routing is how you decide which model does each kind of work in a turn. A turn is not one model call. Writing the reply, ranking rules, reviewing a draft, and summarizing history are different jobs with different needs. Routing lets you keep a fast model for replies and a stricter one for reviews, or use one provider for everything, without touching your policies or tools.

What you'll learn

• The routes a turn uses, and what one provider gives you by default
• How to point a family of work at a different model
• How to vary the model by the turn's effort level
• How agents carry their own routes, credentials, and rate limits

Talk this page through with your coding agent

Parlant calls each kind of model work a route. A route has a name, such as react for reply generation or review.message for checking a draft. The models argument to p.Server maps those names to the providers or models that serve them.

You can configure routing at three levels of detail. Most deployments stop at the first, one provider for everything:

import parlant.sdk as p

async with p.Server(models={"*": "openrouter"}) as server:
    ...

Here, * is the provider fallback: one bare provider name, whose registered default model serves every route. If you omit models altogether, Parlant uses OpenRouter.

info

OpenAI and OpenRouter ship with the base package. Gemini, Anthropic, and LiteLLM need an extra, such as pip install 'parlant[gemini]', plus the provider's API key in its usual environment variable.

The routes

Each route names a job the engine does during a turn, or, in a couple of cases, at startup:

Route	Purpose
react	The agent's response steps.
matching.rank	Rank and prune rules for the current turn.
review.tools / review.message	Review proposed tool calls and drafted messages.
compaction	Summarize older session history.
embedding.rules / embedding.tools	Embed rule signals and tools for relevance selection.
moderation.customer / moderation.agent	The two opt-in moderation checkpoints.
indexing	Generate rule signals and other indexes at startup.
tokenizer	Estimate token counts.

Note that a route your engine never exercises costs nothing. If you never opt into moderation, for example, the two moderation routes are never called.

What the fallback picks

Each provider registers a default model per kind of work. For replies on react, the default also depends on the turn's effort (the level, from MIN to MAX, that decides how much model work a turn gets):

Provider	Replies by effort	Matching and review	Other work
OpenRouter	anthropic/claude-haiku-5.5 at MIN, LOW, MEDIUM; openai/gpt-6.1-sol at HIGH; anthropic/claude-sonnet-5.5 at MAX	openai/gpt-5.6-terra for reviews; rule ranking by decision model ~typesafe/jev-latest, otherwise openai/gpt-oss-20b:nitro	openai/gpt-5.6-terra for indexing, openai/gpt-oss-20b:nitro elsewhere
OpenAI	gpt-6-luna at MIN through HIGH; gpt-6.1-sol at MAX	gpt-5.6-terra for reviews; rule ranking by decision model gpt-6-luna, otherwise gpt-5.6-luna	gpt-5.6-terra for compaction, gpt-5.6-luna elsewhere
Anthropic	claude-haiku-5-5 at MIN through MEDIUM; claude-sonnet-5-5 at HIGH; claude-opus-5-5 at MAX	claude-haiku-5-5	claude-sonnet-5-5; embeddings through Voyage AI voyage-3.5
Gemini	gemini-3.5-flash-lite at MIN through MEDIUM; gemini-3.8-flash at HIGH and MAX	gemini-3.8-flash for reviews; gemini-3.1-flash-lite for ranking	gemini-3.1-flash-lite

Note that OpenRouter model slugs use a dot (claude-haiku-5.5), while Anthropic's own IDs use a dash (claude-haiku-5-5).

Decision models

A decision model answers a fixed question with a probability instead of writing prose. Parlant can use one to rank a whole batch of rules in a single request, which is cheaper than asking about each rule. The ranker uses it at MIN, LOW, and MEDIUM effort, where no per-rule explanation is wanted.

The matching.* and review.* routes accept either an ordinary model or a decision model. To route decisions, add the [capability=decision] qualifier to the route key, for example "matching.rank[capability=decision]": p.ModelConfig("openrouter:~typesafe/jev-latest"). A named route without that qualifier never serves decisions. A bare provider fallback such as {"*": "openai"} serves decisions only if that provider registered a decision default.

Today OpenRouter (~typesafe/jev-latest) and OpenAI (gpt-6-luna) register one; Gemini, Anthropic, and LiteLLM do not. So a default server ranks rules with a decision model, and a Gemini-only server ranks them with its ordinary model. Note that the OpenAI decision model reports no cost of its own, so it needs an entry in your price table to be costed.

Overriding a family of work

Once you have picked a fallback provider, the next level of detail is to point some routes at a different model. Parlant lets you do that one route at a time, or for a whole family at once.

The dot in a route name separates related kinds of work: review.tools and review.message belong to the same review family. So to use the same model for both tool and message review, you configure their shared review.* route instead of naming each one.

To bind a route to a concrete model, use p.ModelConfig("provider:model"). The provider prefix selects Parlant's adapter, and Parlant passes the rest to the provider unchanged. A terminal .* wildcard, one that ends the key, covers the whole namespace:

async with p.Server(
    models={
        "react": p.ModelConfig("gemini:gemini-3.1-flash-lite"),    # response generation
        "review.*": p.ModelConfig("gemini:gemini-3.5-flash-lite"),  # both review checks
        "*": "openrouter",                                          # everything else
    },
) as server:
    ...

When several entries could apply, Parlant picks the most specific one: exact routes beat wildcards, and wildcards beat *.

Note that only terminal wildcards work. matching.* is valid, but *.rank is not.

Varying by effort

So far, a route has always mapped to the same model. But not every turn deserves the same model, and Parlant lets you vary the choice by how much work the turn calls for.

Every turn runs at an effort level, from MIN to MAX, which decides how much model work the turn gets. Your agent sets the baseline, and a matched rule can raise it for one turn.

A route qualifier is a condition in square brackets after the route name, such as [effort>=high], that makes an entry apply only on turns that satisfy it. With one in place, a HIGH turn runs on a stronger model than a LOW one, with no change to your rules:

models = {
    "react": p.ModelConfig("openai:gpt-6-luna", reasoning="low"),   # the default for replies
    "react[effort>=high]": p.ModelConfig("openai:gpt-6.1-sol", reasoning="high"),  # replies on HIGH and MAX turns
    "matching.*[effort<medium]": p.ModelConfig("openai:gpt-5.4-nano"),  # cheap matching on MIN and LOW turns
    "*[effort>=high]": "anthropic",   # every other route on HIGH and MAX turns
}

The comparisons you can use are =, !=, <, <=, >, and >=, over the ordering min < low < medium < high < max. To combine several constraints, separate them with a comma, which Parlant reads as AND. The capability qualifier described above takes only =, and a key may name one capability.

Here, too, the narrower entry wins: a narrower qualifier beats a wider one. If two selectors overlap with equal specificity, Parlant rejects them at startup rather than guess.

Note that Parlant resolves the effort value when each operation begins: the agent's baseline, raised by any rule matched so far.

Where effort comes from

Adaptive effort explains where the raises come from.

Reasoning depth

On providers that support it, reasoning sets how deeply the model thinks before answering. It rides on the route the same way the model does. You can pass a shorthand such as "high", or a full p.ReasoningConfig(effort=..., visibility=...).

Note that if you leave reasoning unset, each component maps the turn's effort to a reasoning depth by itself.

Per agent, and live

Everything so far has been configured on the server. Server routes are defaults; an agent's own models override them. That lets two agents on one server use different models, or even different credentials:

quality_agent = await server.create_agent(
    id="claims",
    name="Claims Agent",
    instructions="Handle complex requests carefully.",
    models={
        "react": p.ModelConfig("anthropic:claude-sonnet-5-5"),
        "review.*": p.ModelConfig("anthropic:claude-opus-5-5"),
    },
)

Agent routes win over server routes at every level. So an agent's matching.* beats a more specific matching.rank on the server, even though the server's entry is narrower.

Parlant looks routes up immediately before each operation. That means await agent.update(models={...}) takes effect on the next operation, with no restart. Assigning None to a key removes the override.

Credentials and rate limits

Once agents can carry their own routes, they can carry their own provider keys, too. That raises the question of how to store those keys safely.

Never put a plaintext key in an agent route. Agent routes are durable configuration. Instead, use p.SecretRef("locator"), a reference that names where a secret lives, or p.SecretValue(...) for a key that must stay in memory only.

Parlant resolves a p.SecretRef through a p.SecretLoader, a class your coding agent writes that fetches the secret from your own vault. You register it in the container (the registry that maps each server component to the implementation serving it) through configure_container, a function p.Server calls once at startup:

class VaultSecretLoader(p.SecretLoader):
    async def load(self, ref: p.SecretRef) -> str:
        return await my_vault.read(ref.key)


async def configure_container(container: p.Container) -> p.Container:
    container[p.SecretLoader] = VaultSecretLoader()   # Parlant now resolves every SecretRef through your vault
    return container


agent = await server.create_agent(
    id="claims",
    name="Claims Agent",
    instructions="...",
    models={
        "react": p.ModelConfig(
            "openai:gpt-5.6-terra",
            args={"api_key": p.SecretRef("agents/claims/openai")},   # the locator your loader receives
        ),
    },
)

Pass configure_container when you create the server, before creating the agent above: p.Server(configure_container=configure_container). Here, my_vault stands for your secret-manager client.

Learn more about the container

Overriding engine behavior covers the container in full.

Rate limits

Each route also needs to stay inside its provider's quota. For that, attach a p.RateLimit to the same config. It caps how many requests on that route run at once, and how many start per minute. Requests beyond the limit queue in order rather than fail:

models = {
    "*": "openai",
    "indexing": p.ModelConfig(
        "openai:gpt-6-luna",
        rate_limit=p.RateLimit(max_concurrency=4, requests_per_minute=60),
    ),
}

To share one quota across several routes, give the limit a bucket name.

Note that the default limiter is process-local. For a distributed one, bind your own p.RateLimiter.

Choosing a model in code

Fallbacks, families, qualifiers, and per-agent routes cover most needs. But sometimes the choice depends on something the selector grammar cannot express: which customer is talking, which tenant they belong to, or a feature flag. For those cases, a route's model can be an async callable.

Parlant runs the callable before every operation on that route and hands it the p.EngineContext, the live state of the turn being processed. Inside it, p.Customer.current identifies the customer.

The example below assumes your application has already created default_react and the generators in react_by_tenant, and that tenant_for_customer looks the customer up in your records. The callable returns one of those ready reply generators:

async def provide_react(context: p.EngineContext | None):
    if context is None:            # no turn in progress: an offline resolution at startup
        return default_react

    tenant = tenant_for_customer(p.Customer.current.id)
    return react_by_tenant[tenant]


models = {"react": p.ModelConfig(provide_react)}

Keep the callable cheap and free of side effects, and have it select a prebuilt object rather than construct one. Parlant does not cache what it returns. A callable on a matching.* or review.* route may also return a decision model.

Note that Parlant also runs it with context=None, for offline resolutions at startup, so it must handle that case. And for plain by-effort selection, prefer qualified route keys; there is no need for code.

Naming a provider in usage reports

Usage and cost records are keyed by provider:model, for example anthropic:claude-sonnet-5-5 or openrouter:anthropic/claude-sonnet-5.5. The provider part comes from the object serving the route: its provider_name. A built-in adapter reports under its own prefix (openai, anthropic, gemini, openrouter, litellm).

When you build generators yourself, as in the example above, two tenants' spend would land under the same provider. To see them apart, name each object when you build it. Every built-in schematic generator, embedder, ReAct generator, and decision maker accepts provider_name, for example OpenAIReactGenerator(..., provider_name="openai-eu").

The name is free-form and must not contain a colon. It cannot be passed through ModelConfig.args.

Note that a custom adapter must pass it too: BaseSchematicGenerator.__init__ and BaseEmbedder.__init__ take a required keyword-only provider_name. Price each name with a provider:model entry in your price table, or let a bare model entry cover every provider. Spend management covers the price table.