Model routing harness
Not every task needs a frontier model, and not every task is allowed to use one. Routing decides which model handles a given request, automatically, based on rules instead of habit.
- Policy harness
- Model routing harness
- Data transfer harness
- Guardrails harness
- Security harness
- Evaluation harness
- Observability harness
- Cost and token harness
- Reliability harness
- Audit harness
What it controls
The routing harness sits between your application and the model providers. Instead of code calling one hardcoded model, a request describes what it needs (the task, the data sensitivity, the acceptable cost and latency), and the harness picks a model that fits: a frontier model, an open-weight model, or a small model trained for one job.
It also enforces the policy harness's rules about which models are allowed to see which data, so sensitive requests cannot accidentally reach a model or provider that should not have them.
Why it matters
Without routing, model choice gets hardcoded once and stops being a decision. Every task goes to the same model regardless of whether it needs that much capability, whether it is allowed to see that data, or what it costs. That locks you into one vendor by default and makes it harder to react when a cheaper or better-fit model becomes available.
The point is not that any one model is risky. It is that an enterprise with no routing layer has no way to apply its own judgment about which model is appropriate for which job.
What good looks like
- Routing decisions are based on explicit criteria: data sensitivity, task type, cost ceiling, latency requirement
- Swapping or adding a model is a configuration change, not a code change across every integration
- The same routing layer enforces policy's data-access rules, so a model can only receive data it is approved for
- Routing is model-agnostic: frontier, open-weight and small models are all first-class options, chosen on merit for the task
- Fallback behavior is defined for when a preferred model is unavailable or over budget
How it works with the other harnesses
Policy decides what is allowed; routing decides which model actually does the work within that boundary. Cost and token tracks what each routed call spends, so routing decisions can be tuned by actual cost, not guesswork. Reliability handles what happens when the routed model fails or times out.
In practice
Real patterns on the Resources page that lean on this harness:
- AI agent evaluation in your own tenant
Frequently asked questions
Does routing mean always picking the cheapest model?
No. It means picking the model that fits the task, the data, and the budget together — sometimes that is the most capable model available, sometimes it is a small model trained for one job.
Can we still choose our preferred providers?
Routing is model-agnostic by design: frontier providers, open-weight models and small models are all options the routing rules can select between. Nothing about the harness favors one provider.
What happens if a chosen model is down or over capacity?
A defined fallback path (a different model, or a queued retry) should kick in, which is routing working together with the reliability harness.
Does this require rewriting application code every time we add a model?
No — adding a model should be a configuration change to the routing rules, not an application code change.
Questions about your AI rollout?
Talk to us about which harnesses fit your models and data.