Model Routing with LiteLLM: A Practical Guide

Why the gateway exists

Most systems begin by calling a provider SDK directly from application code, which is entirely reasonable for a prototype. The costs of that decision appear later and all at once. A provider degrades and there is no fallback path. Somebody asks which feature caused the monthly increase and nobody can answer. A new model becomes available and adopting it means touching a dozen call sites.

A gateway consolidates all of that into one boundary. Requests go to the gateway, the gateway decides which provider and model handles them, and the surrounding concerns — retries, failover, caching, budgets, tracing — are implemented once rather than repeatedly. LiteLLM is the common choice because it normalises the interface across providers, which is the part that otherwise leaks into application code.

The architectural argument is familiar: this is a repository layer for models. The reason it is worth stating explicitly is that teams who would never scatter raw SQL through their controllers routinely scatter provider SDK calls through their services, because the cost of doing so is deferred rather than immediate.

Designing the routing policy

The default policy should be explicit rather than emergent. Decide which model handles which class of request, and write it down as configuration rather than encoding it in scattered conditionals. The most common useful split is by task difficulty: a cheaper model for classification, extraction and routine generation, a stronger one for reasoning-heavy work.

Routing by difficulty only works if you can tell which is which, which is why this pairs with an evaluation harness. Without one, difficulty-based routing is a guess that silently degrades quality on the requests you misclassified. With one, you can move a request class to a cheaper model and see immediately whether the trade was acceptable.

Fallback chains are the second half of the policy. A fallback should degrade rather than fail: if the primary model is unavailable, a slightly weaker response is almost always better than an error. Where it genuinely is not — a correctness-critical path — the right behaviour is an explicit failure rather than a quiet substitution nobody notices.

What to instrument from day one

• Tokens in and out, attributed to a feature, team and environment via request tags
• Which model actually served each request, since fallbacks make this non-obvious
• Latency percentiles per model and per request class, not aggregate means
• Cache hit rate, separated by cache type, because a low rate usually indicates a design problem
• Failover events, which should be rare enough that each one is worth reading
• Error taxonomy: rate limits, timeouts, content filters and malformed responses counted separately
• Cost per resolved task, which is the metric that separates growth from inefficiency

The mistakes that show up later

The first is treating the gateway as a proxy and nothing more. A gateway that forwards requests without tagging them provides interface normalisation and none of the operational benefit, which is most of the reason to have one. Request tagging has to be designed in from the start, because retrofitting attribution onto historical data is impossible.

The second is failing silently between models. When a fallback fires and nobody is told, quality changes and the cause is invisible. Every response should carry which model produced it, propagated far enough that debugging a bad answer can begin with the right question rather than with a guess.

The third is hard-coding model identifiers at call sites. Model names change, deprecations arrive with short notice, and a system with model strings scattered through it turns a routine provider update into a search-and-replace exercise across the codebase. Model selection belongs in configuration, resolved by the gateway, with call sites requesting a capability rather than a specific model.

Rolling a gateway into an existing system

Teams usually adopt a gateway after the direct-call version already exists, which makes the migration path more important than the design. The approach that works is to introduce the gateway as a drop-in for one low-risk request class first, verify that tracing and attribution are actually flowing, and only then migrate the rest.

Doing it the other way round — migrating everything and then adding instrumentation — is how teams end up with a gateway that forwards traffic and answers no questions. The instrumentation is the reason for the migration, so proving it works on a small slice before committing is worth the extra week it costs.

The one thing worth doing globally and immediately is removing hard-coded model identifiers from call sites, even before the gateway is in place. Replacing them with a capability name resolved from configuration is a mechanical change, it is independently useful, and it removes the largest single obstacle to the migration that follows.

Part of the AI Engineering Practice cluster · Read the pillar page