Your services shouldn't know which model they're calling

Most systems start the same way. A service needs a model, so it imports the provider's SDK, reads a key from the environment, and calls it. A second service does the same. By the time there are nine of them, the model name is a string literal in nine repositories.

It works. It keeps working right up until one of five perfectly ordinary things happens.

• A provider deprecates the model you named.
• A provider has a bad afternoon.
• Somebody wants to test whether a cheaper model is good enough for one job.
• Finance asks what any of it cost.
• A customer asks you to stop before they spend any more.

Each of those is now a change in nine services, coordinated, deployed. That's the problem a model gateway exists to solve: a single service every other service calls instead of calling a provider directly.

The idea is obvious enough. What's interesting is the handful of decisions inside it that are easy to get wrong, and we've recently had the chance to watch a good one being built.

Services ask for a job, not a model

The central move is an indirection: a service never names a model. It names a slot — the job it wants done. "Summarise this document." "Classify this intent." "Read this aloud."

The slot holds the model choice, and a chain of what to use instead when the first choice is unavailable.

// A slot is a job, not a model. Services ask for the job.
type Slot struct {
    Name      string       // "summarise-document", "classify-intent"
    Kind      Kind         // text, image, speech - a fallback must match
    Primary   string       // what runs unless something is wrong
    Frontier  []string     // tier 1: as capable, costs about the same
    Fast      []string     // tier 2: quicker, cheaper, usually weaker
    Alternate []string     // tier 3: a different provider entirely
    Requires  []Capability // tool calling, web search, code execution
}

// A candidate survives selection only if it is enabled, of the right kind,
// and can do everything the slot requires.
func (b *builder) add(id string, tier int, fallback bool) {
    m, ok := b.lookup(id)
    if !ok || !m.Enabled || m.Kind != b.slot.Kind {
        return
    }
    // A fallback is chosen while something is already broken, which is the
    // worst moment to silently move traffic onto the dearest model you own.
    if fallback && m.HighCost {
        return
    }
    for _, c := range b.slot.Requires {
        if !m.Has(c) {
            b.lacked = true // remembered, so the error can say why
            return
        }
    }
    b.out = append(b.out, Candidate{Model: m, Tier: tier})
}

Changing which model does a job becomes a configuration edit rather than a deployment. That alone pays for the gateway, because it converts "we should try the cheaper model on classification" from a sprint into an afternoon, which is the difference between a question you investigate and one you keep meaning to.

A fallback has to be capable, not just available

The selection code above is doing something less obvious than it looks.

If a job requires tool calling and the fallback can't do tool calling, then failing over to it is worse than failing. You don't get an error — you get a confident, fluent answer produced by a model that silently ignored the tools it was given. The caller has no way to tell that apart from a real answer.

So capability is part of selection, not a runtime surprise. And when nothing in the chain qualifies, the error says which of the three things went wrong: the slot was never configured, or there are models but none are capable, or there are capable models but none are enabled. Those are three different jobs for whoever is on call, and collapsing them into "no model available" wastes the first twenty minutes of an incident.

Fallbacks must skip the expensive models

This is a single clause in the selection logic and it's worth the whole article.

A fallback runs when something is already wrong. That is precisely the moment you are least likely to be watching a cost dashboard, and most likely to be routing a lot of traffic somewhere new. A chain that happily fails over onto your most expensive model will do it at three in the morning, for six hours, across every request.

Marking models as high-cost and excluding them from the fallback path — while leaving them available as a deliberate primary — means an incident degrades quality instead of multiplying the bill. That's almost always the trade you want, and it's a trade nobody makes on purpose at three in the morning.

It's the same principle as the trade between cost, speed and quality: decide it in advance, in daylight, and encode it.

Not every failure is the provider's fault

Here's where most implementations go wrong, and it took us a while to see it too.

A gateway needs a circuit breaker per model, so that a provider having an incident stops receiving traffic instead of absorbing every request into a timeout. Standard practice. The subtlety is what counts as a failure.

// Why an attempt failed, in the only terms that matter: whose fault it was.
type Reason string

const (
    ReasonProvider5xx      Reason = "provider_5xx"
    ReasonNetwork          Reason = "network"
    ReasonTimeout          Reason = "timeout"
    ReasonRateLimited      Reason = "rate_limited"
    ReasonModelUnavailable Reason = "model_unavailable"
    ReasonQuotaExhausted   Reason = "quota_exhausted"

    // Ours, not theirs.
    ReasonContextOverflow Reason = "context_overflow"
    ReasonBadRequest      Reason = "bad_request"
    ReasonInvalidOutput   Reason = "invalid_output"
    ReasonContentFiltered Reason = "content_filtered"
    ReasonCanceled        Reason = "canceled"
)

// CountsAgainst decides whether a failure is evidence the provider is unwell.
// Only the first group is. A malformed request of ours, a prompt that ran past
// the context window, a safety filter doing its job and a user closing the tab
// say nothing at all about the model, and a breaker that counts them will take
// a perfectly healthy provider out of rotation because of our own bug.
func CountsAgainst(r Reason) bool {
    switch r {
    case ReasonProvider5xx, ReasonNetwork, ReasonTimeout,
        ReasonRateLimited, ReasonModelUnavailable, ReasonQuotaExhausted:
        return true
    }
    return false
}

Count every error and you have built a machine that takes healthy providers out of rotation because of your own bugs. A prompt that overflows the context window isn't the model's fault. A malformed request isn't either. A safety filter firing is the system working. A user closing the tab is not a provider outage.

If all of those trip the breaker, then the day you ship a prompt change that occasionally overflows, your gateway will decide the provider is down and fail every request over to tier three. You'll spend the incident staring at a provider status page that says everything is fine, because it is.

Separating "the provider is unwell" from "we sent something bad" is the difference between a breaker that protects you and one that amplifies your own mistakes. It's the same discipline as naming your failure modes before they happen: the taxonomy is the control.

Once output starts flowing, failover is over

Streaming adds a constraint that's easy to miss until it bites.

While you're waiting for a provider's first token, failing over is free — nobody has seen anything. The moment the first token reaches the caller, the model is locked in. You cannot switch providers halfway through a response, because the second half would be written by something that never saw the first.

That means tracking two separate moments: when the provider started responding (which stops the time-to-first-token clock) and when output was actually delivered to the caller (after which the choice is final). They aren't the same instant, and a gateway that conflates them will either give up failover too early or attempt it too late.

Now the cross-cutting things have somewhere to live

This is the part that justifies the hop, and it's the reason we think every team running more than two or three agents ends up here eventually.

Once every model call in the estate passes through one service, that service is the obvious place for all the concerns that were previously impossible to apply consistently:

• Authorisation — who is allowed to use which slot.
• Budget — asked once per billed request, before any model call, with a refusal that says whether the budget is depleted, unallocated or simply never configured, so the caller can show the right message instead of a generic failure.
• Pricing and usage — token classes counted once, in one place, against one rate card.
• Metering — every attempt recorded, including the ones that failed and the ones that were retried.

That last point is the one we've written about before from the other direction. Measuring what a unit of work actually costs requires recording every call made on its behalf, and no provider console can do it for you. Both of those articles end at the same place: it belongs at the one point every model call already passes through.

A gateway is how that point comes to exist. Without one, "instrument every call" means instrumenting nine services and keeping them in step forever. With one, it's a single integration that covers everything written afterwards, including the services nobody has written yet.

What it costs you

An honest accounting, because this is a real architectural commitment and not a free win.

It's another hop. Milliseconds, not seconds, and negligible against a model call — but it's not nothing, and it's on the latency path of everything.

It's a dependency that can be down. You have concentrated every model call in the business behind one service, which means its availability is now your availability. It needs the operational seriousness that implies: health checks that reflect real downstream state, alerting, and a deployment process that assumes a bad version is catastrophic rather than annoying.

It's a place for complexity to accumulate. Every per-caller special case wants to live in the gateway, and a few of them should. Most shouldn't. The discipline is that the gateway knows about models, capabilities, health and money — and nothing whatsoever about your domain.

It needs its own health story. Breaker state lives per instance, so with several instances running, no single one knows whether a model is genuinely unwell or just unlucky from where it's standing. Aggregating outcomes across instances into one view of model health is work, and skipping it gives you a fleet that disagrees with itself about which providers are up.

Where we'd start

If you have one service calling one provider, you don't need this. Genuinely — don't build it.

The signal to build it is the second or third service acquiring its own copy of the provider client, because that's the point where every subsequent question about cost, reliability or model choice becomes a question about all of them at once.

Start narrow. Slots and a registry, so model choice leaves the code. Then the breaker and the failure taxonomy, because that's what makes the fallback chain trustworthy rather than theatrical. The budget and metering hooks come next and are nearly free once every call already flows through one place.

The test of whether it's working isn't latency or uptime. It's whether swapping a model in production is a decision somebody can make on a Tuesday afternoon without calling a meeting.