A spend alert at 100% is just a receipt

Most teams running agents have a budget alert. It fires when the budget is gone.

That alert is a receipt. It tells you, accurately and far too late, that money you can't get back has been spent. Everyone has one, everyone feels covered by it, and it has never once prevented anything.

The useful question isn't whether you have alerts. It's whether any of them arrive while there's still a decision to make.

Thresholds are necessary and not sufficient

Start with the obvious ones, because you do need them: a notice at three quarters, something louder at ninety-five per cent, and the final one when the limit is reached.

Three quarters is the first one worth sending. Earlier than that and you're telling people the month is proceeding normally, which trains them to delete the next one unread.

But a level on its own is a weak signal, and it's worth being precise about why. Eighty per cent through a monthly budget is completely fine on the twenty-eighth and genuinely alarming on the fourth. The number is identical. Only one of them should wake anybody.

Pace is the alert that arrives in time

The signal that actually helps is a forecast: at the rate you're going, this budget runs out before the period ends.

// Level alone is not a signal. Eighty per cent through the budget is fine on
// the twenty-eighth and alarming on the fourth, and only one of those deserves
// to wake somebody.
func (b Budget) Assess(now time.Time) []Finding {
    var out []Finding

    used := float64(b.SpentMicros) / float64(b.LimitMicros)
    switch {
    case used >= 1.0:
        out = append(out, Finding{Level: Urgent, Kind: "exhausted"})
    case used >= 0.95:
        out = append(out, Finding{Level: Urgent, Kind: "near_limit"})
    case used >= 0.75:
        out = append(out, Finding{Level: Notice, Kind: "three_quarters"})
    }

    // The forecast is the part people leave out, and it is the only one that
    // arrives while there is still something you can do about it.
    elapsed := now.Sub(b.PeriodStart)
    total := b.PeriodEnd.Sub(b.PeriodStart)
    if elapsed > 0 && used > 0 {
        projected := float64(b.SpentMicros) * float64(total) / float64(elapsed)
        if int64(projected) > b.LimitMicros && used < 0.95 {
            out = append(out, Finding{
                Level:   Notice,
                Kind:    "pace",
                RunsOut: b.PeriodStart.Add(time.Duration(float64(total) / projected * float64(b.LimitMicros))),
            })
        }
    }
    return out
}

The difference between those two kinds of finding is the difference between a metric and a warning. "You are at 80%" is a fact about the past. "You will run out on the nineteenth" is a statement about the future with a date attached, and a date is what makes somebody act.

One detail in there that's easy to skip: the pace finding is suppressed once you're past 95%. At that point the near-limit alert is already shouting, and two alerts about the same budget in the same minute is how people learn to filter your alerts into a folder.

Compare against your own ordinary day

The second signal worth building is a spike: spend today against what a normal day looks like for this account.

The temptation is to pick a number — alert above two hundred dollars a day. Don't. That number is a crisis for your smallest customer and invisible for your largest, and you'll spend the next year maintaining per-customer overrides for it.

// A spike is judged against this organisation's own ordinary day, never a
// number we picked. Ten dollars is a crisis for one customer and noise for
// another, and no threshold we choose is right for both.
func (d Detector) Spike(org string, now time.Time) *Finding {
    today := d.spend(org, now.Add(-24*time.Hour), now)
    days := d.dailyTotals(org, now.Add(-15*24*time.Hour), now.Add(-24*time.Hour))

    // Without a baseline there is nothing to compare against, and a new
    // account would otherwise alert on its first real day of use.
    if len(days) < 7 {
        return nil
    }
    ordinary := median(days)

    // The absolute floor matters as much as the ratio. Four times nothing is
    // still nothing, and an alert about it teaches people to ignore alerts.
    if today < ordinary*2 || today-ordinary < 5_000_000 {
        return nil
    }

    level := Notice
    if today >= ordinary*4 {
        level = Urgent
    }
    return &Finding{Level: level, Kind: "spend_spike", Ordinary: ordinary, Today: today}
}

Three things in there are the difference between a detector people trust and one they mute.

A median, not a mean. One bad day in the baseline window drags an average up enough to hide the next spike. The median ignores it.

A warm-up period. An account with four days of history has no ordinary day yet. Alerting on its first real day of usage is the fastest way to teach a new customer that your notifications are noise.

An absolute floor alongside the ratio. Four times nothing is still nothing. A quiet account that goes from twenty cents to eighty has quadrupled its spend and nothing has happened. Without that floor, your most-alerting customers will be the ones using the product least.

Failures are a spend signal, not just a quality one

This is the one almost nobody wires up, and it's the one we'd build second.

When a tool starts failing, you pay twice: once for the call that failed, and again for the retry. A tool that breaks quietly at two in the morning doesn't show up as a spend spike for hours, but it is already costing money and producing nothing — which is exactly the gap between cost per attempt and cost per success widening in real time.

The non-obvious part is that you have to judge each tool separately from the agent's model calls, and separately from each other.

An agent makes model calls on every single turn. It calls any given tool only on the steps that need it, so that tool's ordinary volume is a fraction of the total. Pool them together and a tool failing every single time disappears into a healthy overall rate. You need a lower call threshold for tools than for model calls, precisely because there are fewer of them to look at.

One exclusion worth making explicit: calls refused because a budget ran out are not failures. They're the system working, and the budget raises its own alert. Counting them means every account that hits its ceiling gets two alarms, one of which is wrong.

The two properties that decide whether anyone trusts this

Everything above is detection. Whether it's useful in month three comes down to two unglamorous properties.

One finding raises one alert, ever. Derive the alert's identity from what it's about — this budget, this period, this threshold — so that raising the same finding again collides with the one already there and writes nothing. Without it, a job that runs every fifteen minutes sends the same 95% warning ninety-six times a day, and the ninety-seventh is the one that actually mattered.

Hold down repeats. Even for genuinely distinct incidents, the same agent or the same tool shouldn't be able to alert more than once every few hours. Read the hold-down from the last stored alert rather than from a clock boundary, or every incident that starts at five to the hour alerts twice.

Both of these are about the same thing, which is that an alerting system has exactly one currency and it is attention. Every alert that didn't need sending is spent from the same account as the one that did.

Delivery is where this quietly fails

An alert that was detected and never arrived is worse than no alerting, because you believe you're covered.

So write the delivery record before attempting it, and have a sweep retry anything still outstanding. A process that dies mid-send then leaves evidence rather than silence. Keep the delivery's identifier stable across retries, so a receiver can tell your third attempt from a third incident — otherwise you've built a system that turns one outage into three alarms and an argument.

Give it a stop, too. Retrying a day-old alert helps nobody, and the channel it's going to may well have been deleted.

Two details that cause real arguments

Cut periods in the customer's timezone, not yours. A daily budget that resets at midnight UTC resets in the middle of the afternoon for some of your customers. They will report it as a bug, and they'll be right.

Decide whether a budget stops or only warns, per budget. Teams want a hard ceiling on some things and a heads-up on others, and conflating the two means someone eventually has their production traffic stopped by a limit they set as a reminder. Make it an explicit property, and make the alert wording differ, so "you've reached your limit" and "you've reached your limit and we've stopped" are never the same sentence.

The question worth asking

Not "do we have budget alerts." You do.

Ask instead: when did an alert last reach somebody while they could still do something about it?

If every alert you've ever sent arrived after the money was gone, you don't have an alerting system. You have a reporting system with an exclamation mark on it, and the difference only becomes obvious during the month you most needed the other thing.