Concurrency-safe isn't tenant-safe - the shared state hiding in your agent

A tool finishes its work and produces something the user should see — a generated spreadsheet, a preview of an edit, a modal the frontend needs to open. The model doesn't need that payload; it needs a sentence about what happened. The transport needs it, so the response loop can stream a card to the browser alongside the text.

Those two things are in different places. The tool returns to the agent framework, which returns to the model. The response loop is somewhere else entirely, holding the event queue. There is no obvious channel between them — so you write the obvious thing: a package-level variable the tool sets and the loop drains. It works immediately. It survives review. It passes go test -race.

And then one day two customers are on the same instance at the same second, and one of them sees the other's data.

The lock is correct. That's the problem.

Here is the shape, and the reason it gets waved through: there is no data race in it. The mutex is real, the critical section is right, the race detector is silent. A reviewer looking for concurrency bugs finds none, because in the Go sense there aren't any.

// The shape that passes review and leaks in production. The lock is real,
// the lock is correct, and it protects the wrong thing: access to ONE slot
// that every concurrent turn in this process shares.
var (
    pendingArtifact   *ArtifactPayload
    pendingArtifactMu sync.Mutex
)

func setPendingArtifact(p *ArtifactPayload) {
    pendingArtifactMu.Lock()
    pendingArtifact = p
    pendingArtifactMu.Unlock()
}

func getAndClearPendingArtifact() *ArtifactPayload {
    pendingArtifactMu.Lock()
    defer pendingArtifactMu.Unlock()
    r := pendingArtifact
    pendingArtifact = nil
    return r // whichever turn drains first takes it, whoever wrote it
}

A mutex serialises access to one shared slot. It does not give each request its own slot. Those sound like the same guarantee and they are nothing alike — the first is about when two goroutines touch a value, the second is about whether they should be touching the same value at all.

Correct locking around shared state is not the same as not sharing state. The race detector can only tell you about the first.

The absence becomes visible on the read side. The response loop holds a request context that identifies exactly whose turn this is. The payload it drains carries its own identity. At no point does anything compare the two:

// The read side is where the absence shows. reqCtx belongs to this turn;
// the payload belongs to whichever turn wrote last. Nothing compares them.
if p := getAndClearPendingArtifact(); p != nil {
    ex.sendStatusUpdate(ctx, q, reqCtx, a2a.TaskStateWorking,
        "ARTIFACT_CREATED:"+mustJSON(p), false)
}

Why agents make this worse than an ordinary server

Package-level state in a request-handling process is an old mistake, and most web frameworks have trained it out of people. Agent servers reintroduce it, for four reasons that compound.

• Turns are long. A request that lives for twenty minutes overlaps with far more other requests than one that lives for 200 milliseconds. The window isn't a window; it's most of the runtime.
• The handoff is genuinely awkward. A tool's return value goes to the model, not to the transport. Frameworks rarely give you a first-class way to say "also send this to the client" — so the global is a workaround for a real gap, which is why smart people write it.
• Scale-to-zero packs tenants together. Agents are bursty and idle, so platforms are tuned to keep few instances warm and put many concurrent requests on each one. The deployment strategy that makes agents affordable is the same one that puts two tenants in a single process.
• Nothing fails. No error, no denial, no alert. The authorization layer was never consulted, because no RPC was made — the data was already inside the process. It simply arrives in the wrong stream, and the only way anyone finds out is a human noticing something that isn't theirs.

That last point is what makes this a security bug rather than a correctness bug. Every control described in letting an agent act as you, safely — forwarded identity, per-user authorization, the audit trail — sits on the RPC path. A payload that crosses between two goroutines never goes near it.

The fix: key the handoff to the turn that produced it

The slot exists to carry a payload from a tool call to the response loop of the same turn. That relationship is the thing the code never states. State it, and the bug is structurally gone.

The key is already available at both ends — every request carries a task id, tools reach it through their tool context, the executor holds it on the request context. Nothing new needs plumbing:

// The turn id is already in the context at both ends: tools receive it
// through their tool context, the response loop holds it on reqCtx. No new
// plumbing is needed to key the handoff — only the decision to do it.
func turnKeyFromContext(ctx context.Context) string {
    if t := taskIDFrom(ctx); t != "" {
        return t
    }
    return "" // no key means no handoff — never fall back to a shared slot
}

Note the empty-string branch. When no key is available the payload is dropped, never parked in a fallback slot. A shared fallback reintroduces the entire bug for exactly the edge cases nobody tests, and a missing card is a support ticket while a leaked one is a breach.

Then the store itself. A generic keyed container replaces every hand-rolled global, and registers itself so the cleanup path stays exhaustive:

// turnStore is the shared slot with an owner. Payloads are filed under the
// turn that produced them and are only ever visible to that turn's drain.
//
// Registering every store in one package-level list is what makes release
// exhaustive: a store added next year is swept by the same defer, so the
// memory-leak half of this fix cannot be forgotten at the call site.
type turnStore[T any] struct {
    name string
    mu   sync.Mutex
    m    map[string][]T
}

func newTurnStore[T any](name string) *turnStore[T] {
    s := &turnStore[T]{name: name, m: make(map[string][]T)}
    turnStoresMu.Lock()
    turnStores = append(turnStores, s)
    turnStoresMu.Unlock()
    return s
}

func (s *turnStore[T]) replace(ctx context.Context, v T) {
    key := turnKeyFromContext(ctx)
    if key == "" {
        return // drop rather than leak
    }
    s.mu.Lock()
    defer s.mu.Unlock()
    s.m[key] = []T{v}
}

func (s *turnStore[T]) takeOne(turnKey string) (T, bool) {
    var zero T
    if turnKey == "" {
        return zero, false
    }
    s.mu.Lock()
    defer s.mu.Unlock()
    vs := s.m[turnKey]
    if len(vs) == 0 {
        return zero, false
    }
    delete(s.m, turnKey)
    return vs[0], true
}

The generic matters more than it looks. Four hand-written slots are four chances to get the keying subtly different; one turnStore used four times has one implementation to review. It's the same argument as writing the pagination drain loop once — a rule that lives in a helper holds, and a rule that lives in everyone's memory doesn't.

Sweep on turn exit, or you've traded a leak for a leak

A keyed map that is only drained on the happy path grows by one entry for every turn that errors, times out, or has its browser tab closed. The fix has to include the release, and the release has to be exhaustive — which is what the store registry buys:

// Every turn ends exactly once, and every store is swept when it does.
// Without this the map grows by one entry per errored or timed-out turn,
// and a tenant-safety fix quietly becomes a memory leak.
func releaseTurn(turnKey string) {
    turnStoresMu.Lock()
    stores := append([]turnReleaser(nil), turnStores...)
    turnStoresMu.Unlock()

    for _, s := range stores {
        s.release(turnKey)
    }
}

// In the executor:
func (ex *Executor) Execute(ctx context.Context, reqCtx *RequestContext, q EventQueue) error {
    defer releaseTurn(reqCtx.TaskID)
    // ...
}

One defer in the executor, and a store added six months from now is swept by it without anyone remembering to wire it up.

Then check scope at the boundary anyway

Keying the store fixes this bug. Asserting scope at the send boundary is what contains the next one — some future handoff that finds a new way to be wrong. Before anything is written to the wire, the payload's tenant is compared against the turn's resolved scope:

// Defence in depth. Even with keyed stores, a payload whose project or
// organisation disagrees with the turn's resolved scope is refused at the
// wire rather than rendered. Turns the next bug of this shape into a
// dropped message and a loud log line instead of a disclosure.
func payloadInTurnScope(ctx context.Context, scope *ConversationScope, label, payloadProject string) bool {
    if scope == nil || payloadProject == "" {
        return false
    }
    if normalizeProjectName(payloadProject) != normalizeProjectName(scope.Project) {
        alog.Errorf(ctx, "%s payload rejected: out of turn scope", label) // ids only
        return false
    }
    return true
}

This is deliberately redundant with the keyed store, and that is the point. Defence in depth turns a disclosure into a dropped message and a log line loud enough to page someone. It is also the only part of this that keeps working if a future refactor quietly reintroduces a shared slot.

Note what the log line carries: a label and the fact of rejection, never the payload. A control that fires on tenant confusion must not write the confused data into a log that a wider group can read.

The test that makes it un-regressable

Every fix in this post is invisible to the type checker. Nothing stops someone reintroducing a package-level slot next quarter, and no reviewer reliably catches it — this class of bug survives review precisely because it looks correct. So the fix isn't finished until a test encodes it:

// This test fails on the package-level version and passes on the keyed one.
// Once it exists, the bug cannot come back quietly.
func TestTurnStore_NoCrossTurnDelivery(t *testing.T) {
    store := newTurnStore[*ArtifactPayload]("test")

    ctxA := withTaskID(context.Background(), "turn-a")
    ctxB := withTaskID(context.Background(), "turn-b")

    var wg sync.WaitGroup
    wg.Add(2)
    go func() { defer wg.Done(); store.replace(ctxA, &ArtifactPayload{Owner: "a"}) }()
    go func() { defer wg.Done(); store.replace(ctxB, &ArtifactPayload{Owner: "b"}) }()
    wg.Wait()

    gotA, okA := store.takeOne("turn-a")
    gotB, okB := store.takeOne("turn-b")

    if !okA || gotA.Owner != "a" {
        t.Fatalf("turn A drained %v, want its own payload", gotA)
    }
    if !okB || gotB.Owner != "b" {
        t.Fatalf("turn B drained %v, want its own payload", gotB)
    }
}

Two turns, interleaved, each asserting it received only its own payload. Run it against the package-level version and it fails. That is the whole value: this bug cannot return silently once the test exists.

What doesn't fix it

The tempting operational shortcut is to stop putting concurrent turns on one instance — --concurrency=1, or a cap on instances. Resist it.

It narrows the window without closing it: two turns on one instance is all this needs, and a container that serves one request at a time still serves the next one from the same memory if the slot was never cleared. It costs a great deal, because it forfeits exactly the packing that makes agent hosting affordable — the economics described in what an AI agent actually costs. And it is one deploy away from being reverted by someone tuning throughput who has no idea a correctness guarantee is hiding in a concurrency flag.

A tenant boundary enforced by a deployment setting is not a tenant boundary. It is a coincidence with good uptime.

The general shape

Go to your agent and grep for package-level mutable state. Not just the obvious caches — the handoff slots, the "pending" anything, the singletons a framework handed you whose partitioning you have never actually verified. For each one, ask a single question: if two tenants hit this at the same second, which one owns what's inside?

If the answer is "whichever got there last," you have this bug, whether or not it has fired yet.

The deeper pattern is the one running under everything we write about agents. A convention people have to remember — don't put request state in globals — fails eventually, because it depends on every engineer holding it at 6pm on a Friday. A type that requires a turn key at both ends holds by itself. Put the constraint in the architecture, where the compiler and the test suite enforce it, rather than in the prompt, the code review, or the deployment flag, where it is only ever a suggestion. It's why we build agents in a language that makes ownership explicit, and why the boring structural decisions are the ones that decide whether an agent is safe to run for more than one customer at a time.