Microservices that survive contact with production

"Microservices" went through the hype cycle and came out the other side with a bad reputation, most of it earned. The classic failure is well documented: a team splits a monolith into twelve services, keeps the shared database, keeps the shared release train, and ends up with a distributed monolith — all of the coordination cost, none of the independence.

We run several products as service fleets and the split does earn its keep, but only because of a set of rules that are, individually, boring. Nothing below is novel. All of it is the difference between a system you can hand to an auditor and one you're afraid to deploy on a Friday. Nine practices, in the order they matter.

1. The contract is a repository, and it ships before the code

Our API definitions don't live inside the services that implement them. They live in a separate repository of nothing but protobuf, and pushing to it regenerates and publishes versioned Go modules and TypeScript packages. A service then depends on its own API the way it depends on any other library — by version.

Two things follow. First, an API change gets a review surface of its own: you can read what the contract is becoming without reading anybody's implementation, which is when interface mistakes are cheap to fix. Second, a proto edit and the code that uses it become two steps with a wait between them. That feels like friction for about a week, and then it's just how you work.

The rule that makes versioned contracts survivable is backwards compatibility as an absolute: never remove or rename a field, ever. Deprecate it, reserve the number, add a new one. A consumer you forgot about is always still running.

2. Resource-oriented, not endpoint-oriented

Endpoint-shaped APIs drift because there's nothing to drift from: every team invents its own verbs, its own paging, its own idea of what an id looks like. Resource-oriented design (Google's AIP guidelines are the clearest write-up) fixes the shape once. Resources get hierarchical names, and every service exposes the same standard method set over them.

// A resource, not an endpoint. The name is server-minted and hierarchical:
// orders/{uuid}. Every mutable resource carries an etag.
message Order {
  string name = 1;         // orders/{uuid} - never client-supplied
  string customer = 2;     // customers/{uuid}
  int64 total_cents = 3;   // money is always integer minor units
  OrderStatus status = 4;

  string etag = 97;
  google.protobuf.Timestamp create_time = 98;
  google.protobuf.Timestamp update_time = 99;
  google.protobuf.Timestamp delete_time = 100;
}

service OrdersService {
  rpc CreateOrder(CreateOrderRequest) returns (Order);
  rpc GetOrder(GetOrderRequest) returns (Order);
  rpc UpdateOrder(UpdateOrderRequest) returns (Order);            // update_mask + etag
  rpc ListOrders(ListOrdersRequest) returns (ListOrdersResponse); // always paginated
  rpc DeleteOrder(DeleteOrderRequest) returns (Order);            // soft delete
  rpc UndeleteOrder(UndeleteOrderRequest) returns (Order);
}

Three details in there earn their place. Ids are minted server-side, so a client can never choose a resource name and collide with — or overwrite — someone else's. Money is an integer count of minor units, because a float will eventually cost somebody a cent and an afternoon. And the metadata fields sit at the far end of the field-number space, so the domain fields have room to grow without a renumbering discussion.

The payoff isn't aesthetic. When every service has the same shape, an engineer who has worked on one is productive on the next that afternoon, and generic tooling works everywhere — including turning any RPC into an agent tool without hand-writing a schema.

3. One door, and nothing else is reachable

Exactly one service in the fleet is on the public internet: the gateway that fronts the SPA. It verifies the caller's bearer token — signature and expiry, against the identity service's published keys — then forwards the verified identity to the backends as a request header. No backend accepts traffic from anywhere else.

That is worth stating as an obligation rather than an assumption, because it is exactly the kind of thing that quietly stops being true. The ingress settings in a service's Terraform are part of its security design, not plumbing; a service made publicly reachable "temporarily, for testing" invalidates the reasoning behind every backend that trusts the forwarded header.

4. Verify what you forward

Which brings us to the trap. "The gateway verified it" turns, over a few sprints, into "backends trust a header" — and many identity helper libraries will happily decode a forwarded token without checking its signature. Decode-only is fine exactly as long as the network boundary holds, and network boundaries are one misconfigured ingress away from not holding.

The fix is small enough that there's no excuse for skipping it: an interceptor in each service that verifies the forwarded token against the identity service's key set. Roughly forty lines, written once, applied to every RPC on the server.

// Runs before authorization. A request with no forwarded token is a
// system-to-system call with no asserted user - that path is governed by
// platform IAM, not by this check.
func verifyForwarded(ctx context.Context) error {
    tok := forwardedToken(ctx)
    if tok == "" {
        return nil
    }
    if err := jwks.Verify(tok); err != nil {
        return status.Error(codes.Unauthenticated, "forwarded identity failed verification")
    }
    return nil
}

grpcServer := grpc.NewServer(
    grpc.ChainUnaryInterceptor(verifyForwardedInterceptor, authzInterceptor),
)

Any sentence in your architecture that begins "we can trust this because the network guarantees" deserves a second control behind it.

5. Every RPC authorises, and the policy is something a human can read

Authentication and authorisation get conflated constantly, and the bug it produces is always the same: a service that carefully establishes who is calling and then never asks whether they may. Every RPC needs an answer. Two return codes, kept distinct: Unauthenticated when there's no valid identity, PermissionDenied when there is one and it isn't enough.

What has survived audits for us is a declared table — which methods need which role, checked in an interceptor, with owner-scoped resources checked inside their handlers instead. One place to read the policy, one place to change it.

The table needs one more property to be worth having: default deny. A default-open table — where a method nobody listed stays callable by any authenticated user — fails in the way that is hardest to notice, because a new RPC ships wide open the moment somebody forgets a line. An unclassified method should refuse everyone, so the gap surfaces the first time you call it rather than the first time somebody else does.

// Classification is mandatory: adminMethods needs a role, ownerScoped methods
// check ownership in the handler. Anything in neither map answers nobody.
var adminMethods = map[string]bool{
    "CreateOrder": true, "UpdateOrder": true, "DeleteOrder": true, "ListOrders": true,
}

func authzInterceptor(ctx context.Context, req any, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (any, error) {
    method := info.FullMethod[strings.LastIndex(info.FullMethod, "/")+1:]
    caller, err := resolveCaller(ctx)
    if err != nil {
        return nil, status.Error(codes.Unauthenticated, "no caller identity")
    }
    if adminMethods[method] {
        if !caller.IsAdmin() {
            alog.Warnf(ctx, "denied %s for %s", method, caller.ID) // id, never email
            return nil, status.Error(codes.PermissionDenied, "admin role required")
        }
    } else if !ownerScoped[method] {
        return nil, status.Error(codes.PermissionDenied, "method not classified")
    }
    return handler(context.WithValue(ctx, callerKey, caller), req)
}

6. Etags, not last-write-wins

Two people open the same record. Both save. In a service that just writes what it was handed, the second save silently erases the first person's edit and nobody ever finds out. It's the most common data-loss bug in business software and it costs about fifteen lines to remove.

Every mutable resource carries an etag, regenerated on every write. An update sends the etag it read; a mismatch means somebody wrote in between, and the correct response is to refuse. Pair that with a field mask so a client only sends what it actually changed, and concurrent edits to different fields stop fighting each other.

func (s *server) UpdateOrder(ctx context.Context, req *pb.UpdateOrderRequest) (*pb.Order, error) {
    if err := requireRole(ctx, pb.Role_ORDER_ADMIN); err != nil {
        return nil, err
    }

    v := validation.NewValidator()
    v.String("name", req.GetOrder().GetName()).IsPopulated().Matches(regex.Order)
    v.MessageIsPopulated("update_mask", req.GetUpdateMask() != nil)
    if err := v.Validate(); err != nil {
        return nil, status.Error(codes.InvalidArgument, err.Error())
    }

    row, err := db.Orders.Read(ctx, req.GetOrder().GetName())
    if err != nil {
        return nil, err
    }
    if !req.GetUpdateMask().IsValid(new(pb.Order)) {
        return nil, status.Error(codes.InvalidArgument, "invalid update mask")
    }
    if row.GetResource().GetEtag() != req.GetOrder().GetEtag() {
        // Someone wrote since this client read. Their edit stands; this one retries.
        return nil, status.Error(codes.FailedPrecondition, "etag does not match")
    }

    row.Merge(req.GetOrder(), req.GetUpdateMask().GetPaths()...)
    row.GetResource().UpdateTime = timestamppb.Now()
    row.GetResource().Etag = uuid.New().String()
    if err := row.Update(ctx); err != nil {
        return nil, status.Errorf(codes.Internal, "update order: %s", err)
    }
    return row.GetResource(), nil
}

7. Bound everything — and then loop on the client

Every list paginates, with a default page size and a hard cap, and the client's requested size is clamped rather than passed through. An unbounded list is a service that works beautifully until the day one customer has forty thousand records, and then takes the instance down with it.

The subtler half of that contract is on the reading side, and it bites harder. A bounded API hands back one page and a token; a client that takes the page and ignores the token gets a plausible-looking answer that is silently missing almost everything. We shipped an internal dashboard that reported 26 records where the store held 4,622, for exactly that reason. Nothing errored. The number just looked small enough to believe and wrong enough to matter, and it sat there for weeks. Write the drain loop once, in a helper, and use it everywhere.

func clampPageSize(requested int32) int32 {
    switch {
    case requested <= 0:
        return defaultPageSize // 50
    case requested > maxPageSize:
        return maxPageSize     // 1000, always
    default:
        return requested
    }
}

// The other half of the contract. Stopping at page one is a bug that looks
// like data, which is the worst kind.
func listAllOrders(ctx context.Context, req *pb.ListOrdersRequest) ([]*pb.Order, error) {
    var all []*pb.Order
    for {
        resp, err := clients.Orders.ListOrders(ctx, req)
        if err != nil {
            return nil, fmt.Errorf("list orders: %w", err)
        }
        all = append(all, resp.GetOrders()...)
        if resp.GetNextPageToken() == "" {
            return all, nil
        }
        req.PageToken = resp.GetNextPageToken()
    }
}

8. Long work leaves the request path

Anything that can outlast a request deadline — indexing a document set, a bulk import, a model job — must not be attempted inside the request. The pattern: start the work, store its operation handle on the row, return immediately. A task queue then calls a handler back to poll it to completion and write the result. Locally the same closure runs in-process, so nobody needs a queue on their laptop to develop the feature.

One invariant makes this safe: every background handler must be idempotent. A task queue will deliver the same message twice eventually — that isn't a failure, it's the delivery guarantee — and a handler that appends a row per delivery will quietly double your data.

9. The service owns its data, its infrastructure and its deploy

This is the one that separates real service boundaries from a monolith with extra network hops. A service owns its storage and no other service reads it — cross-service reads go through the API, always, even when reaching into the other database would be one line and nobody would notice. Shared tables are how two services quietly become one service that happens to run twice.

Infrastructure lives with the code it provisions: a folder of Terraform inside the service, one concern per file, deployed by the team that owns the service. And commits stay scoped to one service — a mid-history build break costs an afternoon, while a commit bundled across four services costs you the ability to revert or bisect any of them.

Logging deserves its own line. Levelled, structured, and free of personal data: log opaque resource ids, never emails, names, tokens or whole request payloads, at any level. The one place a person's details legitimately belong is the audit record — which is a first-class, immutable artefact, not a log line you hope is still in retention when somebody asks who did what.

The through-line

Read the nine back and they're the same idea nine times: put the contract, the boundary and the invariant somewhere a person can read, then let the compiler or an interceptor enforce it, so being correct doesn't depend on every engineer remembering every rule at 6pm on a Friday. That's what makes a dozen services feel like one system instead of twelve opinions — and it is how the custom software we build is put together, not a standard we describe and skip.

It's also what makes a fleet safe to point an agent at. Typed contracts, one identity that travels, per-RPC authorisation and an audit trail are exactly the properties an autonomous caller needs before it can be trusted with real actions — which is why the teams that did this work first tend to be the ones whose agent pilots reach production.