Session affinity across replicas

A stdio server has no session problem, because the session and the process are the same object: one client, one child, one lifetime. Streamable HTTP separates those. Connections come and go, the conversation outlives them, and the server issues an identifier so that a sequence of independent HTTP requests can be recognised as one exchange. That identifier only means something to the process that minted it.

While there is one process, that limitation is invisible. Add a second and it becomes the dominant fact about the deployment, because a load balancer's default behaviour is to spread requests evenly and most requests in an MCP session are not the first one. Everything in this chapter follows from that: what a session actually holds, the three ways to make a later request land somewhere that knows about it, and what a rolling deploy costs once sessions are hours long.

The belief that gets carried into the second replica is that a server which never wrote anything to a database must be stateless, and it is tempting for a good reason: there is no save call anywhere in the code, and nothing that looks like persistence was ever designed. What makes it false is that state is not defined by where something was written down. It is defined by whether an answer depends on something that happened earlier in the same conversation. A handler reading a dictionary that initialize populated depends on an earlier event exactly as much as one reading a row from a table. The dictionary simply happened to be in the same process on every request, which was a property of running one copy rather than a property of the code, and it stopped being true the moment a manifest said three.

What a session is actually holding#

The identifier is worth being concrete about, because "the session" sounds abstract until you enumerate what sits behind it.

The negotiated protocol version and the capabilities each side declared live there, so a replica that never saw the handshake does not know whether this client can accept server-initiated sampling requests or whether it asked for listChanged notifications. The event buffer that makes resumption work lives there, keyed by the event ids from the streams this session has opened. Any subscriptions the client took out — resources/subscribe on a file or a table — live there, along with the machinery watching for changes. Progress tokens for calls still running live there, since a progress notification has to be correlated with the request that is going to answer. If the deployment authenticates callers, the validated subject and any token exchange performed on its behalf are cached there rather than re-derived on every request. And in most implementations there is a per-session workspace: a checked-out revision in a temp directory, an open database connection, a warm index.

None of that is unusual, and none of it is replicated anywhere by default. A second replica receiving a request for a session it never opened has no way to construct any of it, which is why returning 404 is the correct behaviour rather than a bug. What is a bug is a deployment that produces the situation.

The step that makes 404 correct rather than unhelpful is worth naming, because the first reaction is to look for the defect in the replica that answered. The identifier is not a description of a session; it is a key into a map that lives in one process's heap, and a key means something only where its map is. A replica that never ran the handshake has no entry under that key, and — this is the half that decides it — no way to manufacture one. Every item in the list above was either negotiated with a client this process never spoke to or produced by work it never did. Answering anything other than 404 would mean inventing a session: guessing which capabilities the client declared, starting an empty event buffer that cannot serve any resume request, and opening a workspace at a revision nobody named. The honest reply to "continue a conversation you were not part of" is that there is no such conversation here.

The edge that catches people is the one from Active to Terminated on a rollout, because it is drawn by the platform rather than by either party to the conversation.

Three ways to make the second request land somewhere useful#

The options are session affinity, meaning the load balancer is configured to send every request bearing a given session identifier back to the replica that issued it; moving the state somewhere every replica can reach; or arranging for there to be no state.

Affinity is the cheapest thing that works and it is where most deployments start. The complication is that MCP carries its session in a header rather than a cookie, and a good deal of load balancer stickiness is cookie-based on the assumption that the client is a browser. AWS Application Load Balancer stickiness sets an AWSALB cookie, which an MCP client written with a plain HTTP library will not store or return. What you need instead is consistent hashing on the Mcp-Session-Id header.

# ingress-nginx
metadata:
  annotations:
    nginx.ingress.kubernetes.io/upstream-hash-by: "$http_mcp_session_id"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"

# Envoy: ring hash on the same header
route:
  cluster: repo_index
  hash_policy:
    - header: { header_name: "mcp-session-id" }

Consistent hashing rather than a simple modulus matters here, because a modulus over the replica count remaps every session when the count changes, and the count changes whenever the autoscaler acts. A ring hash remaps roughly 1/n of sessions when one replica is added or removed, which turns a fleet-wide session reset into a small one.

The difference between the two schemes is worth seeing rather than accepting. A modulus assigns a session by taking the hash of its identifier modulo the replica count, so the answer depends on the count itself; change the count and nearly every session's answer changes with it, because the divisor moved underneath all of them simultaneously. A ring hash instead places both the sessions and the replicas on one circle of hash values and gives each session to the first replica clockwise of it. Adding a replica inserts a single new point on that circle. The only sessions that move are the ones lying in the arc between the new point and whichever point used to serve them; for every other session the first replica clockwise is exactly where it was. That arc is one replica's share of the circle, which is where the 1/n comes from, and it is the whole reason a fleet that autoscales through the day is survivable under one scheme and not the other.

The shared store moves the state instead. The event buffer, the negotiated capabilities and the subscription list go into Redis keyed by session id, with a TTL a little longer than the session idle limit, and every replica reads them on arrival. This is the arrangement that lets a client resume a dropped stream on a different pod than the one it started on, which is the combination the previous chapter's resumption story quietly assumed.

async def load_session(session_id: str) -> Session | None:
    raw = await redis.get(f"mcp:sess:{session_id}")
    if raw is None:
        return None
    await redis.expire(f"mcp:sess:{session_id}", SESSION_TTL)
    return Session.model_validate_json(raw)

async def append_event(session_id: str, event_id: int, message: dict) -> None:
    key = f"mcp:evt:{session_id}"
    await redis.xadd(key, {"id": event_id, "msg": json.dumps(message)})
    await redis.xtrim(key, maxlen=512, approximate=True)
    await redis.expire(key, RESUME_WINDOW)

Not everything moves cleanly. A per-session temp directory holding a checked-out revision does not go into Redis, and a warm in-memory index of a 42,000-file repository does not either. Deployments that need those end up with affinity as the fast path and the store as the fallback: hash on the session header so a client normally returns to its own replica, and keep enough in the store that landing elsewhere degrades into a slower first call rather than a 404.

Stateless mode is the third option and it is a real one. Several server frameworks offer a flag that issues no session id at all, treats every POST as complete in itself, and answers with plain JSON rather than a stream. Deployed that way an MCP server behaves like any other stateless HTTP service: any replica answers any request, rolling deploys are invisible, and horizontal scaling is a number in a manifest. What you give up is everything that needed continuity — resumption, subscriptions, server-initiated sampling requests in which the server asks the client to run a model inference for it, elicitation requests in which the server asks the client to put a question to the human, and progress notifications on a held-open stream. For a server whose tools are all short, idempotent lookups, that is a good trade and worth taking deliberately rather than discovering.

One process, many callers, one event loop#

Concurrency arrives with the same change and is easy to overlook because it does not produce an error. Under stdio the server handles one client, and a handler that blocks the event loop for 900 ms — a synchronous subprocess.run of ripgrep inside an async def, a CPU-bound ranking pass, a psycopg2 call without an async driver — delays that one user by 900 ms. The same handler in a shared deployment holding 60 concurrent sessions delays all of them, because it is one process and one loop. Latency that was invisible in development becomes a queue in production, and it shows up as unrelated calls taking 3 seconds while the profiler says every handler is fast.

Both halves of that are true at once, which is why the profiler is neither lying nor helping. A single-threaded event loop advances one coroutine at a time and switches between them only where a coroutine yields. A synchronous call inside an async def never yields, so for the 900 ms it runs, nothing else in the process moves: not the other sessions' handlers, not the code writing to their response streams, not the heartbeat generator from the previous chapter. Requests continue to arrive throughout and simply wait. Measure any individual handler afterwards and it is fast, because the measurement covers the time that handler was running rather than the time its call spent queued behind somebody else's. The number that would expose it is the gap between a request arriving and its handler starting, and that is not a number a function-level profiler is built to report.

Memory is the other resource that stops being free. Sixty concurrent sessions each holding a temp checkout, a pooled connection and a 512-event resume buffer is a working set that a single-user process never had to think about, and the failure when it is exceeded is an out-of-memory kill by the kernel that takes every session on the pod rather than a slow request. The metrics that make this legible are small in number and worth exposing from the first day: the count of live sessions, the age distribution of those sessions, the rate of 404 responses to requests bearing a session id, and the number of resume attempts that found their event buffer. A rising 404 rate with a flat error rate everywhere else is the signature of affinity that has stopped working, and it is otherwise invisible because each individual client recovers by re-initializing.

The bill that arrives with a rolling deploy#

A deployment that ships four times on a normal day is a non-event for a stateless service: pods are replaced one at a time, in-flight requests finish inside the grace period, and nobody notices. For a session-bearing service the arithmetic is different, because an editor session is open for as long as the editor is, which in practice means something like 3 to 7 hours. With sessions that long and deploys that frequent, no drain window — the interval a pod spends refusing new work while it finishes what it already has — is long enough to let sessions end naturally, and every deploy terminates essentially every open session.

The two figures settle that without anybody having to exercise judgement. Four deploys spread across a working day leave roughly a couple of hours between one and the next, and the shortest editor session is longer than the whole gap. So at the moment any deploy starts, essentially every session in existence was opened before the previous deploy and would still be open after the next one. Draining is supposed to let in-flight work finish, and finishing here means waiting for somebody to close their editor, which is a wait measured in hours against a grace period measured in minutes. The configuration below does not close that gap, because at these two timescales the gap cannot be closed. What it changes is the kind of failure.

lifecycle:
  preStop:
    exec: { command: ["/bin/sh", "-c", "kill -TERM 1; sleep 120"] }
terminationGracePeriodSeconds: 150

That configuration stops new sessions from being routed to the pod, lets in-flight tool calls finish, and then kills whatever remains. It converts a hard failure for everyone into a clean failure for the sessions that were merely open and idle, and those clients re-initialize. Whether that is invisible or catastrophic depends entirely on how the client handles a 404 on a session it believed in — which is why the specification's rule that a client receiving 404 must start a new session with a fresh initialize is worth implementing carefully on both sides. A client that treats it as a fatal transport error surfaces "connection lost" to 200 people four times a day.

The autoscaler creates the same event more quietly. Scaling from six replicas to four at 18:00 because traffic dropped will terminate the sessions on the two pods that go, and with consistent hashing it also remaps a share of the surviving sessions to replicas that never had them. A deployment whose sessions cost nothing to rebuild can autoscale freely; one that keeps a warm 8 MB index per session should be told to scale down slowly, or not at all during working hours, because the saving is a fraction of a node and the cost is a visible interruption for whoever was on those pods.

The realistic posture is to make sessions cheap to lose. Keep expensive per-session state to a minimum, make re-initialization fast, avoid making the client's user-visible state depend on continuity that a pod restart can revoke, and deploy less often during working hours if none of that is achievable.

This is where the choice from the first chapter of this topic gets its operational teeth. Concurrency and scaling are deciding criteria rather than deployment details, because they change what the code has to be: a stdio server may block freely and nobody but its owner suffers, while a shared server needs its blocking work in a thread pool or a worker process and needs to be measured under concurrent load rather than one call at a time.

The lifetimes tell the same story from the other side. A stdio server exists exactly as long as the client that spawned it, so its session and its process are the same object — there is no session store because there is nothing to store between, no affinity because there is one destination, and no deploy that can interrupt a session because upgrading means replacing a binary that will be picked up next time the editor starts. An HTTP server outlives every session it serves and expects to be replaced while sessions are open, which is why all of the machinery in this chapter exists at all. Choosing a transport is choosing which of those two lifetime models you are signing up to operate.

All of this assumed the server is somewhere else — a fleet, an ingress, a DNS name that other people resolve. The shortcut that keeps coming back is to run this same HTTP build on the developer's own machine, where none of the affinity or deploy questions apply, and the bill for that one is not operational.