Rehearse the failures before the new store is on the critical path

The page fired at 03:14: retrieval p99 above two seconds, sustained for five minutes. The engineer on call had four years of Postgres behind her and had never opened the new store's console. She spent the first twenty-two minutes finding a dashboard, because the vector store's metrics lived in the vendor's UI rather than in the company's Grafana, and the link in the alert pointed at the application's own service dashboard. Another forty minutes went into reading documentation to learn that a cluster — several machines sharing one collection, which is how these products are deployed at any size — had entered a read-only state when one node crossed its memory limit. That is not a state Postgres has, and not one she had any reason to expect. The store did have a runbook, meaning a document an on-call engineer opens to find out what a symptom means and what to do about it, and it consisted of a Notion page titled "Vector store — TODO" created during the migration. She escalated at 04:30 to the engineer who had run the cutover, who was on holiday and answered at 05:50. Service was restored at 06:24, three hours and ten minutes after the page, against a team median of twenty-five minutes for incidents on the primary database.

Nothing about that timeline is a criticism of the engineer. She was competent at the systems she had been given the chance to become competent at, and she was handed a new one at 03:14 with no runbook, no dashboards in the place she looks, and no prior exposure. The decision to adopt the store had been made on the basis of latency and build time. The cost that materialised was three hours of downtime and a rotation that now has a system on it which one person understands.

Adoption is a staffing decision wearing a technical costume#

The measurements in chapter 3 and the two lists in chapter 4 are about what the software does. This one is about who will be awake with it. A second datastore adds a diagnostic surface — its own failure states, its own metrics, its own vocabulary, its own upgrade procedure — and that surface has to be held in the heads of everyone on the rotation, not in the head of the person who introduced it. The single-expert configuration is stable right up until that person is asleep, on leave, or has changed jobs, and it is worth noticing that the second and third of those are certainties rather than risks.

There is a way to find out whether the surface is affordable before it is load-bearing, and it is the same technique that works for any operational unknown: run the failure on purpose, with the people who will actually be paged, and see what happens. Calling that the only instrument is a strong claim, and it follows from what a failure path is. A failure path is a stretch of behaviour — in the code, in the operator's hands, in the vendor's product — that executes only in states the system is designed to avoid. Every other instrument you have observes states the system reaches on its own. Monitoring reports what production visited. A load test exercises the success path harder. A design review reads the intended behaviour out of the design. None of those produce the failure state, so none of them can report what happens in it.

The distinction that makes this more than wordplay is between questions the design answers and questions only the deployed artifact answers. Whether the architecture is supposed to fall back to lexical search is a design question, and a review settles it. What the HTTP client's default timeout is in the version currently running, whether the dashboard exists, whether the credentials in the runbook still work, and how long a restore takes on production-sized data are all questions whose answer lives in the deployed system, and a review cannot read them because they are not written in the design. That is what a drill obtains and a review cannot.

Two things about a drill are choices rather than consequences, and both are worth making on purpose. Running it on production-sized data is not required for the drill to complete, and without it the restore figure means nothing, because restore time scales with what is being restored. Running it with the on-call rotation rather than with the engineer who built the store is also not required, and it is the choice that converts the exercise from a test of the software into a test of the team — which is the variable the decision record from chapter 3 recorded as a sentence and never tested.

Five drills, and what each one is really measuring#

The right-hand column is the point, and the reason it is the point is that it contains almost no information the middle column did not already carry. Read the first row across: the pass condition is that on-call finds the cause from existing dashboards and runbooks, and the failure statement is that your observability lives in the vendor's console. Those are one sentence written twice, once as a condition on the exercise and once as a property of your system. The fourth and fifth rows do the same, and so does the second. The third column falls out of the second, which is what makes the table something a reader can extend: given any drill you can state its pass condition, and negating it and re-attributing it to your own system writes the row you did not have.

One row breaks the pattern, and the break is the informative part. The version upgrade row's failure statement adds a horizon the pass condition does not mention — upgrades get deferred and you run an unsupported version within eighteen months. That is a consequence which accrues rather than one which simply exists. The other four failures are true from the moment the drill fails; this one gets worse while nobody looks at it, which is why it is the drill teams are most likely to skip and most likely to regret skipping.

Each of these is a question about your own system that has a definite answer, and the drill is simply the cheapest way to obtain it. Running all five takes a team about a week, spread over a sprint, which is roughly the same cost as the shadow-read phase of the migration and considerably less than one three-hour outage.

What a drill leaves behind#

The output of a drill is not a pass mark; it is a set of artifacts that did not exist before, and those artifacts are most of the value. The diagnosis drill produces the dashboard, because the first thing an engineer does when she cannot find one is build one, and it produces it in the place your team actually looks rather than in a vendor console behind a separate login. It produces an alert that names the store, links to the runbook, and fires on a symptom somebody chose rather than on whatever the vendor enables by default.

Most of all it produces the runbook, and the useful form of that document is a list of the states the system can be in, how each announces itself, and what to do. The store in these chapters has at least four: healthy, degraded by a rebalancing operation, read-only after crossing a memory limit, and unreachable. Each looks different in the metrics and each has a different response.

The value of that list can be priced, and the timeline at the top of this chapter prices it. That incident ran three hours and ten minutes; the same team's median incident on a database it knows runs twenty-five minutes. Subtracting the second from the first leaves two hours and forty-five minutes with no residual, and roughly all of it went on establishing things the state list would have stated outright: which state the cluster was in, that such a state exists at all, and who to ask. The arithmetic being exact is what licenses treating the difference as the price of unfamiliarity rather than as bad luck. It also gives the drill a budget to be measured against, in a currency the team already tracks.

The capacity drill produces the alarm thresholds, and it usually produces them by demonstrating that the existing ones are useless. The cluster in that incident entered its read-only state at 92% memory utilisation; the alarm the team had configured, copied from their Postgres alerting, fired at 95%. An alarm is only a warning while its threshold sits below the failure it warns about, and copying a threshold across systems keeps the number fixed while the thing it is supposed to discriminate — the utilisation at which this particular store stops accepting writes — moves. Here it moved past the alarm, so the ordering inverted. Under the growth rate the corpus was seeing, the window between the alarm and the failure it was meant to warn about was negative: the store had already stopped accepting writes by the time anyone was told. You cannot derive that threshold from documentation, because it depends on your vector dimensionality, your replication factor and your write pattern. You can find it in an afternoon by filling a staging node up and watching.

The restore drill produces the one number that most product SLAs quietly assume and almost nobody has measured: how long it takes to get retrieval back after total loss of the store. Write it in the decision record next to the SLOs. If it exceeds the recovery objective the rest of the stack meets, then either the objective changes or the architecture does, and both of those are conversations that go better before the outage than during it.

What the application does while the store is gone#

The fifth drill is the one that most often reveals a defect in your code rather than in the vendor's. Retrieval is usually called from a request path, through an HTTP client with whatever default timeout the library ships, which is frequently thirty seconds or none at all. Block the store's network path and the calls do not fail; they hang, and the thread or connection pool drains, and endpoints that never touch retrieval start returning 503 because there are no workers left. In the drill this took ninety seconds from the first blocked packet to unrelated pages failing.

The step in that chain worth slowing down on is the third one, because it is where a local fault becomes a global one. A worker occupied by a hanging call is not available to any other request, and the pool is shared across every endpoint the service exposes. Requests that never call the vector store are therefore competing for a resource the vector store is holding. Nothing about the blocked dependency has to be reachable from those endpoints for them to fail; they fail because they need a worker, and the workers are gone. A timeout is what bounds how long a single call can hold one, which is why the absence of a timeout is what converts an outage in one dependency into an outage in the service.

The correction is a bounded timeout, a circuit breaker — a wrapper that stops calling a dependency once it has failed a set number of times and retries it on a schedule instead of on every request — and a defined degraded mode that the product team has agreed is acceptable.

BREAKER = CircuitBreaker(failure_threshold=5, reset_timeout_s=30)

def retrieve(query: str, workspace_id: str, k: int = 10) -> Retrieval:
    if BREAKER.is_open():
        return lexical_only(query, workspace_id, k)
    try:
        with BREAKER:
            hits = vector_store.search(
                query_vector=embed(query),
                filters={"workspace_id": workspace_id},
                limit=k,
                timeout_s=0.25,          # retrieval SLO is 80 ms p95
            )
        return Retrieval(hits=hits, mode="semantic", degraded=False)
    except (TimeoutError, StoreUnavailable):
        return lexical_only(query, workspace_id, k)

def lexical_only(query: str, workspace_id: str, k: int) -> Retrieval:
    """Postgres full-text search over the same chunks. Worse ranking,
    same corpus, same filters, and it is up whenever the primary is."""
    hits = db.query(TSVECTOR_SEARCH_SQL, (query, workspace_id, k))
    return Retrieval(hits=hits, mode="lexical", degraded=True)

The timeout in that call is set to roughly three times the retrieval SLO rather than to the SLO itself, which is deliberate. A timeout equal to the target would abandon every request that was merely slower than usual, converting ordinary variance into fallbacks; one far above it would let a stalled store hold workers for long enough to reach the cascade in the paragraph above. A small multiple of the target is the band where a call that is going to succeed still has room to, and a call that has stopped is released before it costs anything shared.

The breaker's own two numbers do a similar job in the other direction. Requiring several failures before opening keeps a single timeout from demoting the whole service, and waiting a fixed interval before probing keeps the recovery attempt from becoming a second load source against a store that is already struggling. A breaker that probed on every request would hold the store down by testing it.

The degraded branch is only credible if it has been exercised with real traffic and somebody has looked at the answers it produces, because a fallback that returns nonsense is a worse outcome than an honest error. It also has to be visible: the degraded flag should reach the response, the logs and a metric, so that a week spent silently serving lexical results is not something you discover from a support ticket about answer quality. The reason a lexical fallback is available at all is that the chunks and their text stayed in Postgres, which is a choice worth making explicitly during a migration rather than discovering you did not make.

When the rotation cannot complete the drill#

The framing that makes this chapter useful in a planning meeting is that the drills are not a gate you pass before adoption; they are part of the evaluation. If the rotation cannot diagnose an injected fault, cannot restore a backup within the objective, and cannot describe the out-of-capacity failure mode, then the honest reading is not that the drills need to be rescheduled. It is that the organisation cannot currently operate this system, and the choice is to invest in that capability deliberately — training, runbooks, dashboards in your own stack, a second person who knows it — or to not adopt the store and revisit the mitigations from chapter 2 instead.

The belief that has to be given up to reach that reading is that a written procedure is evidence the procedure works. It is tempting because the artifact is real and inspectable, and because in a review a document and a demonstration look alike. It is wrong because a document is constant across every state of the world it describes: the same Notion page reads the same whether the restore takes twenty minutes or half a day, whether the credentials in it have rotated, and whether anyone on the rotation can find it at 03:14. A defence that does not change when the thing it must discriminate changes cannot discriminate, and "Vector store — TODO" is only the visible end of a spectrum whose other end is a beautifully written runbook nobody has executed.

Both of the outcomes above are legitimate. What is not legitimate is proceeding while treating operability as something that will develop naturally, because it develops through incidents, and incidents on a system nobody understands are the expensive kind.

Conditions a real incident does not let you choose#

A drill demonstrates a capability under conditions you chose, which is not the same as being ready. Real incidents arrive with two faults at once, during a deploy, while the person who ran the drill is unavailable, and with a symptom that does not match any of the five scenarios. What the drill actually buys is narrower and still worth the week: it converts unknowns into knowns, it puts the dashboards and runbooks in existence before they are needed rather than during, and it gives at least two people the muscle memory of having logged into the thing while nothing was on fire.

Putting the drills on a schedule#

A rehearsal also decays. A restore procedure verified in March is a claim about March; vendor versions change, credentials rotate, and the scratch environment gets deleted for cost reasons. Each of those changes something the drill measured without changing the record that it passed, which is the written procedure's defect returning in a slower form. Put the drills on a schedule — twice a year is a reasonable cadence for most teams — and treat a drill that has not been run in twelve months as equivalent to not having run it.

A store that has been rehearsed is genuinely adopted, and everything in this topic turns into ordinary operational work: the reconciliation job from chapter 5 running hourly and reporting zero, the recall monitor from chapter 3 comparing against its stored ground truth every week, and the revisit_when block in the decision record wired to a scheduled check so that the corpus crossing eight million chunks arrives as a ticket rather than as a surprise. Each of those is the same move as the drill, applied to a slower failure: put the system into the state you are worried about, on a schedule, while nothing is on fire.

What the drills buy, in the end, is that the surface a second datastore adds is held by the rotation rather than by one person, and that the numbers the product already promises have been checked against the dependency that now has to meet them. The next thing to build is the retrieval quality loop that sits on top of all of it, because a store that is fast, consistent and operable is still only as useful as the chunks and embeddings you put in it.