Planning around reviewer capacity

The scarce input to a human evaluation is the calendar time of the specific people qualified to make the judgement — a clinician for medical summaries, a paralegal for contract extraction, a compliance specialist for anything a regulator names — and that quantity is fixed on the timescale a release plan operates over. Everything in the preceding chapters draws on it: the pilot rounds, the second annotator who makes agreement measurable at all, the adjudication of contested items, the standing production sample that a judge cannot replace.

This chapter treats that capacity as the design constraint it is. What follows is how to measure realised capacity, how to cost a sampling plan against it, where to spend the minutes once there are fewer of them than the plan assumed, and how to put the ceiling somewhere the project can see it.

The plan is set by the ceiling whether or not it is written down#

A medical-summary product had a clean release plan: ship every two weeks, and before each ship, have a clinician review 150 sampled outputs for clinical accuracy. One clinician was assigned, with four hours a week allocated to the work. Plotting what she actually completed per release gives 96, 104, 88, 110, 71 and 99. The mean is 95 and the plan says 150.

The arithmetic was available on day one. Four hours a week over a two-week cycle is 480 minutes. The measured median time to review one summary, including reading the source record and writing a one-line note, is 3.4 minutes, so 150 outputs is 510 minutes. The plan exceeded the nominal allocation before anyone accounted for a clinic that runs late, a week of leave, or the fact that "four hours allocated" and "four hours of protected calendar" are different things. Her realised throughput of 95 outputs per release is 323 minutes, which is 67% of nominal — a ratio that shows up consistently wherever a specialist's time is shared with clinical duty.

Every step of that is a multiplication or a division and each one is worth watching, because the plan failed at the first of them and nobody performed it. Four hours is two hundred and forty minutes, and two of those weeks is four hundred and eighty. A hundred and fifty summaries at three and two-fifths minutes each is five hundred and ten. Five hundred and ten is larger than four hundred and eighty, so the plan was impossible against the nominal allocation before any real-world friction was allowed for — which is the part worth pausing on, because it means the failure did not require a clinic to overrun or a week of leave to fall in the wrong place. It only required somebody to multiply.

The realisation factor is the second division and it is the one that turns an impossible plan into a badly wrong one. Her six completed counts average out at a shade under ninety-five, and ninety-five summaries at three and two-fifths minutes is three hundred and twenty-three minutes of work actually done against four hundred and eighty allocated. Three hundred and twenty-three over four hundred and eighty is a little over two thirds. That ratio is not a measure of how hard she works. It is what a nominal allocation converts into when the person holding it also has clinical duty, and the reason to measure it rather than assume it is that every plan built on the nominal figure is overcommitted by half again before anything goes wrong.

Once realised capacity is 95 and the plan calls for 150, one of four things happens, and the project chooses which by default if it does not choose deliberately. The review lags and releases ship on partial evidence. The backlog grows, which on this product reached 330 unreviewed outputs across six releases. The clinician compresses her per-item time, which is the quiet failure — median review time on the last two releases had fallen to 2.6 minutes, and the note quality with it. Or the cadence stretches to fit.

Writing the ceiling down converts it from an emergent property into a parameter. The right unit is minutes rather than items, because review cost is not uniform across the sample. On this product the split was measurable:

A "150 random outputs" plan draws from that mixture and costs 150 × 3.4 = 510 minutes on average, with a standard deviation across releases of roughly 60 minutes purely from sample composition. That variability is the argument for the unit and is easy to miss, so state where it comes from: a routine discharge summary and a summary with a medication change differ by a factor of three in review time, and a uniform draw of 150 gets however many of each the month happened to contain. Two releases can therefore agree exactly on the item count and differ by an hour of the clinician's time, which means an item count does not constrain the thing the plan is actually spending. A plan expressed in minutes with an explicit stratum allocation costs what it says it costs.

One caution before the code, because the two figures in play are not the same kind of average and treating them as interchangeable will produce numbers that do not reconcile. The 3.4 is a median across all reviews, and the per-stratum column is a median within each stratum. A median of a mixture is not the mixture of the medians — half the reviews being faster than 3.4 minutes says nothing about the arithmetic mean of the four stratum medians weighted by their shares, and the two quantities can differ by a fair margin without either being wrong. Which one a plan should be built on depends on the question: a median is the right summary for "how long does a typical review take", and a total is what a budget needs, and a total is built from means. The habit worth keeping is to say which of the two any figure in a capacity plan is, since the arguments that go wrong here go wrong by silently swapping them.

STRATA = {                        # (share_of_traffic, minutes_per_item)
    "routine":       (0.62, 2.1),
    "comorbidity":   (0.24, 4.6),
    "oncology_paed": (0.09, 5.5),
    "med_change":    (0.05, 6.2),
}

def cost(plan: dict[str, int]) -> float:
    return sum(n * STRATA[s][1] for s, n in plan.items())

def fits(plan: dict[str, int], weekly_hours: float, weeks: float,
         realisation: float = 0.67) -> bool:
    budget = weekly_hours * 60 * weeks * realisation
    return cost(plan) <= budget

uniform_150 = {"routine": 93, "comorbidity": 36, "oncology_paed": 14, "med_change": 7}
targeted    = {"routine": 20, "comorbidity": 30, "oncology_paed": 25, "med_change": 20}

for name, p in [("uniform_150", uniform_150), ("targeted_95", targeted)]:
    print(f"{name:12s} items={sum(p.values()):3d} minutes={cost(p):6.1f} "
          f"fits={fits(p, weekly_hours=4, weeks=2)}")

uniform_150  items=150 minutes= 508.6 fits=False
targeted_95  items= 95 minutes= 428.5 fits=False

Both fail, which is the useful output. A checker that only ever said yes would not be worth writing; the value of fits is that it declines the plan somebody already agreed to, in a sentence nobody has to interpret. Under the 0.67 realisation factor the real budget is 322 minutes — four hours, times sixty, times two weeks, times a little over two thirds — and the second plan has to come down further, to roughly 15 routine, 25 comorbidity, 20 oncology and 15 medication-change items, which is 75 outputs and 311 minutes. That is the number the release plan should have said from the beginning, and it is half of what was promised.

Notice which figure the realisation factor is applied to, because applying it in the wrong place is the common error and it produces a plan that fails again. It multiplies the budget, not the cost: her hours are what shrink, and the minutes a review takes are unaffected by how much of her week she gets. A team that instead discounts the per-item time — reasoning that she will go faster under pressure — has written down the compression that this chapter's opening identifies as the quiet failure, and has written it down as a plan.

Fewer items, chosen for what they can tell you#

Shrinking a uniform sample from 150 to 75 halves the precision of an estimate nobody was using. What a release gate needs is coverage of the failures that would block a ship, and a uniform sample allocates attention in proportion to how common a case is rather than how much is at stake or how uncertain the system is on it.

Consider the regulated version of the same problem, which is where the trade-off is sharpest. Weekly releases, a compliance specialist who is the only person permitted to sign the safety label, six hours a week, and an eval sample of 400 items per release. Her budget is 360 nominal minutes, and at 3 minutes an item that is 120 items against 400 required. The obvious plan is to have her review a uniform random 130 and treat the pass rate as an estimate of the whole.

The uniform plan does not even fit: 130 items at 3 minutes is 390 against a 360-minute budget, before the realisation factor. Beyond the arithmetic, its output is a confidence interval on a base rate, and a base rate has never been what blocks a release. Releases are blocked by specific failures in specific categories, and the interval is wide enough that a pass rate moving from 94% to 91% is inside the noise.

That last clause is worth checking rather than accepting, since it is the whole case against the uniform plan. A pass rate measured on 130 items carries an interval of roughly seven points either side, which the table's second column states directly. Two releases whose true quality is identical will therefore routinely report figures several points apart, and a three-point move from 94% to 91% sits comfortably inside that. Widening the sample to close the gap runs into the square-root rule: cutting the interval to two points would need something like a tenfold increase in items, which is far more than the whole release contains and many times more than her hours. The uniform design is not underfunded. It is aimed at a quantity that cannot be resolved finely enough to gate on at any budget available here, which is a different problem and has a different answer.

The hybrid plan spends her hours in three deliberate places. Every item in the categories where regulation requires her signature is reviewed, with no sampling at all, because sampling a mandatory sign-off is not a smaller version of the control, it is the absence of the control. A validated judge supplies the next 45 items: everything it flagged, plus the items where it was least confident, plus any item where it disagreed with a deterministic structural check. The final 25 are a uniform random draw, which exists to detect the failure modes that neither the regulation nor the judge anticipated and which will usually return nothing.

Two operational details move the numbers more than they look like they should. Batching by case type dropped her median from 3.0 to 2.4 minutes on this workload, because reviewing twenty medication-change summaries in a row keeps one mental model loaded instead of reloading it every item. And routing her only the items that need her credential — pushing schema, formatting and citation checks into code, as chapter 5, on the limits of automated evaluation, established — removed about 18% of what had been in her queue.

Putting the ceiling on the schedule#

A capacity constraint that lives in someone's head collides with a release date during the release. The mechanisms that surface it earlier are unglamorous and cheap. Review capacity becomes an artifact in the repository alongside the eval config. A continuous-integration check, running on every pull request the way the test suite does, computes the plan's cost in reviewer-minutes and fails when it exceeds the declared budget, which turns "we added a new eval track" into a visible trade against an existing one rather than an invisible addition to a queue. Queue depth goes on the release readiness dashboard next to test pass rate, with the burn-down for the current cycle — the count of items still unreviewed plotted against the days left before the ship date — so a growing backlog is a number people see weekly rather than a discovery at sign-off. At sprint planning, the projected review minutes for the next release get read out the way projected engineering days are.

For the regulated product, the honest statement to product is that the real cadence is weekly for changes whose mandatory review fits inside 360 minutes and fortnightly for anything larger, and that the current 400-item plan is a fortnightly plan wearing a weekly label. That sentence is more useful early than any amount of effort spent making 400 items fit into 120.

The fixes that look like they solve it#

Hiring a second specialist is the first suggestion and it is a real option, with two costs that get left out. The new reviewer needs the guideline, the anchors and a pilot round before their labels can be pooled with the first reviewer's, and until agreement between them has been measured on a shared slice, two reviewers produce two datasets rather than twice the throughput. That is the first chapter of this topic applied to a hiring decision: two people scoring against a guideline whose gaps have not been closed between them are two instruments sharing a scale, and pooling their output produces a series in which a change between releases may be a change in which reviewer did more of the work. Nothing about the second reviewer being good at their job prevents this, which is why the shared slice is a cost of the hire rather than an optional quality step. Budget the calibration — 60 shared items, both reviewers, plus adjudication of the disagreements — as part of the hiring cost, and re-run a shared slice quarterly, because drift between two reviewers is the same failure as drift within one.

Paying more per hour does not help when the bottleneck is a credential rather than a wage, which is the case for clinical sign-off, legal review and anything a regulator names. Crowdsourced annotation is excellent for the tracks where the criterion can be taught in a page and terrible for the tracks that motivated the specialist in the first place; CQA-Eval is a good case study of budgeting expert time under exactly this constraint in a clinical QA setting, including how far non-expert labels can be pushed before they stop tracking expert judgement.

The one genuine multiplier is moving work off the specialist without moving the decision off her. She reads the judge's flagged items rather than the whole sample, she reviews structured extractions rather than raw records where the structure is already verified, and she signs a summary of twenty items in a category she has spot-checked rather than each of them. Each of those trades coverage for throughput, and each trade should be written down with the coverage it gave up, because that record is what an auditor asks for.

A review capacity is a budget in minutes, and an evaluation plan is a way of spending it. Stated that way the failure mode is obvious in advance rather than at the sixth release: 510 minutes of plan against 480 nominal and 322 realised, with the difference coming out of partial reviews and faster reading, which are the two ways a person absorbs an impossible plan without anybody filing a defect.

Everything in this topic reduces to a small number of things to do with the next evaluation you set up. Write the guideline with observable criteria and worked examples, pilot it on 50 items with two people, and read the disagreements before writing another rule. Report prevalence, a chance-corrected coefficient and positive-class agreement together, because any one of them alone will mislead. Adjudicate what the pair disagreed on and keep those items tagged. Hold out half the reference set before touching a judge prompt, and print the human ceiling beside every judge number you publish. Then count the reviewer hours you have, in minutes, and build the sampling plan that fits inside them before promising a cadence.