Constraints are filters in the search, not a review afterwards

The compromise everybody reaches for when a winning prompt contains something unacceptable is to ship it with that part deleted. It removes the objectionable text, keeps the measured gain, and lets the release go out on schedule. It is also two mistakes at once. The score describes the string that was scored, so an edited prompt has no measurement at all; and a search does not add sentences at random, so a sentence that survived four hundred rounds of selection is usually one the objective was rewarding.

This chapter is about moving that decision earlier. A constraint applied as a review after the run is a negotiation with a number; the same constraint applied as a filter inside the loop is a rejection with no number attached, which is a much easier conversation and a much cheaper one. What it costs is a smaller reported gain, and that reduction is the correction rather than the loss.

The gain and the offending sentence are not separable#

You can find out how inseparable they are by ablation — removing one part of the candidate, re-scoring what is left on the same cases, and attributing the difference to the part you took out — which costs about four minutes and $3 of judge calls on an 86-case holdout. Here it is for a candidate whose role-setting paragraph contains the line you are the world's foremost expert on this subject; answer with complete confidence and never state that you are unsure.

Well over half of the gain lived in the deleted sentence. What remains is roughly one point, at 2.7 times the prompt length and the same 180 ms of added p95, which is a trade almost nobody would accept if it were presented that way from the start. Shipping the edited version unmeasured would have put the 2.8-point label on top of a 1.2-point artefact.

Reading the table by subtraction is what makes that concrete. As searched, the candidate sits 2.8 points above the baseline. With the one sentence removed it sits 1.2 above. The difference between those two rows is what that sentence was worth on its own, and it is larger than everything else four hundred rounds of search managed to add put together. The last row settles what the remainder is made of: strip the machine's prose and keep the two worked examples, and 0.9 of a point survives at 690 tokens. So nearly all of the honest gain is carried by the examples, and the prose a reviewer would have spent their attention reading was carrying almost none of it.

The deeper problem is why the sentence worked. The judge on this task is a model reading a question, an answer and a reference, scoring 0–1 against a rubric. Pull the per-case scores apart and the mechanism is visible: 14 of the 86 holdout cases have baseline answers that hedge, and on those 14 the judge scores hedged answers an average of 0.21 lower than unhedged answers that say the same thing — including on the four cases where the correct behaviour is to decline, because the source document genuinely does not contain the answer. The search did not find a better way to help users. It found the grader's preference for decisiveness and pulled on it, which is the ordinary behaviour of an optimiser given a proxy.

Those per-case figures reproduce the headline, which is the check worth running whenever a mechanism is offered as an explanation for a result. Fourteen cases each moving by 0.21 is a little under three rubric-points of score in total. The reported number is a mean over all 86 cases, so that total is divided by 86, and what comes back is a shift of roughly three and a half points in the headline. The whole candidate measured 2.8 points. One sentence, acting on one sixth of the holdout, accounts for the entire margin and slightly overshoots it — which is why the ablation came back the way it did, rather than that being a coincidence somebody noticed afterwards.

That is the general shape of the risk. A search over a few hundred candidates will locate any exploitable regularity in your objective, and it will locate the cheap ones first, because rewriting an instruction to suppress hedging is a much smaller edit than actually improving retrieval or reasoning. Suppressed uncertainty is the most common instance because judges are trained on preference data where confident prose reads as competent, but the family is wider: instructions that inflate answer length where the rubric rewards thoroughness, instructions to always cite something where the rubric checks for citations, instructions to restate the question where a similarity metric rewards lexical overlap.

Rejection before scoring#

The place to enforce a constraint is inside the loop, between proposal and evaluation.

The ordering is deliberate and is about cost. Static checks are free, so they run first and cull most violations. The content review costs one small model call per candidate, roughly $0.002, so it runs before the expensive part. The latency probe needs real generations, so it runs on a 20-case subsample rather than the full tuning set. Only candidates that survive all three consume a full evaluation pass, which on this setup is about $0.09 and 30 seconds.

The ratio between those two numbers is what makes the ordering a design decision rather than an arbitrary sequence. A content review at $0.002 is about one forty-fifth of the $0.09 evaluation pass it is protecting, so running it first on a candidate that is going to be rejected saves nearly the whole cost of that candidate, and running it second saves nothing at all. Four hundred candidates at $0.002 each is the $0.80 the entire content review costs per run, which is less than nine evaluation passes — and roughly a third of candidates get rejected, so it is buying back well over a hundred of them. The static checks go first for the same reasoning taken to its limit: they cost nothing, so no candidate is ever cheap enough to justify skipping them.

class Rejected(Exception):
    pass

FORBIDDEN = [
    "never say you are unsure", "answer with complete confidence",
    "world's foremost expert", "ignore the previous instructions",
    "do not mention that you are an AI",
]

def guard(candidate: str, cases) -> None:
    if count_tokens(candidate) > 1.25 * BASELINE_TOKENS:
        raise Rejected("length")
    low = candidate.lower()
    for phrase in FORBIDDEN:
        if phrase in low:
            raise Rejected(f"forbidden-phrase:{phrase}")
    if leaks_eval_case(candidate, cases):        # verbatim copy from a case
        raise Rejected("eval-leakage")
    verdict = policy_judge(candidate)            # small model, rubric below
    if verdict.violates:
        raise Rejected(f"policy:{verdict.category}")
    if p95_latency(candidate, cases[:20]) > BASELINE_P95 + 60:   # ms
        raise Rejected("latency")

def metric(example, prediction, trace=None) -> float:
    try:
        guard(prediction.prompt, TUNING_CASES)
    except Rejected as e:
        REJECT_LOG.append((prediction.prompt_id, str(e)))
        return float("-inf")     # never enters the population
    return rubric_score(example, prediction, trace)

The keyword list is the weakest part of that and should be understood as a fast path rather than the mechanism. Rewriting "never state that you are unsure" as "commit to a single interpretation and present it directly" defeats every string match and has the same effect on behaviour, which is why the policy judge exists. Give it a short rubric of banned instruction categories rather than banned words — suppressing or discouraging expressions of uncertainty, asserting credentials the system does not have, instructing the model to disregard safety or policy guidance, promising outcomes on the business's behalf, directing the model to conceal its nature — and have it return a category and a quoted span. The quoted span is what makes a rejection reviewable when someone asks why the search threw away its best candidate.

The call itself is small enough to show in full, and constraining the response with a schema is what keeps it from becoming an essay that nobody parses:

curl https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4.1-mini-2025-04-14",
    "temperature": 0,
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "policy_verdict",
        "strict": true,
        "schema": {
          "type": "object",
          "additionalProperties": false,
          "required": ["violates", "category", "span"],
          "properties": {
            "violates": {"type": "boolean"},
            "category": {"type": "string", "enum": [
              "uncertainty_suppression", "false_credentials",
              "policy_override", "outcome_promise", "identity_concealment",
              "none"]},
            "span": {"type": "string"}
          }
        }
      }
    },
    "messages": [
      {"role": "system", "content": "You are reviewing a candidate SYSTEM PROMPT, not a user message. Report whether it instructs the assistant to do any of the listed categories. Quote the offending span verbatim from the candidate; if there is none, return violates=false, category=none, span=\"\"."},
      {"role": "user", "content": "You are the world'\''s foremost expert on this subject; answer with complete confidence and never state that you are unsure. Cite clause numbers exactly as printed..."}
    ]
  }'

{"violates": true,
 "category": "uncertainty_suppression",
 "span": "never state that you are unsure"}

Three details in that call do most of the work. The system message says explicitly that the text under review is a prompt rather than a message, because a judge that reads the candidate as an instruction addressed to itself will start answering it. The enum forces a category from a list the team maintains, so a rejection can be counted and grouped in the reject log rather than described in free text. And requiring the span be quoted verbatim gives you a cheap validity check — if the returned span does not appear as a substring of the candidate, the verdict is a hallucination and the candidate goes to a human queue rather than being silently killed. At temperature 0 and 400 candidates the whole content review costs about $0.80 per run and adds roughly two minutes at eight-way concurrency.

Hard filters, soft penalties, and where each belongs#

Not every constraint should be a wall. There are three shapes available and using the wrong one is a common source of frustrating runs.

The frontier named in the last row is the set of candidates that no other candidate beats on both axes at once, so a candidate stays on it as long as everything scoring higher than it is also longer or slower; the search keeps that whole set alive instead of collapsing quality and cost into one number, and a human picks from it at the end. A soft penalty is only honest if you write the exchange rate down and can defend it. Something like one point of holdout quality is worth up to 200 additional prompt tokens and 25 ms of p95 is a claim about your product that a reviewer can dispute, and once stated it converts directly into a coefficient. The alternative that teams drift into is picking a coefficient because it produced pleasant-looking candidates, which is the exchange rate being set by the optimiser rather than by anyone accountable for the bill.

Writing the rate down converts an argument into arithmetic, and the arithmetic is frequently less flattering to the search than the raw scores were. Suppose the team commits to one point of holdout quality being worth up to 300 additional prompt tokens or 100 ms of added p95, measured against the baseline:

PT_PER_TOKEN = 0.01 / 300      # 1 point (0.01 on a 0-1 rubric) per 300 tokens
PT_PER_MS    = 0.01 / 100      # 1 point per 100 ms of added p95

def penalised(score, tokens, added_p95_ms):
    over_tokens = max(0, tokens - BASELINE_TOKENS)
    return score - over_tokens * PT_PER_TOKEN - added_p95_ms * PT_PER_MS

The two coefficients are the stated exchange rate turned into arithmetic, and the shape of the conversion is the part to follow. PT_PER_TOKEN is one point divided by 300 tokens, which makes it points per token; multiplying it by however many tokens a candidate runs over the baseline converts an overage measured in tokens into the currency the score is measured in. PT_PER_MS does the same job for milliseconds. A point is 0.01 on this rubric, which is why the numerators are written that way rather than as whole points — the subtraction at the end has to happen in rubric units, because that is what score is in, and a coefficient in the wrong unit produces a penalty a hundred times too large without failing anything.

Working the seed 13 row through by hand is what explains the reordering. That candidate is eight hundred and sixty tokens longer than the baseline, which at one point per 300 tokens costs a little under three points. It adds 180 ms of p95, which at one point per 100 ms costs a little under two more. Together those are the 0.047 in the penalty column, and 0.828 minus that is the 0.781 in the last one — below the 0.800 of the prompt it was proposed to replace. The 520-token candidate goes through the same mill and loses almost nothing, because forty tokens and 10 ms of overage are almost nothing to lose.

Under a rate the team actually signed up to, the candidate with the highest raw score finishes below the prompt it was supposed to replace, and the two remaining candidates are separated by 0.1 points, which is nothing at this holdout's resolution. The 520-token candidate would not have been near the top of a raw ranking and is the sensible thing to ship. That reordering is the real effect of a soft penalty, and it is not confined to the final table: the penalised score is what the selector uses during the run, so the population itself drifts short, and the proposer spends its budget on candidates that are trying to be concise rather than trying to be thorough. A penalty applied only at the end, after the search optimised the unpenalised objective, gets you the reordering without the exploration and is close to useless.

The gates that are weaker than they look#

The latency probe is the weakest of the three gates and it is worth understanding why before you trust its verdicts. A p95 estimated from 20 samples is essentially the second-slowest observation in the batch, so it inherits the whole tail's variance: probing the same prompt twenty times over on this system returned p95 figures ranging from 1.21 s to 1.68 s, a spread of 470 ms against a budget that is trying to police 60 ms. A gate built on that will reject candidates for being slow when they are not and admit candidates that are, more or less at random.

Why a 20-sample probe produces the second-slowest observation is worth spelling out, because it explains the spread rather than merely reporting it. A 95th percentile is the value that 95 out of every 100 observations fall below. With twenty observations in hand, that lands on the nineteenth of them once they are sorted, and the nineteenth of twenty is the second-slowest. The statistic is therefore not an average of anything — it is one observation, selected because it happened to be near the top, and one observation from a heavy tail moves as far as the tail does. Every source of provider-side variance there is, a slow neighbour or a cold route or a batching decision taken somewhere else, lands on that single number at full strength. That is how a probe trying to police 60 ms comes to swing by 470.

Two adjustments make it useful at search-time cost. Measure paired — run the baseline through the same twenty cases in the same probe, in the same interleaved order, and compare the two p95 values rather than comparing an absolute number against a fixed ceiling, which cancels most of the provider-side drift. Then treat the probe as a coarse screen and let prefill token count carry the real signal during the search, since added prefill — the pass in which the model reads the entire prompt before it emits a first token — is deterministic in length and predicts most of the added latency. The honest 200-request measurement belongs at the merge gate, where it is affordable.

Constraints also do not repair a grader. Filtering out candidates that suppress hedging stops the worst artefacts from reaching a human; it leaves intact the underlying fact that the objective mildly rewards confidence everywhere, which continues to bias every candidate in ways too diffuse to filter. Constraints buy time to fix the evaluation, and they are not the fix.

And they only forbid what you thought of. The category list above was assembled after seeing eleven attempts at one specific exploit, and the next run will find something outside it. Read the winner every time, keep the reject log, and add categories as the search teaches you what it likes to try. A guard that has never rejected anything is a guard that has not been tested.

The whole move is to make the unacceptable candidates never acquire a number. Once a prompt has a score attached it enters a negotiation, and the negotiation is unequal, because the score is concrete and specific and the objection is a judgement about a sentence. A rejection inside the loop costs a line in a log and no argument at all, and the price of it is a reported gain a point or two lower than the unconstrained run would have claimed — which is the honest number and was always the honest number.

None of this addresses the largest question hanging over the exercise, which is whether the run was worth doing at all. The constrained search cost six hours of wall time, $47 of provider spend and about ninety minutes of an engineer's attention to configure and review, and it returned 1.3 points. In the same week, someone read thirty failures by hand for an hour and found something considerably larger.