Chain-of-thought prompting

The technique arrived in 2022 as an observation about models that answered immediately. Wei and colleagues showed that including worked examples with intermediate steps lifted arithmetic and commonsense reasoning benchmarks substantially, and Kojima and colleagues showed that the bare instruction "Let's think step by step" recovered much of that gain with no examples at all. The mechanism was straightforward: a model that emits an answer token immediately after the question has spent a fixed, small amount of computation on it, and text emitted before the answer is additional computation whose results are visible to the later tokens. Making the model write its working gave it somewhere to work.

That was a workaround for a deficiency in how the models were trained, and the labs treated it as one. Reasoning models are the result of training the behaviour in directly, with reinforcement learning — rewarding the model for traces that reach a checkable correct outcome, rather than for resembling human-written steps — so the model generates a long internal trace and then an answer. The trace is usually hidden or summarised, and it is billed as output tokens whether or not you see it. A port that moves a carefully tuned step-by-step prompt onto one of those models is the case this chapter is about, and it is a case where the prompt work is subtraction rather than addition.

Three ways an explicit procedure competes with an internal one#

The instruction is no longer supplying a missing behaviour. It is arriving at a model that already has one, and it competes with it in three distinct ways.

The decomposition is imposed rather than chosen. "First list the key claims, then weigh them" is one strategy for summarisation, and it is a good one for the average document set. On sources that contradict each other, the strategy a reasoning model would have selected on its own — identify the disagreement first, then trace what each source's claim rests on — is better, and the instruction prevents it from being selected. That is where a quality loss concentrated in one subset comes from.

The steps get done twice. The model reasons internally about how to satisfy the request, then also produces the visible enumeration the prompt asked for, because the prompt asked for it. In one set of traces, a typical response contained a numbered list of eleven claims averaging 340 tokens before the summary began, none of which any downstream consumer read.

And the reasoning budget gets spent on compliance. Some of the internal trace is taken up working out how to map the requested procedure onto the task, which is overhead that the same model without the instruction does not pay. That overhead is invisible in the ordinary sense — the trace is hidden or summarised, so you cannot read what fraction of it went on the mapping — and it is visible in the bill, because reasoning tokens are metered like any other output tokens.

The three have one root, and making it explicit also explains why the technique ever worked. A transformer produces each token with one pass through a fixed stack of layers, so the computation available for any single token is fixed and cannot be raised by asking more insistently. Separately, every token that is emitted is appended to the context that later tokens attend over. Put those two together and there is exactly one lever available from outside the model: more tokens is more computation, and the tokens already written are a scratchpad the rest of the response can read. Chain-of-thought prompting pulls that lever from the prompt. A reasoning model pulls it from the inside, on a trace selected for the task in front of it rather than for the average of all tasks, and the three problems above are what happens when both pull it at once.

Worked examples that include a reasoning trace are the sharpest version of the same problem, because they are demonstrations rather than instructions and therefore carry more authority. A three-example few-shot block showing the claim list before each summary is telling the model, in the most persuasive form available, that the visible enumeration is part of the expected output. Removing those examples entirely is usually too blunt, since they are also carrying the output shape and the register; converting them so the assistant side shows only the final summary keeps what they were teaching about the product and drops what they were teaching about the procedure.

The token ceiling is what catches most teams first. On several providers the internal trace is charged against the same max_tokens limit as the visible answer, so a cap of 1,200 that was generous for a 700-token summary now truncates: the model spends 1,100 tokens thinking, emits a hundred words, and the response comes back with a stop reason of length that the calling code has probably never handled and scores as a bad summary rather than as a configuration error. Raising the ceiling to cover both, and reading the stop reason instead of assuming completion, is the first change to make when a ported prompt starts producing short answers on a reasoning model.

The subtraction is the whole of that failure: a cap of 1,200 with 1,100 spent before the answer begins leaves a hundred tokens, and a hundred tokens is a small fraction of the 700 the summary needed. What makes it expensive to diagnose is not the arithmetic but the disguise. The response is well formed. It is a summary. It is a summary that stops, and a rubric grader marking it down for incompleteness returns a quality number that supports exactly the wrong conclusion — that the model is worse at this task. The stop reason is the one field that tells the two apart, and code written against a model that never truncated has usually never read it.

Strip the procedure, keep the specification#

The useful cut is between text that tells the model how to think and text that tells it what counts as a correct answer. The first travels badly onto reasoning models. The second is information the model cannot derive and should be kept, or strengthened, since a model with more room to plan can use more constraints.

The rows that get kept are worth more attention than the rows that get removed, because stripping a procedure often exposes information that was buried inside it. A definition of what counts as a key claim written as a parenthetical inside step one is a criterion wearing a procedure's clothes, and pulling it out and giving it three sentences of its own does more work than the enumeration ever did.

Keeping the two variants separate in code is the mechanical half of the same idea:

# prompts/summarise/core.md is shared. The procedure block is not.
REASONING_VARIANT = CORE                      # task, criteria, constraints, contract
GENERAL_VARIANT   = CORE + PROCEDURE_BLOCK    # the tuned "first list, then weigh" steps

params = {
    "reasoning": {"model": "reasoner-v1", "reasoning_effort": "medium", "max_tokens": 4000},
    "general":   {"model": "general-v4",  "temperature": 0.3,           "max_tokens": 1200},
}

Telling the two cases apart#

The provider's marketing language is unreliable here, and there are three signals that are not.

The billing breakdown is the most direct: if the usage object reports reasoning or thinking tokens separately from completion tokens, the model is doing its own trace and you are paying for it.

{"usage": {"prompt_tokens": 3120, "completion_tokens": 700,
           "completion_tokens_details": {"reasoning_tokens": 1780}}}

The presence of a reasoning-effort or thinking-budget parameter is the second, since a provider only exposes a knob for a thing the model does. Those knobs also give you a cheaper experiment than a full prompt rewrite: running the ported prompt at minimal effort and at high effort on the same cases tells you how much of the latency is the model's own trace before you change any text.

The third is a two-arm run on your own set — the prompt as ported against the prompt with the procedure block removed — which is the only signal that answers the question for your task rather than in general. On a 180-case set that is 360 calls and about eleven minutes.

The 360 is the set run once per arm — 180 cases, two arms, and no repeats — and it is worth being precise about what one pass per arm can support. It sizes an effect. It gives neither arm a spread, so a gap of a few hundredths of a rubric point coming out of it means nothing, in the same way that a single run of two models with no error bars around either is not a comparison. What it is good for is deciding whether the question deserves the fuller version: two arms that come back several tenths apart on some subset are worth repeating properly, and two that come back level on every cut are worth dropping.

When the procedure stays#

Stripping is the default, not a rule, and three cases justify keeping an explicit decomposition on a reasoning model.

The decomposition encodes domain knowledge rather than generic problem-solving. A prompt that says "apply the four-part test from the 2019 guidance: first jurisdiction, then materiality, then the safe-harbour exceptions, then the disclosure obligation" is not telling the model how to think; it is telling it which specific procedure the regulator requires, and no amount of internal reasoning will invent it. Keep that, and consider making it more explicit rather than less.

The intermediate artifacts are outputs. If the claim list is displayed to the user, written to a database, or consumed by the next stage of a pipeline, then it is part of the output contract and belongs in the prompt as such — expressed as a required field in a structured response rather than as a thinking instruction, which also brings it under the guarantees from the previous chapter.

The target is not actually a reasoning model. The whole argument collapses if the model you are porting to answers immediately, and plenty of the cheap, fast models a cost-driven migration lands on do exactly that. On those, the procedure block is still worth what it was worth, and stripping it because "modern models don't need step-by-step" is the same portability assumption in the opposite direction.

None of this says explicit reasoning instructions are obsolete. It says they are a per-model patch like any other, and the direction of the patch depends on the target. It also does not say reasoning models are strictly better: at 2.3 times the cost per summary and 65% higher p50 latency even after stripping, the reasoning model wins on a workload whose value lives in the hard subset and loses outright on a workload of clean single-source documents, where the migration would be a mistake made with good numbers. Both multiples come off the same table: $21.20 per thousand against the original model's $9.30 is a little over two and a quarter times, and 6.9 seconds against 4.2 is a bit under two-thirds again as long. Neither is small, and neither is settled by anything in this chapter. What settles it is the mix, because the advantage is concentrated in one subset, so the trade is worth making in proportion to how much of the traffic that subset is.

And the stripped prompt is a patch, written for one target, tested on 180 cases. The next question is what a patch does to the cases it was not written for — because the process that produced it, running the port, finding the failures, rewriting until they pass, has a well-known way of moving quality backwards while every number you looked at goes forwards.