Open coding and the codebook

Fifty traces read end to end produce fifty notes written in fifty different vocabularies. The next step is to turn those notes into categories that can be counted. Everything after this rests on them: they become the priority order, the roadmap argument, the before-and-after check on a fix, and eventually the criteria a judge model is given. What downstream inherits is their quality, and their quality is not a matter of how carefully the names were chosen.

What makes a category usable is that its definition names conditions present in the trace rather than impressions present in the reader. Underneath that sits the question the rest of the chapter has to answer: what property does a name need before two people reading the same fifty conversations write down the same number beside it? The answer is a property of how the definition is built.

Notes first, categories second#

The order matters, and it is the reverse of what most engineers do. The instinct is to write down five plausible categories before reading anything and then sort cases into them, which guarantees that you find the five things you already suspected and miss everything else. The method that works comes from qualitative social research, where it is called open coding, and it defers the categories entirely: while reading each trace, write one or two sentences in the vocabulary of that specific conversation, describing what went wrong and where. No shared vocabulary, no label list, no attempt to be consistent with the previous case.

The reason that order works is mechanical rather than psychological. A category list fixed before the reading turns the reading into a matching step, and such a step can only ever return a member of the set it was handed. Whatever a trace contains that matches none of the five has nowhere to be put, and since nothing is written down about it, it does not survive the reading at all. A note written in the conversation's own words survives the categories being wrong, so badly drawn categories can be redrawn against fifty notes rather than against fifty conversations that would all have to be opened again.

c_8842  asked about a march order; date filter built for march 2026, order was
        march 2025; empty result read as "no such order"; then get_order 404s on
        the hyphenated id and the model says the order doesn't exist
c_9103  user asks for return window on a lighting SKU; retrieval pulls 4 chunks
        all from shipping FAQ; model answers 60 days from general knowledge
c_9250  promo POOLSIDE20 expired 3 July, model quotes it on 19 July as active
c_9288  three-turn clarification about sizes then the model re-asks the size
        question it already got the answer to at turn 2
c_9301  refund request; model says "I've processed that for you" but no tool
        call was made in the whole trace
c_9344  customer writes in Portuguese, model answers in English throughout
c_9350  four sentences of preamble ("Great question! Let me look into that…")
        before a correct answer
c_9377  price quoted excludes VAT; storefront shows inclusive; 22% discrepancy
c_9402  model invents a "manager override" that lets the customer return after
        90 days; no such policy exists
c_9418  promo expired 30 June quoted as active on 12 July

Read fifty traces this way and the second pass — axial coding — is the one that groups. c_9250 and c_9418 are clearly the same thing. c_9103 and c_9402 both end in the model asserting a policy it did not retrieve, but the mechanisms differ: one had wrong chunks retrieved, the other had nothing to retrieve and confabulated a policy that does not exist anywhere. Whether those are one mode or two depends on whether they take the same fix, and here they do not, so they are two.

Fifty traces typically yield somewhere between eight and fifteen modes, and the two ends fail in opposite directions. Fewer than six usually means the definitions are too coarse to act on, because a boundary drawn that wide encloses several different fixes and the count inside it cannot say which one to buy. More than twenty means you have written down instances rather than modes, with boundaries drawn so tightly around individual conversations that most modes hold a single case.

A definition is a rule, not a name#

The name of a mode does almost no work. What makes it countable is the condition attached to it, written so that a person reading a trace can answer yes or no without knowing what you were thinking.

That requirement follows from where a count comes from. No trace emits one. A count is produced by a person opening a record, deciding whether a label applies and moving to the next; the count is the number of times that decision came back yes. Two people arriving at the same count therefore needs more than care. It needs the procedure each runs to return the same verdict on every case.

Only two things are available to both procedures: the trace, and the words of the definition. Everything else one reader has is unavailable to the other — a theory about what the model was doing, a sense of which failure is the interesting one. Anything the definition leaves to those private inputs gets completed differently by each reader, and neither completion is visible to the other. Agreement is possible only where the yes-or-no boundary can be settled from the trace and the definition alone.

That is why a mode has to be defined as a predicate over observable features of the record — a predicate being a question with a yes-or-no answer, asked of one specific trace — rather than over an intention inferred from it. A name that quantifies the model's state cannot be counted the same way twice, because the model's state is not in the record. There is no field in a log for confusion. Each reader has to infer it, and two inferences drawn from the same evidence by two people are two different quantities that happen to share a word.

Take c_8842 from the notes above and try both names on it. Called "the model got confused about the date", the label is applied by one reader who sees a filter built for march 2026 against an order from march 2025 and answers yes. A second reader goes on to the get_order call that 404s on the hyphenated id, concludes the model was never confused about dates at all but about identifiers, and files the trace elsewhere. Neither misread anything, and neither can show the other a step they got wrong, because the label named nothing either of them could point at.

Now state the same failure as a condition: the tool call carries a date filter whose year appears in no user turn of the conversation. Applying that is three lookups — open the tool call, read the filter argument, scan the user turns for that year — and both readers reach the same answer. Which traces are bad has not changed. What moved is where the decision is taken: out of the reader and into the record.

The predicate also has to be over fields that are actually there, which is an easy second condition to miss. A definition turning on whether a retrieved chunk contained a figure is decidable only if the retrieved chunks were logged (chapter 1, where reading a trace end to end depends on the four things logging has to capture). Where the field is missing, the reader reconstructs it from what is present, reconstructions differ between readers, and the definition has quietly become as private as "confused" while still reading like a rule.

The plain version is that a codebook is a program and a trace is its input. Two coders agree because they run the same program rather than because they share a sensibility, and a good sentence that leaves a branch undecided is still a program with a hole.

Every accepted row above compares two things with places in the record; every rejected row asks for something the trace cannot supply.

Writing these out is where most of the value of the whole exercise gets created, and it is slow — an afternoon for eleven modes is normal. A workable format keeps the modes as rules with explicit exclusions — the definition, the boundary cases and the trace ids that anchor each — in one file. That file is the codebook, the durable artefact this exercise produces and the thing the remaining chapters keep referring back to: the counts that become rates, the modes that get ranked, the definitions a re-measurement has to reuse. Holding it in one place is what makes the argument, when a disagreement arises, an argument about the definition rather than about memory.

- id: expired_promo_quoted
  definition: >
    The assistant states a promotional price or discount code as currently
    valid, and that promotion's valid_to is earlier than the conversation
    timestamp.
  includes:
    - quoting the promotional price without naming the code
    - saying "that discount still applies" about an expired code
  excludes:
    - promotion is live but the assistant computes the discount wrongly
      -> discount_arithmetic
    - assistant correctly says the promotion has ended but gives the wrong
      end date -> stale_policy_detail
  anchors: [c_9250, c_9418, c_9522]
  severity_unit: usd_per_occurrence

- id: invented_policy
  definition: >
    The assistant asserts the existence of a policy, exception or entitlement
    that does not appear in any retrieved chunk and does not exist in the
    policy source of truth.
  includes:
    - inventing an approval or override path ("a manager can waive this")
    - inventing an eligibility carve-out
  excludes:
    - stating a real policy with the wrong number -> stale_policy_detail
  anchors: [c_9402]
  severity_unit: compliance_flag

The excludes block is what stops two modes from bleeding into each other, and it is nearly always written after somebody disagrees with you. Each line names the other mode by id rather than merely fencing this one off. A one-sided fence leaves the excluded case homeless and the next reader puts it wherever it looks least wrong, which is a coin toss. A line saying where the case goes instead makes the boundary a partition, and a partition returns the same answer whichever side of it a reader approaches from.

Measuring whether the labels travel#

The sentence a reader would confidently write after the section above is that once the definitions are on paper, two people applying them produce the same counts. That does not follow. Writing a definition down fixes the words; it does not fix the set of traces the words select, and the count is a property of the set.

A definition is written from the cases that prompted it, and its author tests it silently against those cases as the sentence is composed. Every one sits comfortably inside the rule or outside it, which is what makes the sentence feel finished. Those are not the cases that decide a count. A trace both readers would code identically anyway contributes nothing to the difference between two counts, so every unit of difference comes from a case near a boundary — and the boundary is precisely the region the definition was not written from. The disagreement does not disappear when the definition is written down. It moves into the cases the definition never addressed, where it is harder to see.

More labels and better wording in the abstract do not close that gap, because they still describe the centre. What closes it is adjudicated boundary cases written back into the file: an excludes line naming the neighbouring mode, and anchors pinning a definition to trace ids a reader can open. Neither can be written from an armchair, since a boundary case is only identifiable once two people land on opposite sides of it. The measurement below is an index into those disagreements, and the disagreements are the product.

A taxonomy only one person can apply is a diary. The check is cheap: take 20 of the 50 traces, have a colleague code them using the written definitions and without seeing your codes, and compare. The blindness is load-bearing rather than ceremonial: a colleague who can see your label on a borderline trace resolves the boundary by deference, agreement rises, and the defect survives into every count the codebook produces. Raw agreement — how many of the 20 you both gave the same primary mode — is the first number, and it is optimistic when a couple of modes dominate, since two people guessing the most common label agree by accident.

The size of that accident is worth working out on a small case. Suppose a codebook had one dominant mode that each coder reached for on nine cases in ten, plus one rare mode taking the rest. Two coders who ignored the traces entirely and merely kept those habits would both reach for the dominant mode on nine-tenths times nine-tenths of the cases, which is eighty-one hundredths, and both reach for the rare one on a tenth times a tenth, one hundredth more. Their agreement by habit alone comes to a little over four-fifths. Coders genuinely agreeing on nineteen cases out of twenty have therefore bought about seven-tenths of the room that was available, and quoting their raw agreement instead credits the definitions with almost everything the label distribution was doing on its own.

Cohen's kappa corrects for that agreement by chance, and of its two ingredients the second is the one doing the work. Observed agreement is the share of cases where both coders wrote the same label. Expected agreement is the share they would have reached under a model in which each coder keeps their own frequency for each label and ignores the trace entirely. Under that model the two choices are independent, so the chance both pick a given label is the product of the two rates; and agreeing on one label excludes agreeing on another, so the per-label chances add. That is the whole of the expected term: multiply per label, sum across labels.

def cohens_kappa(a, b):
    """a, b: lists of mode ids assigned by two coders to the same cases."""
    n = len(a)
    observed = sum(x == y for x, y in zip(a, b)) / n
    labels = set(a) | set(b)
    expected = sum((a.count(l) / n) * (b.count(l) / n) for l in labels)
    return (observed - expected) / (1 - expected), observed

# 20 double-coded traces, 11 modes in the codebook
mine   = ["expired_promo_quoted", "invented_policy", "preamble", ...]
theirs = ["expired_promo_quoted", "stale_policy_detail", "preamble", ...]

kappa, observed = cohens_kappa(mine, theirs)
print(f"raw agreement {observed:.2f}, kappa {kappa:.2f}")
# raw agreement 0.70, kappa 0.66

The two printed numbers can be reconstructed from the codebook's own shape. Observed agreement of 0.70 across 20 cases is agreement on fourteen of them. With eleven modes and no single mode above about a fifth of the sample, the largest label contributes a fifth times a fifth to the expected term, which is one twenty-fifth; the remaining modes divide the rest between them and each contributes far less, so the whole sum lands near a tenth. Chance agreement that low is why kappa tends to land close to raw agreement here. Subtracting it from seven-tenths leaves about six-tenths of genuine agreement over a denominator of about nine-tenths, which is roughly two-thirds — a shade above the printed 0.66, because the real distribution is slightly more concentrated than the flat approximation and buys slightly more chance agreement.

The shape of the ratio says what those two-thirds mean. Perfect agreement is one, and the chance model already delivers the expected term for nothing, so the room a codebook can win is one minus the expected term and the room it did win is the observed term minus the expected term. Kappa is the fraction of the available room that the written definitions bought. Zero means they added nothing to what the label frequencies produced anyway, and a negative value means two coders agreed less often than strangers guessing. Dividing rather than subtracting is what makes the number comparable across codebooks, since the same gap is a much larger achievement where little room existed than where plenty did.

A kappa of 0.66 on 20 cases means you and your colleague disagreed on six of them, which is too many to publish counts from and is also the most useful diagnostic you will get all week. Look at which pairs the disagreements fell between: if four of the six are invented_policy against stale_policy_detail, the boundary between those two definitions is the defect, not the coder. Sharpen the excludes block, recode the twenty, and remeasure. Two rounds usually take kappa above 0.8, which on 20 cases means agreeing on 17 or 18 — enough that the counts are a property of the traces rather than of who read them.

The sample size here is small enough that the kappa itself is imprecise; on 20 cases the confidence interval — the range of true values a sample this small is consistent with — spans roughly ±0.2. The arithmetic behind that width is why a small improvement in the number means nothing. One case switching from agreement to disagreement moves observed agreement by a twentieth, and moves kappa by more than that once the smaller denominator is applied, so three or four cases' worth of ordinary sampling movement covers the whole quoted interval. It is a tool for finding colliding definitions rather than a certificate, and treating it as a certificate is a way of spending a day producing a number nobody acts on.

Two conditions sit on the statistic as written. It compares one label per case, so a trace exhibiting two modes has to be reduced to a primary before it enters the calculation, and modes that habitually co-occur are under-represented by the measurement even where the codebook handles them correctly. And the correction turns against you when one mode dominates: as a single label's share rises, the expected term climbs towards the observed term, and coders agreeing on nearly every case can still print a low kappa. What is worth arguing about there is the raw agreement together with the label distribution.

Which merges keep information#

Eleven modes with the largest at 9 cases and four at a single case each is an awkward thing to put in front of an exec update, and someone will ask for four buckets so the chart is readable. Some of those merges are free and some destroy the analysis.

Those verdicts look like judgements about how similar the traces are, and they are not. A count exists in order to be turned into a decision — what somebody builds next, and who — so the test of a merge is whether it can preserve an answer to that question. The three retrieval modes pass because one workstream closes all three: the merged bar names exactly one piece of work, and its height is the prize for doing it. The pricing merge fails on the same grounds. An expiry check, an arithmetic bug and a display convention are three separate purchases at three separate prices, and the merged bar's height is a sum of prizes attached to none of them. Anyone deciding what to build has to take that bar apart again, so the merge did no work and cost the reader the distinctions.

A merge is therefore a decision about the fix rather than about the traces: two modes belong in one bar when the count is downstream of an action a single person takes. That is what the rule underneath the table encodes: a merge is legitimate when the merged members share a cause, an owner or a severity, and destructive when they share only a surface or a count. Cause decides which change closes the mode; owner decides who makes it; severity decides what one occurrence is worth, which is the multiplier every count is about to be run through (chapter 5, where a count out of fifty failures is projected into a rate over conversations and multiplied by a cost per occurrence). Frequency is not among them, which is why merging by count is the specific failure worth guarding against: an "other" bucket sorted by frequency reliably swallows the rare severe modes. The single case where the assistant invented a policy exception goes into the same bin as the markdown inconsistency, and the chart then shows precisely the modes that are cheapest to care about.

Keep the eleven codes in the file regardless of what the chart shows. A merge encodes a hypothesis about which fix closes which mode, and that hypothesis gets revised — sometimes by the fix itself, when a change aimed at one mode closes two, or none. A grouping should therefore be a presentation choice made on top of the raw codes and reversible with a group by, so that a revised hypothesis costs a query rather than another afternoon of reading. The moment you merge in the source data you have thrown away the distinctions you spent an afternoon defining, and a distinction destroyed there is recoverable by no query.

A mode that appeared once#

Four modes at n = 1 are not noise to be discarded and they are also not measurements. At a sample of fifty, one occurrence is consistent with a true rate anywhere between about 0.05% and 10% of failures, which is wide enough to be useless on its own. That width is not a defect of the arithmetic. A mode running at a twentieth of a per cent and a mode running at one in ten both throw up exactly one case in a draw of fifty often enough that seeing one case cannot separate them. Three moves are reasonable, and which applies depends on the mode.

If the mode is cheap and cosmetic, leave it named in the codebook, do nothing, and let the next round of reading tell you whether it recurs. If it is severe — an invented policy exception, a data leak across customers, anything legal would want to hear about — do not wait. Turn the definition into a search over the whole corpus of stored conversations: a regular expression, or regex, matched against the text, or a similarity search over embeddings — numeric representations of text that place passages with similar meanings near each other — for the signature (manager override, we can make an exception, as a one-time courtesy). Run across a month of traffic — around 61,000 conversations for a support assistant of this size — that turns n = 1 into a real count within an hour. If that search finds 340 matches and manual review of 30 of them confirms 11 genuine instances, the mode is no longer a single case and the whole conversation about it changes. The third move applies when a mode is severe and unsearchable, where the only honest report is that you saw it once, cannot bound its rate, and the decision must be made under that uncertainty.

The second move is available only because of how the definitions were written. A rule stated over observable features of the record can be approximated by a query over the same records; a rule stated over the model's state cannot, since there is no field to query. The search matches a signature rather than the mode, so its precision against the definition is unknown until somebody checks, which is what reviewing thirty matches by hand establishes. Its recall is not established by that check at all, because a phrasing nobody thought to search for produces no matches to review. The resulting count is a lower bound rather than a rate, and a reviewer should ask to see the phrasings that were tried.

A codebook is also not a bug tracker, and failing to keep them separate is common. A mode is a category of observed behaviour with a count attached; a bug is a specific defect with an owner and a fix. One mode can spawn three tickets and three modes can collapse into one ticket. Nor is it yet a judge rubric, though the definitions written here are exactly what a judge prompt eventually needs, which is why effort spent on excludes blocks pays twice.

What the codebook buys is that counts stop being a property of who did the reading. That is a low bar and almost nothing in this practice clears it by default, because the natural output of reading fifty conversations is a set of impressions in the reader's own words, and impressions do not survive being handed to somebody else. A definition written from observable conditions survives, which is what makes a mode something a second round can re-measure and a fix can be checked against (chapter 7, where the same cases are re-coded with the same codebook to see whether a mode actually moved).

The file now holds eleven modes with definitions, anchors and counts out of fifty, and it is tempting to read those counts straight off as priorities. They cannot be read that way, because a count out of fifty failures is not a rate over anything, and comparing modes requires knowing how often each one happens to real users and what it costs when it does.