TF-IDF and logistic regression

TF-IDF stands for term frequency–inverse document frequency, and it is the durable answer to a problem a linear model cannot avoid: a logistic regression takes a vector of numbers and returns a score per class, and text is not a vector of numbers. Under TF-IDF each document becomes a sparse vector with one dimension per term in the vocabulary of the collection it was fitted on. The value in a dimension is how often that term occurs in the document, scaled down by how many documents contain it at all.

The scaling is the useful half. "The" and "please" occur in nearly every support ticket and carry almost no information about which queue it belongs in, so their inverse document frequency is near zero and they contribute almost nothing, while "chargeback" occurs in 0.4% of tickets and gets a large weight wherever it appears. The idea comes from Karen Spärck Jones's 1972 work on term specificity in retrieval, and it was the standard front end for text classification for roughly three decades before neural encoders arrived. It replaced hand-written keyword rules of the kind the previous chapter used as a floor, and it was in turn largely replaced for hard problems by fine-tuned transformers. It survives as a baseline because it is fast, interpretable and almost impossible to get catastrophically wrong. The Text Classification Pipeline: Starting Shallow going Deeper surveys the shallow methods worth exhausting before reaching for anything larger.

Underneath that description is one line of arithmetic, and it is worth seeing rather than trusting, because it explains why the method is hard to break. The inverse document frequency of a term is the logarithm of a ratio: the number of documents in the collection, divided by the number of them containing that term. Take a term present in every ticket. The ratio is one, the logarithm of one is zero, and the term's weight is not merely small but exactly nothing — it cannot influence any decision, whatever the rest of the pipeline does. Take a term in 0.4% of tickets: one document in every two hundred and fifty contains it, the ratio is two hundred and fifty, and its logarithm is substantial. The scaling is not a heuristic bolted on top of counting. It is the counting, arranged so that a term earns weight in proportion to how surprising its presence is.

The logarithm is doing a specific job there and is easy to read past. Remove it, and a term in one document out of two hundred and fifty is weighted two hundred and fifty times more heavily than a term appearing everywhere, so a single unique typo would dominate a ticket's vector on its own. Compressing the ratio through a logarithm keeps the ordering — rarer is still heavier — while flattening the extremes, so the gap between "rare" and "vanishingly rare" stops being the loudest signal in the representation. sublinear_tf=True in the code below applies the same compression along the other axis. A ticket saying "refund" ten times is more about refunds than one saying it once, but it is not ten times more about refunds, and taking a logarithm of the count is how the vectorizer says so.

Labels that already exist, and what they actually record#

The labels for a routing task are usually already in a database. Every ticket in a support system carries the queue it ended up in, and when a first-line agent moves a ticket from Billing to Refunds, that move is written to an audit table with a timestamp and a user ID. Filtering to tickets whose final queue has not changed for at least 72 hours is a reasonable proxy for "somebody who knew the policy decided this one", and it is a SQL query rather than a data collection project.

The reasoning behind that filter is worth making explicit, and so is the condition it carries. A ticket that has sat in one queue for three days without anyone moving it has been seen by the agents working that queue and not objected to, which is a weak but real endorsement of the label — quiet consent rather than an assertion. The condition is that the filter is not neutral about which tickets it keeps. It selects for tickets that stopped moving, so it under-samples anything still being argued about, and it under-samples classes whose handling is slow enough that a legitimate move often arrives on the fourth day. That skew runs in the direction of easy examples, which is exactly the direction that flatters a trained model. It does not invalidate the export. It does mean the number the export produces should be read against a set whose labels were not filtered this way, which is the evaluation set the previous chapters built.

Those labels come with a caveat that has to be stated before anything is trained on them. They record what agents did, including the routing mistakes they made and the taxonomy as it stood at the time, so a model fitted on them learns to reproduce agent behaviour rather than the written policy. Where the two differ, the model sides with the agents. The way to find out is to check the historical labels against the independently labelled evaluation set on the tickets that appear in both. Agreement at 94% means a training signal with 6% noise, which a linear model tolerates well. Agreement at 78% means the taxonomy changed underneath you, and the older half of the export needs relabelling or dropping.

The gap between those two prescriptions is a difference in kind rather than in degree, and reading it as a degree is the mistake to avoid. Six wrong labels in a hundred, scattered without pattern, pull the fitted boundary in no particular direction: the loss is summed over thousands of examples, and noise that is as likely to push one way as the other largely cancels, leaving a slightly less confident model in roughly the right place. Twenty-two wrong in a hundred, all arising because one class used to mean something else, do not cancel. Every one of them pushes the same way, so the model learns the old policy as a coherent rule and applies it confidently to new tickets. That is why the remedy differs: a few points of scattered noise is a tax you pay, and a block of systematically stale labels is a second, competing training signal. The check that tells them apart is not the agreement rate alone but whether the disagreements cluster — by date, by class, or by both. Clustered by date is a taxonomy that moved. Spread evenly is agents making ordinary mistakes.

What the build actually is#

The whole thing is short enough to read in one sitting, and the figures below come from fitting it on a 6,200-document export of support tickets.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import classification_report
import joblib

pipeline = make_pipeline(
    TfidfVectorizer(
        lowercase=True,
        ngram_range=(1, 2),      # unigrams and bigrams: "not arrived"
        min_df=3,                # drop terms in fewer than 3 documents
        max_df=0.6,              # drop terms in more than 60% of documents
        sublinear_tf=True,       # 1 + log(tf), damps repeated words
        strip_accents="unicode",
    ),
    LogisticRegression(
        max_iter=1000,
        class_weight="balanced", # the 4% class is not allowed to vanish
        C=1.0,
    ),
)

pipeline.fit(X_train, y_train)
print(classification_report(y_eval, pipeline.predict(X_eval),
                            labels=LABELS, zero_division=0, digits=3))
joblib.dump(pipeline, "router-2026-08-08.joblib")

Two arguments carry disproportionate weight. ngram_range=(1, 2) lets the model see "not arrived" as a feature distinct from "not" and "arrived", which recovers a chunk of the negation handling a unigram model throws away. class_weight="balanced" reweights the loss by inverse class frequency, so a refund class at 4% of traffic contributes as much to the objective as a billing class at 62%; without it, the optimiser discovers that predicting billing on everything is a decent local solution and refund recall collapses in exactly the way a keyword rule set's does.

The chain under class_weight="balanced" is short enough to walk and explains why the argument is not optional. The optimiser minimises the total loss summed over the training set. Every example contributes the same amount to that sum by default, so a class holding four tickets in a hundred can contribute at most four hundredths of the total loss no matter how badly it is handled — and abandoning it entirely is therefore a cheap move that buys whatever accuracy is available on the other ninety-six. balanced sets each example's weight inversely to how common its class is, so a refund example at four in a hundred is weighted roughly fifteen times a billing example at sixty-two in a hundred. Multiply the count by the weight and each class now contributes about the same to the sum. The optimiser has not been told to care about refunds; it has been shown a loss function in which ignoring them is expensive.

The two remaining vectorizer arguments prune the vocabulary, and they prune from opposite ends for different reasons. min_df=3 drops any term appearing in fewer than three documents. Such a term has almost no evidence behind it, so whatever coefficient the model fits to it is a memory of the one or two documents it appeared in rather than a pattern, and on a fresh ticket that coefficient fires as noise. max_df=0.6 drops terms appearing in more than sixty documents in a hundred. Those terms were already crushed to nearly nothing by the inverse document frequency, so the cut is not about their influence — it is about density. A term present in most documents puts a non-zero entry in most rows of a matrix whose speed comes entirely from being mostly zeros, and it does that for the life of the artifact in return for a weight near nothing.

The scoring line matters as much as the fit. It is the same classification_report call, with the same labels= list and the same evaluation set, that scores the model arm — not a similar one written alongside it. A trained classifier invites a bespoke evaluation path, because scikit-learn hands you cross-validation scores for free during development, and a cross-validation figure computed over the training export is not comparable to an LLM's score on a held-out set. Report the held-out number and keep the development one out of the comparison table.

The numbers around it are unglamorous and worth carrying as orders of magnitude. The vectorizer produces about 18,000 features after min_df=3 pruning. Fitting takes 2.1 seconds on a laptop CPU. The serialised pipeline is 1.4 MB, most of it the vocabulary dictionary rather than the coefficients. Inference is around 0.3 ms per document single-threaded and roughly 40,000 documents per second on a batch, because the whole thing is one sparse matrix multiply. There is no GPU anywhere in this story and no service to call.

Two of those figures look inconsistent until the word "batch" is taken seriously. Three tenths of a millisecond per document, done one document at a time, is a few thousand documents a second — and the batch figure is more than ten times that, on the same arithmetic and the same machine. The extra speed is not the classifier getting cleverer. A single-document call pays the cost of entering the vectorizer, allocating a one-row sparse matrix and returning through Python once per document, and that overhead is larger than the multiply it wraps. Batched, the same fixed costs are paid once for thousands of rows, and the multiply itself runs as one call into a compiled sparse routine. This is the shape of nearly every latency figure for this family of model: the per-item cost is dominated by overhead, so the throughput number and the latency number describe two different regimes and neither can be derived from the other by dividing.

The artifact size splits the same way. The coefficients are one number per feature per class, which on eighteen thousand features and a handful of classes is tens of thousands of numbers — real, but small. The vocabulary has to store the eighteen thousand n-gram strings themselves along with a hash table mapping each to its column index, and a short string in such a table costs several times what a number costs. That is why the file is measured in megabytes rather than kilobytes, and why adding classes barely moves it while adding documents to the training export does.

How many labels it actually needs#

Subsampling the same export and refitting gives a learning curve, and its shape is consistent across most three-to-five-class text tasks. Macro F1 is the summary these curves are read on: F1 is the harmonic mean of a class's precision and recall, and the macro average takes the plain mean of the per-class F1s, so a 4% class counts as much as a 62% one and a model that abandons it cannot hide behind the majority.

The harmonic mean is doing work in that definition that the ordinary mean would not do, and it is the reason F1 cannot be gamed by giving up on one half of the pair. Consider a classifier that sends exactly one ticket to the refund queue and gets it right. Its precision is perfect and its recall is close to nothing. An ordinary average of the two reports roughly a half, which describes that system as middling. The harmonic mean is dominated by the smaller of its two inputs — it goes to zero as either input goes to zero, whatever the other one is doing — so it reports something near nothing, which is the honest description. Every route to a high F1 therefore runs through being decent at both, and that is the whole reason to quote it rather than either number alone.

Read the gaps rather than the levels and the table stops being six data points and becomes a rule you can extrapolate from. A point here is a hundredth of macro F1, the unit the rest of this chapter counts in. Fifty labels per class to a hundred buys ten of them. A hundred to two hundred and fifty buys eight. Two hundred and fifty to five hundred buys five, five hundred to a thousand buys three, and a thousand to two thousand buys one. Each rough doubling of the labelling budget returns about half of what the doubling before it returned. That is what a plateau is — not a wall the curve hits at some particular row, but a constant proportional decay that makes the next step always cheaper to skip than the last one was. It also tells you what the row below the table would say without anyone fitting it: another doubling from two thousand buys about half a point, which is less than the noise on the measurement.

This family of model wants hundreds of examples per class to be worth discussing and low thousands to reach the point where additional labelling stops paying. That plateau is a feature when budgeting and a limitation when you need the last three points, because the curve does not bend upward again: getting past 0.88 on a task like this means changing the representation rather than adding data. Swapping TF-IDF for sentence embeddings — dense vectors produced by a small neural encoder, which place two sentences near each other when they mean the same thing regardless of which words they used — and keeping the same logistic regression on top typically buys two to four points on paraphrase-heavy text, at the cost of an embedding model in the inference path. SetFit, which fine-tunes a sentence encoder contrastively, is the current answer in the few-hundred-label regime and reaches TF-IDF's plateau with roughly an order of magnitude fewer labels.

What keeping it fitted involves#

Fitting the model once is an afternoon. Keeping it fitted is the part that shows up in the cost column, and it has a specific shape.

Shadowing, the final step, means running the new artifact against live traffic alongside the incumbent with its predictions recorded and thrown away rather than acted on, which catches the failures that only appear on inputs a curated set never contained. The frozen evaluation set is the load-bearing element: training data rotates as new tickets arrive and the evaluation set does not, because a metric measured against a moving target cannot detect a regression.

The gate before publication is stated per class rather than in aggregate for the reason the first chapter of this topic established: an aggregate can absorb the collapse of a small class without visibly moving. Requiring every class to stay within a point of the incumbent stops a refit that trades several points of refund recall for one point of billing recall, which is a refit whose headline figure improved. And the shadowing period exists because the frozen set is, by construction, a description of the past. It can catch a regression against inputs somebody has already seen, and it can say nothing at all about wording that first appeared after it was frozen — which is the first of the three failure modes set out below.

Refit the vectorizer and the classifier as a unit. The vocabulary changes when new terms appear, so a classifier's coefficients are meaningless against a different vectorizer's feature indices, and reusing a pickled vectorizer with a newly fitted model is a bug that produces plausible-looking garbage rather than an exception. The other reliable bug is fitting the vectorizer on the full corpus before splitting: inverse document frequencies then carry information from the evaluation documents, the score comes out one to three points high, and the discrepancy only surfaces in production.

Naming the route by which that inflation happens makes it much harder to reintroduce. The vectorizer decides two things when it is fitted: which terms get a column, and what each column's inverse document frequency is. Fit it on everything and both decisions are made with the evaluation documents in view. A term that appears three times only because two of those times were in the evaluation set survives min_df=3 and gets a column it would not otherwise have had — a column that happens to be informative about exactly the documents the score will be computed on. Every inverse document frequency is likewise computed against a collection that includes the test half, so the feature scaling is tuned to the distribution it will be tested against. Nothing here leaks a label, which is why the bug survives review by people who are watching for label leakage. What leaks is the shape of the test set, and the fitted model quietly gets a representation chosen with knowledge it will not have on live traffic. Using a Pipeline rather than calling the vectorizer by hand makes both hard to do by accident, which is most of why the code above is written that way.

Version pinning matters more here than for most dependencies. A scikit-learn minor upgrade can change default parameters or the pickle format, and a joblib artifact loaded under a different version may warn, may silently behave differently, or may refuse to load in the middle of a deploy. Pin the library version next to the artifact and retrain rather than migrate.

Where the linear model runs out#

Three failure modes account for nearly all of the gap between this arm and an LLM. The first is unseen wording. A term absent from the training vocabulary contributes exactly nothing, so when a company launches a product called Halo and customers start writing "my Halo won't pair", the token halo is not in the vocabulary and the document is classified on whatever else it contains. That is not a graceful degradation but a silent one, and it is why a model trained on eighteen months of tickets quietly loses a point a quarter.

The defence people reach for here is the confidence threshold described further down, and it is worth explaining why it does not catch this, because the reasoning is not obvious. A threshold abstains when the model is uncertain, and a document full of words the model has never seen sounds like the definition of uncertain. It is not what happens. A term outside the vocabulary does not enter the vector as a strange value; it does not enter at all, so the document arrives at the classifier as whatever is recognised — "won't", "pair", "my" — and the class scores are computed over that reduced evidence and then normalised against each other. If the surviving words lean at all, the model reports a confident probability for the class they lean toward. Missing evidence and conflicting evidence produce completely different confidence, and only the second one trips an abstain path. What does catch it is watching the share of each document's terms that fall outside the vocabulary — a quantity the vectorizer can report and the classifier cannot.

The second is a label set that moves. Splitting billing_dispute into duplicate_charge and pricing_complaint means the model has no coefficients for either new class, and no amount of clever inference fixes that — you need labelled examples of the new classes, which means another labelling pass over historical tickets, a project with a calendar date on it rather than a prompt edit.

The third is anything requiring knowledge outside the corpus. "The 9a stopped charging after the update" is a product defect if you know the 9a is a handset and an app problem if you do not, and the linear model knows only which tokens co-occurred with which labels in your own tickets. Sarcasm, conditionals and multi-hop reasoning fall in the same bucket.

Against those sits a compensating advantage that regex has no analogue for. A logistic regression emits calibrated-ish class probabilities — calibration meaning that predictions stamped with 0.8 turn out right about 80% of the time — and after a CalibratedClassifierCV pass those probabilities are trustworthy enough to threshold. Routing every prediction below 0.6 confidence to a human queue converts a chunk of the model's errors into reviewed decisions and lets you trade coverage against precision on a dial: a model that is 84% accurate on everything can be 96% accurate on the 78% of traffic it is confident about, and that reshaping is often worth more to the downstream process than the headline number.

The arithmetic hiding in that sentence is the whole argument for the abstain path, and it runs the other way from how most people read it. Overall accuracy is the confident slice's accuracy weighted by how much traffic it holds, plus the unconfident slice's accuracy weighted by the rest. The confident slice is 78% of traffic at 96% accurate, which contributes a bit under three-quarters of the total on its own. The overall figure is 84%. Subtract, then divide by the share of traffic left over, and the model turns out to be right on a little over four in every ten of the tickets it was unsure about — barely better than guessing among three classes. Thresholding made the model no more accurate anywhere. It sorted the same errors into one pile, and reported that the pile is about a fifth of the traffic and that the model's opinion inside it is close to worthless. That is the useful finding, because a fifth of traffic is a staffing question with an answer, whereas the same errors spread thinly across everything is a quality complaint with none.

There is a reason the CalibratedClassifierCV pass is not optional here, and it is sitting in the fitting code above. class_weight="balanced" deliberately trains the model on a reweighted world in which the rare classes are as common as the frequent ones. That is the right thing to do for the decision boundary and the wrong thing for the probabilities: the model's output reflects the base rates it was shown, not the base rates it will meet, so it overstates the probability of the rare classes on live traffic. The number is still a useful ranking — higher means more likely — but it is not yet a percentage anybody should threshold at 0.6. Calibration fits a small correction on held-out data that maps the model's scores back onto observed frequencies, which is what turns an ordering into a probability you can put a cut-off on.

Gradient boosting, which fits a sequence of small decision trees where each is trained to correct the errors the ones before it made, is the other rung on this ladder over the same features, usually worth a point or two on tasks with strong feature interactions, at the cost of a slower fit and a larger artifact.

The abstain path and the boosted model belong in the same mental category, and it is the category that makes this arm different from a prompt: a trained artifact, a retraining schedule, and an evaluation set that never moves. Which raises the question a table of accuracy numbers cannot answer. This arm has been described as cheap in every comparison so far, and the fitting run really does cost two seconds — but the export, the label audit, the relabelling pass when the taxonomy splits, and the engineer who owns the quarterly retrain are all part of what it costs to have the thing exist. The next chapter puts both sides of that ledger in the same column.