Describing an agent without implying intent or autonomy

Describing an agent to a colleague in finance or operations is a separate piece of work from building one, and it fails in a way that is easy to mistake for confusion on their part. A stakeholder outside engineering reads a one-page summary, forms a picture of the system from it, and then plans against that picture: what oversight it needs, whether it can be given a budget, whether headcount can come down. Those plans are made from the summary rather than from the code, so one wrong noun in the summary is enough to send weeks of somebody else's work in a direction the software does not support.

The thing being described here is the four-part machine this topic has been assembling: a model that can be asked for an action rather than only for prose, a fixed set of tools it is allowed to request, a loop that runs what it asked for and feeds the result back, and a condition that ends the loop. Nothing in that list is a claim about reasoning, judgement or independence. The word agent is such a claim in the ears of most of the people who will read it, and every failure described below comes out of the gap between those two facts.

The word arrives with a meaning already attached#

In ordinary English, and in law, an agent is someone who acts on another's behalf with discretion. A travel agent books what they judge best within your budget; an estate agent negotiates on your instructions; the legal doctrine of agency is built on a principal, an agent, a scope of authority, and liability that flows back to the principal when the agent acts within it. That is the meaning your colleagues already hold, and it is coherent, useful and load-bearing in their working lives. It also contains discretion, delegated authority and accountability, none of which describes a for loop that appends dictionaries to a list.

What that meaning then does runs in five links, and each one can be checked on its own. A stakeholder meets the word agent in your summary. They resolve it against the sense they already own, because nothing in the document tells them that a different sense is in force. That sense carries three properties — discretion over means, authority delegated by a principal, and liability flowing back to that principal — so all three are now attached to your system in their head. From those properties, particular questions follow as valid inferences: something with discretion can choose badly, so what stops it; something with delegated authority spends against a budget, so what is its budget; something whose liability flows back has a principal, so who is accountable for a given run. Because each inference is sound given the definition, the person asking is not being careless.

The legal picture is being used here as an analogy — it is offered as the meaning your colleagues bring with them, not as a description of what runs — and it breaks against the software in three specific places. There is no discretion: an agent at law chooses the means to an end, while this program can only name one of the tools written into its tool list before it was deployed, and it cannot name a tool that is not there. There is no delegated authority: the model holds no permissions at all, because every credential, every network route and every write belongs to the dispatcher, which is code you wrote and which decides whether to carry out what the model named. And there is no liability flowing back, because agency law routes liability between two legal persons and there is only one of those here; the team that deployed the loop is responsible for every run of it.

Software borrowed the word for something narrower, and inside engineering the narrowing holds up — chapter 4 showed that "agent" tells a reviewer something specific and checkable about where control flow is decided. Across the boundary the narrowing does not survive. The listener does not hear "runtime control flow"; they hear the ordinary word, and they reason correctly from it to conclusions that are wrong about your system.

Why the narrowing stops at the boundary is mechanical rather than social, and it is worth knowing because it tells you that no amount of care in the writing will fix it. A term of art is narrow only inside the group that maintains the narrowing. Outside that group a reader has no way to suspect that a word they use fluently has been redefined, and a design document carries no signal that a redefinition is in force. So the narrow sense stays with the engineers, the ordinary sense travels with the document, and the only lever you hold is which word goes into the document.

That is the failure to design around: not that people are confused, but that they are reasoning well from a definition you handed them by accident. The consequence for where you spend your effort is direct. A caveat added after the fact has to defeat an inference the listener drew correctly, and it arrives as a hedge where a description was wanted, which is a much harder job than never prompting the inference at all.

Two sentences, anchored to something they already run#

The version that works is short, mechanical, and free of any verb that implies judgement:

It is software that, over and over, asks the model which one of a short list of actions we wrote should run next, runs it, shows the model what came back, and asks again. The list of actions is fixed before it is deployed and every run has a hard limit on how many times round it can go, after which it stops and reports what it found.

Every verb there was chosen to put an action somewhere specific. It asks the model, which places the model in the position of being consulted rather than of acting. It runs the action, which places the side effect in code you own. The list is fixed before it is deployed, which is a statement about ordering in time that somebody could verify by looking at a deploy artefact rather than a claim about intent. And the hard limit on how many times round it can go is the fourth part of the machine, stated as a guarantee because it is one.

That is the whole definition and it fits the four parts from chapter 1 without using any of their names. What makes it land, though, is the anchor that follows it — an existing workflow the listener owns, described as a comparison to the process, not to the person who performs it:

Your triage checklist has four steps: look up the account, check the last thirty days of charges, check for an open ticket, then write a note or escalate. This runs against the same four systems. The difference is that instead of going down the checklist in order, it picks the next step after seeing the result of the last one — so it may skip the charges lookup when the ticket is about a password reset. It cannot invent a fifth step, and after twelve steps it stops and hands back whatever it has.

Comparing to the process rather than to the person is the whole of the technique, and the reason is that a person performing a checklist supplies exactly the properties the two sentences above spent their effort removing. Anchor the description to the colleague who works the checklist, and discretion walks back in through the anchor.

The checklist is an analogy as well, and it breaks in two places worth naming aloud. A person working a checklist who discovers that a fifth step is needed will take the fifth step; the software cannot, because the fifth step does not exist for it, and no amount of its being obviously required will bring it into existence. That is the break most likely to mislead, since the ability to step outside a procedure is precisely the property a listener assumes carries over from the human version. The second break is milder and the paragraph above already contains it: a checklist's steps are all performed, in order, while this may skip one and may take them in a different order from one run to the next.

Everything a non-technical colleague needs in order to plan around the system is in those two paragraphs: what it can do (four things), what it cannot do (a fifth), what varies between runs (order and count), and what is guaranteed (it stops). Nobody has been told the software understands, reasons, wants, tries or decides.

The substitutions that do the work#

Most of the damage comes from a small set of verbs, and most of the repair is replacing them. Keeping a list like this in your head is worth more than any amount of caveating after the fact.

The right-hand column is the point. Each substitution is not a matter of taste or humility; it is the difference between a colleague forming an accurate model of what to expect and forming an inaccurate one.

The table is read across rather than down, because a row is a gap and not a level. The left column is what an engineer says without thinking about it, the middle says the same thing about the mechanism, and the right names the conclusion a listener draws from the left. What a row is worth is the width of that gap and what sits in it. Two of the gaps cost hours directly: the learning row hides a project behind a property, so somebody plans on accuracy improving by itself and nobody funds the pipeline that would have made it improve; the self-healing row hides the difference between failures being handled and failures being bounded, which is discovered when a bounded failure is met by an operations plan written for a handled one.

The other gaps cost risk rather than time. The autonomy row's gap is accountability for a particular run, and the goal row's is the possibility of objectives diverging from yours; both surface as agenda items in a risk review about a system that has neither. The understanding row's gap is the shape of the failures, since a listener who believes a system understands expects its mistakes to be sensible ones, and a labelling error is not sensible in that sense — it is a class boundary the sample never covered. The reasoning row's gap is an audit trail that does not exist, found when somebody asks to see the chain of judgement behind a decision and there is no chain, only a sequence of selections.

The refund row is the sharpest, and it is worth stating in its strongest form, because it is a belief that a defence works. A stakeholder who believes the software decides will ask you to instruct it not to exceed £50, and will be satisfied when you say the prompt tells it not to.

That belief is tempting for good reasons. The prompt is where instructions go, it is text you control and can show to anybody, and it says exactly the same thing on every single run. That constancy is what makes it feel like a control. But the quantity being held constant is the instruction, while the property that decides whether money leaves the company is whether the model's next requested action respects it — and that varies with the ticket, with what the conversation has already said, and with the model version behind the endpoint. An inspectable quantity that never moves, sitting beside a dangerous property that does move and that nobody measured, is the signature of a defence that is not one. A stakeholder who has been told the model selects an action and your code checks the amount will ask where that check lives, which is the question that protects the company.

One row replaces a word with evidence rather than with another word, and the evidence repays being taken apart, because a reader who cannot take it apart cannot tell what it covers. The row that retires "it understands the ticket" offers a labelling rate on a fixed sample, and there are three separate commitments inside it. There is a denominator somebody chose: three hundred tickets, selected once and then held still, so that the score can be compared against itself next quarter. There is a set of right answers written down before the system was run against them, which is what stops the score from being the system's own opinion of itself. And there is a rate over that denominator, which converts directly into a count: nine tickets in every hundred labelled wrongly, and over three hundred tickets that is exactly twenty-seven mislabelled tickets sitting in a file somebody can open and read.

The count is what makes the sentence useful and it is also what bounds it. Twenty-seven readable failures license two things: a prediction about tickets drawn the same way the three hundred were drawn, and a list of the specific ways this system gets tickets wrong. They license nothing whatever about tickets that are not drawn that way — a new product line, a different language, a week in which a billing bug generates a shape of complaint the sample never contained. That is the honest scope of any measured rate, and attaching it costs one clause and stops a stakeholder carrying the figure into a domain it never measured.

Learning, budgets, and actions nobody approved#

Does it learn from corrections? No. The model is the same on every run; the message list is discarded when the turn ends; a correction typed into a ticket goes into your database and never reaches the model unless a tool reads it back. Those three clauses are three links of one chain, and following them is what turns the answer from a policy into a structural fact: the weights behind the endpoint do not move, the only input that varies between calls is a list the loop throws away, and the store the correction lands in is not among the model's inputs. If you want corrections to change behaviour, someone has to build that — as a tool that retrieves similar past corrections, or as an update to the prompt — and it is a project, not a property.

Can we give it a monthly refund budget? Not by telling it. A running total that must not be exceeded is state the model cannot see and would not reliably respect if it could, so it belongs in the code that executes the refund: read the month's total, compare, refuse. The first half of that sentence carries the weight. The model sees exactly one thing, the message list for the turn in front of it, and a total accumulated across other turns, other users and other days is not in that list — so an instruction to stay inside the budget is an instruction to respect a number the model has no way to read. That answer is more reassuring than the one the question expected, because a limit enforced in code is a limit, whereas a limit in a prompt is a preference.

What if it does something we have not approved? Within the four actions, it will sometimes choose the wrong one or the wrong arguments, and the mitigations are the ordinary ones — validation, thresholds, an approval step on anything irreversible. Outside the four actions, nothing can happen, because there is no mechanism by which the model reaches anything you did not write a function for. That impossibility is worth deriving rather than asserting, since it is the strongest claim you have. The model's entire contribution per iteration is a name and an argument object. Your dispatcher looks that name up in a table you wrote. A name absent from the table has nothing behind it to execute, so no execution occurs — not as a matter of policy, but for want of a callable.

That distinction between "unlikely" and "impossible" is the most valuable sentence in the whole conversation, and it is only available to you if you have not already implied the software has initiative. It also rests on one property that a later change could quietly remove. The impossibility holds because the table is closed and you wrote it; a dispatcher that resolves the model's string against a registry other teams can register into no longer has a table you wrote, and what can run stops being knowable from your file. The claim degrades from impossible back to unlikely on the day that change lands, which makes it a design decision to defend rather than a fact about agents in general — and it is where the next topic opens.

The nightly job, seen from the other side of the table#

Chapter 4 settled that the 800-ticket classification job is a chain, and concluded that renaming the heading was the right fix. The reason the rename matters is mostly outside engineering, and it is worth being explicit about what the word cost there.

An operations lead reading "our classification agent" in an architecture inventory forms a picture of software that makes choices about tickets. From that picture, entirely reasonable plans follow: asking whether the agent could also take on escalations, asking what oversight it needs, raising it in a risk review as a system that acts on customer records, wondering aloud in a planning meeting whether triage headcount can come down. Each of those consumes real hours, and each one is a response to a system that does not exist. Meanwhile the actual property the lead needed to know — that a re-run after a partial failure silently overwrote categories a human had corrected — was invisible, because nothing in the word "agent" points at re-run semantics.

The second half of that cost is worth separating from the wasted hours, because it is the half that does damage rather than merely burning time. A name does not only invite questions about a system that is not there; it routes attention away from the questions that would have found the defect that is. "Agent" points a reader at bounds, oversight and autonomy. "Pipeline" points them at idempotency, partial failure and what a second run does. The defect in this job was of the second kind, and nobody went looking for it, because the name had already told them where to look.

Call it the nightly ticket categoriser and the same lead reads it correctly in four seconds: something fills in a column overnight. The questions that follow are the useful ones. What does it do when it is not sure. What happens when it runs twice. Naming a thing what it is costs nothing and routes attention to where the defects actually are, which is the argument for precision in both directions — an agent that gets called a pipeline is the same failure with the risks pointing the other way.

There is a version of this that goes too far. Deflating the description until it sounds like nothing is its own distortion, and it makes people underestimate a system that really can take several unplanned steps against production data. The target is not modesty; it is a description from which the listener can predict what happens. "It picks the next of four steps after each result and stops after twelve" is not modest and not inflated. It is what the software does.

The working test for anything you write down is whether a listener could use it to predict the next case without asking you. Mechanical verbs, a stated count of actions, a stated bound, and a comparison drawn to a process rather than to a person give them that much. A noun borrowed from agency law gives them a picture instead — coherent, confidently held, and wrong in ways that consume other people's weeks before anybody notices which sentence caused it.

The vocabulary is now the same on both sides of that table, which means the remaining work is construction rather than definition: deciding which of your existing endpoints deserve to become tools, writing descriptions the model can select on, choosing the cap from your own distribution of iteration counts rather than a round number, and reading enough transcripts to find out which of those decisions were wrong. That is where the rest of this material goes.