Screenshot or API, judged on success, upkeep and access

A vendor admin console with no public API. The ops team does about 400 updates a month in it, three minutes each — open the account, change a plan code, save, confirm. A week of prototyping with everything from this topic in place, marks over an accessibility tree, identity re-reads before each click, a fixture with scheduled interference, gets a screenshot-driven agent to 92% task success. The vendor ships UI changes roughly monthly. The question on the table is whether to build it.

The instinct is that 92% is respectable and the remaining 8% can be logged and iterated on. The arithmetic does not agree, and it is worth doing in full because the reason it does not agree is the maintenance term rather than the success rate.

The arithmetic, with the failure costs included#

Twenty hours a month of ops time is what the manual process costs — 400 tasks at three minutes. An agent at 92% does not save twenty hours, because three things still consume time. The 32 failures need detecting, triaging and redoing by hand, which is the original three minutes plus roughly two minutes to work out what went wrong, so 32 × 5 minutes is 2.7 hours. The 368 successes need some level of confirmation, because an agent that silently succeeds and an agent that silently does the wrong thing look identical in a log unless something checks; at 30 seconds per confirmation that is 3.3 hours. Running the agent, watching the queue and handling the odd stuck run is another hour or so. The saving lands near 13 hours a month.

Against that sits the churn, meaning the rate at which the vendor's interface changes underneath an agent that was built against how it looked last month. A monthly UI change to the console will break something — a renamed control, a new confirmation step, a field that moved into a collapsed section — and repairing it costs a developer somewhere between half a day and a day, including noticing, reproducing, fixing and re-running the fixture suite. Call it six hours a month, and note that it is engineering time rather than ops time.

tasks_per_month   = 400
minutes_manual    = 3
success_rate      = 0.92
minutes_recover   = 5      # triage + redo a failed task
minutes_confirm   = 0.5    # verify a successful task
ops_rate, eng_rate = 35, 95            # loaded €/hour
eng_hours_churn   = 6                  # repair after each monthly UI change

manual  = tasks_per_month * minutes_manual / 60
agent   = (tasks_per_month * (1 - success_rate) * minutes_recover
           + tasks_per_month * minutes_confirm) / 60 + 1

saving  = (manual - agent) * ops_rate          # €455/month of ops time
cost    = eng_hours_churn * eng_rate           # €570/month of engineering time
print(f"{manual:.1f}h manual, {agent:.1f}h with agent, net €{saving - cost:+.0f}/month")
# 20.0h manual, 7.0h with agent, net €-115/month

Roughly break-even, slightly negative, before counting the week of prototyping already spent and the further two to three weeks to make a prototype into something ops can run unattended. A project that saves thirteen hours of one team's time by spending six hours of a scarcer team's is not obviously worth doing, and the honest answer at these numbers is not to build it — with a short list of things that would change that.

What moves the answer#

Volume moves it most, and not linearly, because the churn term is fixed. Ten times the tasks means ten times the saving against the same six hours of repair, and at 4,000 updates a month the same model is clearly worth building. The confirmation term scales with volume too, though, so somewhere on the way up the review strategy has to change from confirming every task to sampling, and sampling only works if the failures are the kind you can find later. This is where the same 92% becomes a different proposition. At 400 tasks a month, 32 failures are a person's afternoon. At 400,000, they are 32,000 wrong or incomplete updates, which no team reviews, so the entire safety argument has to shift from human review onto the properties of the failures themselves: they must be self-detecting, they must be reversible, and the actions must be idempotent so a retry is free.

Failure shape matters as much as failure rate. An agent that fails by throwing a stale-target exception from chapter 2's identity re-read is cheap, because you know which tasks did not happen and you can retry them. An agent that fails by saving the right value on the wrong account is expensive at any rate, because the cost is a support incident rather than a redo. Before accepting a success number, separate it into the visible failures and the silent ones; a system at 92% with all failures visible is a different system from one at 96% where a quarter of the misses are silent.

The environment from chapter 1 changes both the success rate and the churn cost. A console rendered as ordinary DOM in a browser gives you handles, occlusion tests and assertions, and a renamed button is a one-line fix to a selector fallback rather than a re-tuned prompt. The same console inside a Citrix session gives you none of that, so the same monthly redesign costs more to absorb, and the success ceiling is lower to begin with. If the console is DOM-backed, it is also worth asking whether a plain selector-based script does most of the job at a fraction of the cost, using the model only for the steps that genuinely need judgement.

Locale is the axis people forget until it bites. A screenshot agent is grounded in rendered text, so switching the console to German changes every label the model matches on, and a layout tuned for English strings reflows when the German ones are 30% longer. An API is indifferent to this. If the vendor tool will be used by teams in more than one language, the maintenance term multiplies by the number of locales rather than staying fixed, and the case collapses.

Two more things would change the answer outright. If the vendor has an API on their roadmap for the next two quarters, building a screenshot agent means paying the churn tax until it lands and then throwing the work away. And if any of the 400 updates is irreversible — cancelling a subscription, releasing a payment, deleting an account — the analysis stops being about hours, because the cost of the 8% is no longer a redo.

The four axes side by side#

The right-hand column wins on every axis except the last, and the last is the reason this topic exists. The strongest case for driving a system by screenshot is the one where the comparison never happens because there is nothing to compare against: a vendor admin tool with no API at all, running in a single locale, with a layout that changes rarely, at a volume low enough that a human reviews the output, doing work that is reversible if it goes wrong. Each of those five clauses is load-bearing. Take away the single locale and the maintenance doubles; take away the human review and the 8% goes unexamined; take away reversibility and a wrong click is an incident.

The cases that look similar and are not are worth naming. A system with an API that is merely inconvenient — badly documented, requiring a procurement conversation, missing two of the fields you want — is not a case for screenshots, because a week of negotiation is cheaper than a year of layout churn. A public website you do not control is a weaker case than it appears, since it changes without notice, runs A/B tests that show different layouts to different sessions, and may prohibit automation in its terms. A high-volume internal process is the weakest case of all, because internal means you can get the database.

Where this leaves the reader#

The through-line of this topic is that vision makes an interface reachable and does not make it verifiable, and everything worth building rests on adding the verification back. Which controls are available to you at all is set by whether a DOM or an accessibility tree sits under the pixels. What you assert before acting is set by the gap between the capture and the click. Whether your clicks land where the model pointed is a question about six coordinate conventions with distinguishable signatures. Numbered marks remove the coordinates from the runtime path entirely where candidates can be enumerated. Axis brackets and legend attribution turn chart values from estimates into bounded readings, and where a structured source exists the model should be locating the data rather than reading it.

The next move is to run all of it against something real. Take an interface you have — an internal admin page is ideal, because you can change it — and get a model to read a screenshot of it and act on one control, then check the result against the state you know rather than against how it looked. Ground a bounding box on an element and verify it against the rendered viewport at two device pixel ratios. Extract data from a chart you already know the numbers for and see how many values survive the bracket check. Take a dense table at small type and reproduce the failure deliberately, then add the guardrail that turns it into an abstention. Then do the arithmetic above for one process your team actually runs by hand, with your own rates and your own churn, and see which side it comes out on.