Acting on a screenshot means asserting the state you assumed

A support-tooling team ran their new agent against a staging queue of forty tickets and it deleted the wrong one. The log was not ambiguous about what happened, only about whose fault it was:

14:22:07.104  capture   viewport 1280x800, 214 KB png
14:22:07.196  request   claude, 1 image + 1.2 KB text
14:22:07.982  response  {"action":"click","target":"Delete button, row 4",
                         "row_label":"TCK-7741","point":[1102,318]}
14:22:08.011  dispatch  click (1102,318)
14:22:08.240  observe   row TCK-7739 removed

Nine hundred and seven milliseconds separate the capture from the click. In that window a background poll finished, three tickets that had been escalated moved to the top of the queue, and every row below them shifted down by one. Row 4 at 14:22:07.104 was TCK-7741. Row 4 at 14:22:08.011 was TCK-7739. The model was right about everything it could see, the click landed exactly where it was aimed, and a ticket that nobody had asked about is gone.

The loop has two gaps, and they are different sizes#

Every screenshot-driven action decomposes into capture, ask, act, and the interesting part is the intervals rather than the steps. The first interval runs from the moment the framebuffer — the block of memory holding what is currently drawn on screen — is read to the moment the model's answer comes back, and it is dominated by inference. The second runs from the decision to the dispatched event, and it is short unless your code does something else in between — logging to a remote service, waiting on a rate limiter, asking a human for approval.

The numbers are worth carrying around because they set the size of the problem. A viewport capture through the Chrome DevTools Protocol, the low-level channel Chrome exposes for automation and the layer Playwright and Puppeteer sit on top of, costs 30 to 60 milliseconds at 1280×800 and two to three times that at 4K. PNG encoding and base64 inflation add another 40 to 120 milliseconds for a screenshot in the 200 KB to 900 KB range. Model latency for a single image plus a short prompt runs 600 to 1,500 milliseconds for a mid-sized vision model, and considerably more if you are asking for reasoning before the action. Dispatching the event itself is single-digit milliseconds.

Roughly a second of exposure is not an implementation defect you can optimise away. Halving inference latency halves the window and does not close it, and a UI that repaints on a five-second poll will land inside a 400-millisecond window often enough to matter at volume. Treating this as a latency problem is the tempting move and the wrong one, because the fix that works does not depend on the window being small.

Before you design around the window, it is worth measuring how often the interface actually moves inside one, because the answer varies by two orders of magnitude between applications. A MutationObserver, the browser API that calls you back whenever the DOM beneath a chosen node changes, attached to the container you are operating on and logging the timestamp and node count of every mutation for an hour of ordinary use, gives you the collision probability directly: count the one-second intervals containing a mutation that changed layout, and divide. A settings form that only changes when you change it will come back near zero. A ticket queue on a five-second poll will come back near twenty percent, and a live dashboard with streaming updates will come back near one. That number tells you whether the guard below is a formality or the thing holding your agent together.

There is a second way the window grows that has nothing to do with inference speed. Teams frequently add a human approval gate for destructive actions, which sounds like a strengthening of safety and pushes the dispatch minutes rather than milliseconds after the capture. A reviewer looking at a screenshot from four minutes ago and clicking approve is authorising an action against a world that no longer exists, and the coordinates they approved are now close to meaningless. Approval gates need the same re-read as automated dispatch, applied at the moment the human says yes rather than at the moment they were asked.

A fixture that changes on purpose#

You cannot debug a race you cannot reproduce, and staging environments produce these collisions at rates too low to iterate against. What works is a fixture whose mutations you schedule: a static page holding a table you control, with a script that injects exactly the interference you want at exactly the offset you want.

<script>
  // Fixture: the interference an agent will meet in production,
  // fired on a schedule so the same run is reproducible.
  const rows = [...document.querySelectorAll('tbody tr')];

  setTimeout(() => {                       // t+310ms: escalations jump the queue
    const tbody = document.querySelector('tbody');
    [...rows].reverse().slice(0, 3).forEach(r => tbody.prepend(r));
  }, 310);

  setTimeout(() => {                       // t+520ms: toast covers the top-right
    const t = document.createElement('div');
    t.className = 'toast';                 // position:fixed; z-index:9999
    t.textContent = 'Sync complete';
    document.body.append(t);
    setTimeout(() => t.remove(), 4000);
  }, 520);
</script>

Run the agent against this fifty times and you get a measurable failure rate instead of an anecdote. The team above measured 11 wrong-row deletions in 50 runs with no guard, which is what you would expect when the re-sort fires reliably inside the window. More usefully, the fixture separates two failures that look identical from the outside. The re-sort moves the intended element away from the coordinates, so the click lands on a real control belonging to different data. The toast — a notification panel that appears over the page for a few seconds and then removes itself — lands on top of the coordinates, so the click is swallowed by an element that was not on screen when the model looked, and the action silently does nothing — which is the better outcome only if your agent notices, and an agent that assumes its clicks succeed will carry on to the next step believing a deletion happened.

Re-read what is under the point, not what you meant to click#

The control that survives both cases is to ask the live interface what currently occupies the target coordinates, and to compare that against the identity the model reported seeing. In a DOM-backed environment this is elementFromPoint plus one attribute lookup:

async function clickIfStillTrue(
  page: Page,
  point: [number, number],
  expected: { rowId: string }
) {
  const actual = await page.evaluate(([x, y]) => {
    const el = document.elementFromPoint(x, y);
    const row = el?.closest('tr');
    return {
      rowId: row?.dataset.ticketId ?? null,
      topmost: el?.tagName + '.' + (el?.className ?? ''),
    };
  }, point);

  if (actual.rowId !== expected.rowId) {
    throw new StaleTargetError(
      `model saw ${expected.rowId}, point now holds ${actual.rowId} (${actual.topmost})`
    );
  }
  await page.mouse.click(point[0], point[1]);
}

Two properties make this the minimum rather than one option among several. It is keyed on the coordinates the click will actually use, so it catches the case where the intended element still exists somewhere on the page and something else has taken its position — which a selector-based re-check would miss entirely, because page.locator('[data-ticket-id=TCK-7741]') would happily find the row at its new location and report that everything is fine. And it reads identity rather than appearance, so two rows rendered identically apart from their ticket number are distinguishable, which the pixel comparison from the previous chapter cannot manage.

The same check catches the toast, because elementFromPoint returns the topmost element that would receive the event. If the answer is a div.toast rather than a cell inside the expected row, the mismatch fires and you abort, retry the capture, and let the next iteration see the world as it now is. Retrying is the right response to a stale target: the information you acted on has expired, so you refresh it rather than forcing the action through.

One detail decides whether the toast case is caught or missed. An overlay styled with pointer-events: none is invisible to elementFromPoint, which will report the row underneath it and let the click through, and that is the correct behaviour for a decorative banner that genuinely does not intercept input. It is also a trap when the overlay is a modal backdrop that does take events, because then your check and the browser's hit testing agree and you have simply confirmed that the click will be swallowed on purpose. Comparing the topmost element's tag and class alongside the row identity, as the snippet above logs, is what lets you tell an ignorable decoration from a modal that has changed what the interface expects from you.

For the pixel-only environments of the previous chapter the shape is the same and the fidelity is lower. You re-capture the crop, and where the target carries readable text — a row label, an invoice number, an account name — you run OCR — optical character recognition, which turns the pixels of rendered text back into characters — on the crop and compare the string rather than the pixels, which restores the identity check the image diff was missing. It costs an extra 60 to 200 milliseconds and it is the difference between "these pixels look similar" and "this is the same record".

What the assertion does not buy you#

The re-read narrows the exposure window from roughly 900 milliseconds to the gap between the check and the dispatch, which is around a millisecond in the same process. It does not reach zero, and no client-side control can make it zero, because the interface can repaint between the assertion returning and the event being delivered. What that means practically is that this technique is sized for the ordinary case of an interface that updates on human timescales, and it is not a substitute for server-side safety on operations you cannot undo. If the action is a deletion, a payment, or an irreversible state transition, the durable protection is on the other side of the network: an idempotency key, which is a caller-supplied identifier the server uses to recognise a repeated request and apply it only once, a confirmation step that restates the record identity, or an API that takes the record id rather than the row position. The assertion prevents the agent from being wrong; only the server can prevent the wrong request from having effect.

It also does nothing about a model that was wrong at capture time. If the screenshot showed row 4 as TCK-7741 and the model reported TCK-7739 because it miscounted rows, the re-read compares the page against a bad expectation and either fires a spurious mismatch or, worse, agrees. Everything here assumes the model's report of what it saw is faithful to the screenshot, and holding it to that is a prompting question — asking for the row label alongside the action gives you something checkable, which is why the response format in the opening log carries row_label next to point.

The deeper assumption is that the point the model returned corresponded to the element it described in the image it was given. Once the model's coordinates are wrong relative to the screen — a different origin, a different unit, a different scale factor — the assertion still fires and still aborts, correctly, but it aborts on every single step and you have an agent that never does anything. Distinguishing "the world moved" from "my coordinate frame is wrong" is the next thing to get right, and the two have signatures distinct enough to tell apart without guessing.