A DOM under the pixels changes which controls are available

In one standup two engineers described their work with the same four words. The first was automating a supplier portal in Chrome: she took a full-page screenshot every step, sent it to a vision model with the current goal, and got back an instruction like "click the Approve button on the third row". The second was automating a vendor ERP that his company reached through a Citrix session — a remote desktop streamed as video into a browser tab, running a Windows application his team had no credentials to install locally. He also screenshotted, also asked a model, also clicked. Both said they were building a screenshot agent.

Both of them were using the same capability, and it is worth naming plainly before the difference between the two jobs starts to matter. A vision-language model — VLM, a model that takes an image alongside text and answers in text, which is the image input from earlier in the course applied to a new kind of image — can be pointed at a picture of a working interface rather than at a scanned document. Document parsing asks such a model what a page says. Driving software with one asks what is on this screen, where each control sits, and which of them to operate next, which is the loop usually called computer use, or screenshot-driven automation: capture the screen, send the image with a goal, receive an action, dispatch it, capture again. The step in the middle, where a phrase like "the Approve button on the third row" has to become a particular control at a particular place, is UI grounding, and it is where most of the failures in this topic begin.

What makes the trouble worth taking on is access. A great deal of software offers nothing you can call: the vendor admin console with no public API, the ERP reachable only through a remote desktop, the internal tool whose owning team disbanded years ago, the portal that will issue credentials for a browser session and for nothing else. A screen is the one interface every application has, so anything a person can operate by looking and clicking can in principle be operated by a model that looks and clicks, with no cooperation from whoever built it. The same capability is used on images nobody will ever click — a revenue chart in an investor deck, a dashboard pasted into a ticket, a pricing table that survives only as a picture of a PDF page — where somebody wants the numbers and the data behind the picture is unavailable. Chapters five and six deal with that half of the subject; this one and the three after it deal with acting on an interface.

The two engineers differ in what sits underneath the picture they are both looking at. A page in Chrome comes with the DOM (Document Object Model), the browser's live tree of every element that makes up the page, which an automation library can query, hold references into and act through. A Citrix session is a video of somebody else's Windows desktop, and it arrives as pixels with no tree behind them.

The difference showed up three weeks later when the first engineer's agent misclicked once in about two hundred steps and the second engineer's misclicked once in nine, and the fix that worked for her was unavailable to him. She could ask the page, at the moment of the click, which element sat under the coordinates the model had chosen, and refuse to click if it was not the element she had asked about. He had a bitmap. Everything downstream in this topic — where the coordinate frames go wrong, how numbered marks replace raw boxes, what you assert before you act — is cheap or expensive depending on which of those two situations you are in, and it is worth being explicit about which one you have chosen before you write any of it.

What the page can tell you when it is a page#

Chrome running under Playwright or Puppeteer exposes three things at once for the same instant: the rendered pixels, the DOM, and the accessibility tree the browser computes from that DOM for screen readers, using the markup itself together with any ARIA attributes the page carries — Accessible Rich Internet Applications, the role, aria-label and related attributes by which a page declares what a control is to assistive technology. The accessibility tree matters more than most backend engineers expect, because it is already a semantic list of the interactive things on screen with roles and names attached — button "Approve", textbox "Search suppliers", row "INV-4417" — filtered down from the DOM's thousands of nodes to the few dozen a user could actually operate. You can enumerate it, filter it to what is visible in the viewport, and hand the model a numbered list that corresponds one-to-one with objects you hold references to.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page(viewport={"width": 1280, "height": 800})
    page.goto("https://portal.example.com/invoices")

    # Interactive candidates, in DOM order, that are actually on screen.
    candidates = page.locator(
        "button, a[href], input, select, [role=button], [role=tab]"
    ).all()
    visible = [c for c in candidates if c.is_visible()]

    for i, el in enumerate(visible):
        box = el.bounding_box()          # CSS pixels, viewport origin
        name = el.get_attribute("aria-label") or el.inner_text()[:40]
        print(i, el.evaluate("e => e.tagName"), name, box)

Each entry in visible is a handle, not a coordinate. Playwright resolves it back to a live node at the moment you act, re-runs its actionability checks — attached to the DOM, visible, stable in position for two consecutive animation frames, receiving pointer events, enabled — and only then dispatches. If the row moved because a background poll re-sorted the table, the handle follows the row rather than the pixel. If the element was removed, the click raises instead of landing on whatever slid into its place.

It is worth being clear about why a vision model is in this picture at all when the DOM is available. Selector-based automation has been solved for fifteen years and breaks for a well-known reason: the selectors are written against markup that changes, and a redesign that moves a button from a <button class="btn-primary"> to a <div role="button" data-testid="approve-v2"> silently breaks every test that referenced it. The model is there to do the part that was previously a human's job, which is looking at an unfamiliar screen and deciding what to press next. That is a different responsibility from executing the press, and keeping the two separate is what lets you keep the model's judgement while retaining the machinery's reliability.

The same environment gives you a cheap identity check, which is the control the rest of this topic keeps returning to. Having decided to click index 7, you can ask the page what index 7 currently is, and compare that against what the model was looking at:

target = visible[7]
target.scroll_into_view_if_needed()

# Identity re-read: the invoice number in this row, right now.
row_id = target.locator("xpath=ancestor::tr").get_attribute("data-invoice-id")
if row_id != model_choice["invoice_id"]:
    raise StaleTargetError(f"model saw {model_choice['invoice_id']}, page has {row_id}")

target.click()

Four lines, and the class of failure where an agent approves the wrong invoice because the table re-sorted stops being possible. It costs nothing per step because the browser is already holding the information.

What is left when the pixels are all there is#

A Citrix or VNC session, a native macOS application driven through screen capture, a game-engine or canvas-rendered interface, a customer's screen shared over a support tool — none of these hand you an element to hold. The unit of action is an OS-level event at a screen coordinate: move the mouse to (x, y), click, type. There is no handle to resolve, no actionability check, and nothing to re-read.

Canvas is the case people misclassify most often, because it looks like the friendly situation. A Flutter web build using CanvasKit, a spreadsheet grid drawn on <canvas> for scroll performance, a mapping or CAD tool, a trading terminal — all of these serve a real DOM over HTTP, and querying it returns one <canvas> element covering the viewport with nothing inside it. The presence of a DOM is not the question; whether the DOM contains the control you want to press is the question. A useful five-minute test before committing to an architecture is to open the target in Chrome, run document.querySelectorAll('button, [role=button]').length in the console, and see whether the number resembles what your eyes count on screen. Zero, or three when you can see forty, tells you which half of this chapter you are living in.

Between the two extremes sits the platform accessibility layer, which is the substitute worth reaching for before you resort to template matching. macOS exposes the AX API, Windows exposes UI Automation, Linux desktops expose AT-SPI, and all three were built so screen readers could enumerate controls in native applications. Where the target application implements them properly, you get roles, names and screen rectangles for real controls, which restores enumeration and a weak form of identity. The catch is coverage: applications written against custom drawing layers, older Win32 dialogs, Electron apps with accessibility disabled for performance, and anything rendered remotely as video expose either nothing or a single opaque window. A remote desktop session is the hard case precisely because the accessibility tree lives on the machine you are not talking to; the client is showing you a video stream of somebody else's screen.

The right-hand column is not hopeless, and the substitute for an identity re-read is to make the pixels themselves carry identity. Before dispatching a click at (x, y), crop a small region around the target from the screenshot the model reasoned over — 120 by 40 pixels is enough for a button — capture the same region again from the live screen, and compare. If the two crops match closely, the thing you looked at is still there. If they do not, something arrived in between and you abort rather than clicking blind.

import numpy as np, mss
from PIL import Image

def crop(img, x, y, w=120, h=40):
    return np.asarray(img.crop((x - w // 2, y - h // 2, x + w // 2, y + h // 2)))

def unchanged(before, after, tol=0.02):
    diff = np.abs(before.astype(np.int16) - after.astype(np.int16))
    return (diff > 12).mean() < tol      # <2% of pixels moved meaningfully

with mss.mss() as sct:
    shot = sct.grab(sct.monitors[1])
    live = Image.frombytes("RGB", shot.size, shot.rgb)

before = crop(planning_screenshot, cx, cy)
after = crop(live, cx, cy)
if not unchanged(before, after):
    raise StaleTargetError("region under target changed since capture")

This is weaker than a DOM re-read in ways worth stating plainly. It cannot tell you that the button under the cursor is now a different button with identical styling, which is exactly what a re-sorted table of similar rows produces. A blinking cursor, an animated spinner, or a hover state triggered by the mouse arriving trips the threshold and gives you a false abort. And it costs a full screen grab, which on a 2560×1440 display through a remote session is 80 to 300 milliseconds — time during which the screen can change again.

Choosing before you build, not after#

The reason to settle this first is that the two paths diverge in what your test harness even looks like. The DOM-backed path lets you write assertions in the same language as your actions, so a failed grounding attempt raises an exception in CI. The pixel-only path forces you to build a fixture whose state you control absolutely — a local app or a recorded desktop, at a fixed resolution and scaling — because otherwise you have no known-good answer to compare against, and an agent that "seems to work" is an agent whose success rate you cannot measure.

The two paths also price differently per step, which changes what you can afford to do defensively. A DOM re-read is a synchronous call into the browser that returns in about a millisecond, so you can afford one before every action and another after it, and nobody notices. A pixel re-read is a screen grab plus an image comparison, and on a remote session it can cost more than the action itself, which pushes teams toward doing it only sometimes — which in practice means doing it on the steps that turn out not to need it. If you are on the pixel-only path, decide the policy explicitly rather than letting latency decide it for you: every step that mutates state gets a verification, and read-only navigation steps can skip it.

That fixture is worth building on day one either way. The first useful exercise in this topic is small: take a screenshot of a UI whose state you know exactly, ask a model to identify and act on one control in it, then check the result against that known state rather than against your impression of whether it looked right. The reasons this check earns its place are that a screenshot can be stale by the time the model answers, an overlay can be sitting on top of the element the model described, and screen scaling can move where the coordinates land — three separate causes that produce the identical symptom of a plausible-looking action hitting the wrong control.

This chapter has drawn the line between the two environments and named the substitutes available on the poorer side of it. What it has not done is examine the loop itself, which is where both environments leak: the model reasons over a screenshot taken at one instant and the click lands at another, and the interval between them is long enough for the interface to have moved on.