Vision-language capability beyond document parsing

Read these first

• Image inputs

What you will know

• Get a model to read a screenshot and act on it

  Test the result against a known UI state, because stale content, overlays, and screen scaling can make an apparently plausible action target the wrong control.

• Ground a bounding box on a UI element

  Verify the coordinates against the rendered viewport; resizing, device-pixel ratios, and coordinate-system mismatches commonly shift clicks away from the intended element.

• Extract data from a chart image

  Check the extracted values against visible axes, legends, and labels, since series selection and approximate visual estimation can produce convincing but numerically wrong answers.

• Reproduce a vision failure on small text, dense tables, or precise measurements and add a guardrail or fallback

  The useful boundary is operational: reject low-confidence cases, request a clearer image, or route exact work to structured data instead of silently accepting an answer.

• Decide when a screenshot beats an API

  Compare both paths on task success, latency, maintenance, and access constraints; screenshots can bypass integration work but are brittle under layout changes and localization.

Chapters

1. 1

   A DOM under the pixels changes which controls are available

   Two different jobs get called the same thing. Browser automation with a vision layer has a DOM and an accessibility tree beside the screenshot, so elements can be enumerated, marked, clicked by handle and re-read after the model chooses one — and most of what follows becomes verifiable. True pixel-only computer use, which is where the vendor tool with no API, the remote desktop session, the native app and the canvas-rendered UI all live, has none of that: no element handles, no identifier to re-read before acting. What replaces them is re-capturing the crop around the target and comparing it against the pre-click image, or an OS-level accessibility API where one exists. Decide which of the two you are building before you reach for any control in this chapter set.

   Read chapter
2. 2

   Acting on a screenshot means asserting the state you assumed

   The loop is capture, ask, act, and the damage lives in the gaps between the three. Build it against a fixture UI whose state you control, then watch what a toast overlay or a re-sorted table does when it arrives in the 900 milliseconds between the screenshot and the click. The control that survives is re-reading the identity of whatever now sits under the target and aborting on a mismatch.

   Read chapter
3. 3

   Where a click lands when the coordinate frame is wrong

   A box returned as [0.12, 0.34, 0.28, 0.41] might be normalised or absolute, xyxy or xywh, origin top-left or bottom-left, CSS pixels or device pixels. Each mistake leaves a signature: a constant offset accuses the origin or the viewport, while error that grows with distance from the top-left accuses a scale factor such as devicePixelRatio 2. Round-trip a known element at both ratios in CI so the class cannot ship twice.

   Read chapter
4. 4

   Numbered marks beat asking for pixel coordinates

   Overlay indexed markers on candidate elements taken from the accessibility tree, then ask the model for a number rather than a box. That converts coordinate regression, which vision models are measurably weak at, into a discrete choice among labelled options. It also hands you a real element handle, so the click dispatches through the DOM instead of through a guessed pixel that may no longer belong to anything.

   Read chapter
5. 5

   Chart values are estimates until the axes bracket them

   Reading a bar off an image is visual estimation, and estimation yields numbers that look plausible and are wrong. Make the model name the two axis ticks each value falls between and reject anything outside its bracket. On multi-series charts, also require it to quote the legend label and colour of the marks it actually read, because that is the only check that catches values lifted from the wrong blue line.

   Read chapter
6. 6

   Vision locates the table; structured data supplies the figures

   A model can get a dense pricing table's layout exactly right and still transpose two adjacent numeric columns, and no prompt reliably fixes that. Reproduce the failure yourself on small text and tight measurements, then pick a real response: reject on low confidence, ask for a cleaner capture, or use the model only to find the region and read the numbers from a structured source. When no structured source exists — the figures live only in a chart inside a PDF — the design is per-value abstention rather than per-document, because the errors that matter are not the 2% ones: a misread unit, a log axis or a secondary y-axis is wrong by an order of magnitude, and a screening model thresholds the number, so an error near the shortlist boundary flips a decision while the same error mid-range costs nothing.

   Read chapter
7. 7

   Screenshot or API, judged on success, upkeep and access

   Compare the two paths on four axes: task success, latency, maintenance cost, and whether an integration exists at all. Screenshots skip integration work but break on layout changes and second locales, so the arithmetic that decides it is monthly UI churn against hours saved, plus how much human review the volume can absorb. 92% success is a different proposition at 400 tasks a month than at 400,000.

   Read chapter