Numbered marks beat asking for pixel coordinates

The supplier-portal agent from the first chapter had its coordinate frames correct, its device scale factor pinned in CI, and an identity re-read before every click. Over 200 runs against the fixture it still completed only 71% of tasks, and the failures were dull: a click 18 pixels below a 32-pixel-tall row, landing in the gap; a click on the label text next to a checkbox rather than the checkbox; a click that hit the correct button in the correct row of the wrong table because two tables sat side by side with identical columns. The model knew what it wanted. It was inaccurate about where.

Changing one thing in the request moved the same suite to 96%. Instead of asking for a point, the agent drew numbered badges on every interactive element it could enumerate and asked which number to press. The model's job became choosing among 34 labelled options, which is a task it is good at, and the agent's job became looking up option 12 in a list it had built itself.

Enumerate, draw, choose, resolve#

The pipeline has four stages and only one of them involves the model. Candidates come from the accessibility tree, filtered to what a user could actually operate right now — visible, in the viewport, enabled, not covered by something else. Each candidate gets an index and a badge drawn at its corner. The composite image goes to the model with the goal. The answer is an integer, which indexes back into the candidate array, and the click dispatches through the handle that array is holding.

type Candidate = { index: number; handle: ElementHandle; role: string; name: string; box: Box };

async function markCandidates(page: Page): Promise<Candidate[]> {
  const handles = await page.$$('button, a[href], input, select, textarea, ' +
                               '[role=button], [role=tab], [role=checkbox], [role=menuitem]');
  const out: Candidate[] = [];

  for (const h of handles) {
    const box = await h.boundingBox();
    if (!box || box.width < 6 || box.height < 6) continue;       // decorative or collapsed
    if (!(await h.isVisible()) || !(await h.isEnabled())) continue;

    // Is this element actually the topmost thing at its own centre?
    const onTop = await h.evaluate((el) => {
      const r = el.getBoundingClientRect();
      const hit = document.elementFromPoint(r.x + r.width / 2, r.y + r.height / 2);
      return el === hit || el.contains(hit);
    });
    if (!onTop) continue;                                        // occluded by a modal or sticky bar

    out.push({ index: out.length, handle: h, box,
               role: await h.getAttribute('role') ?? await h.evaluate(e => e.tagName.toLowerCase()),
               name: (await h.innerText()).trim().slice(0, 60) });
  }
  return out;
}

The occlusion test — whether anything else is drawn on top of the element, hiding it from a user and from a click — earns its line. Without it the candidate list includes everything behind an open dialog, and a model that can see only the dialog will sometimes still pick a number belonging to a button underneath it, because the number is in the list and the list is what you asked it to choose from.

Drawing the badges is the part people get subtly wrong. A badge centred on the element covers the element's own text, and the model then has to choose between controls it can no longer read. Placing the badge just outside the top-left corner, in a high-contrast colour with a thin outline so it survives on both light and dark backgrounds, keeps the element legible. Where badges would collide — dense toolbars, tightly packed table rows — nudging alternate badges to the opposite corner is enough at typical densities.

await page.evaluate((cands) => {
  const layer = document.createElement('div');
  layer.style.cssText = 'position:fixed;inset:0;z-index:2147483647;pointer-events:none';
  for (const c of cands) {
    const tag = document.createElement('div');
    tag.textContent = String(c.index);
    tag.style.cssText = `position:absolute;left:${c.box.x - 2}px;top:${c.box.y - 12}px;
      font:700 11px/1.4 monospace;color:#fff;background:#d6006e;padding:0 3px;
      border:1px solid #fff;border-radius:2px`;
    layer.append(tag);
  }
  layer.id = '__marks';
  document.body.append(layer);
}, candidates.map(c => ({ index: c.index, box: c.box })));

const shot = await page.screenshot();
await page.evaluate(() => document.getElementById('__marks')?.remove());

The pointer-events: none and the removal immediately after capture matter for the same reason: the overlay must not be present when the click is dispatched, and must not be capable of receiving it even if a timing bug leaves it there.

Ask for the number and the label#

The request should constrain the answer to the candidate set and demand one cross-check. Asking the model to echo the visible text of the mark it chose costs a dozen output tokens and catches the failure where badge 12 in the model's reading and candidate 12 in your array are different elements — an off-by-one in the mark list, a candidate filtered out after the badge was drawn, a badge that landed over a neighbour.

{
  "mark": 12,
  "mark_label_text": "Approve",
  "element_role": "button",
  "confidence": "high",
  "alternatives_considered": [17],
  "none_applicable": false
}

choice = model_response["mark"]
cand = candidates[choice]

if cand.name.lower() not in model_response["mark_label_text"].lower():
    raise MarkMismatch(f"mark {choice} is {cand.name!r}, model read {model_response['mark_label_text']!r}")

cand.handle.click()   # DOM dispatch, actionability checks, no coordinates involved

The click never touches a pixel coordinate, which removes every convention from the previous chapter from the runtime path in one step. Device pixel ratio, axis order, origin, scroll offset and provider-side resize all still affect what the model sees, and none of them can now affect where the event goes. That is the substantive argument for dropping raw coordinates in a production agent: the coordinate is a lossy re-encoding of a decision you already have in a better form.

The identity re-read from chapter 2 stays, because a handle can go stale between the capture and the click just as a coordinate can. What changes is that the re-read gets easier and stricter, since you are comparing a handle against the DOM rather than inferring identity from what sits at a point.

There is an ordering hazard inside the four stages that is easy to introduce and hard to see. The candidate list, the badges and the screenshot must all describe the same instant, so anything that mutates the page between enumeration and capture — a lazy image finishing, a sticky header collapsing on the first scroll, a font swap reflowing a toolbar — shifts elements out from under their badges. The screenshot then shows badge 12 sitting beside a different control than the one your array holds at index 12, and the label cross-check above is what turns that from a wrong click into an exception. Doing the enumeration and the badge drawing in a single page.evaluate call, rather than looping in Node with a round trip per element, closes most of the window; on a page with 40 candidates the per-element round trips cost 60 to 150 milliseconds, and the batched version costs under 15.

Three ways to point at a button#

The text-only accessibility tree is the option most likely to be underrated on the way past. It is cheap to implement and it fails on exactly the things a screenshot is for: which of two same-named buttons is the active one, whether a control is greyed out in a way the ARIA attributes do not reflect, whether an element the tree lists is actually hidden behind a cookie banner. It also grows expensive on dense pages, where serialising the tree can cost more tokens than the screenshot did. Marks combine the two, giving the model the picture and giving your code the handle.

The idea has a short and traceable history. Set-of-Mark prompting was introduced in 2023 as a way to improve visual grounding by segmenting an image — dividing it into regions that each contain a single object, a job normally done by a separate vision model — and overlaying numeric labels on those regions, and the observation that transferred to interfaces is that a UI already comes pre-segmented, since the accessibility tree is a free segmentation with semantics attached that no segmentation model has to be run to produce. Screenshot-driven web agents adopted it quickly, and it is now the default in most production browser agents. Where a segmentation is not free, the same shape still works with a detector in front of it: a model trained specifically on interfaces produces candidate regions, and those regions get the badges. Google Research's write-up on a vision-language approach for foundational UI understanding is a good account of why interface screenshots need their own region-focused treatment rather than a general image model pointed at a screen.

Where marks stop helping#

Marks are a solution to grounding, and they are not a solution to search. If the control the task needs is not in the candidate list — collapsed inside a menu, three screens down, behind a tab that has not been opened — the model is being shown a set that does not contain the right answer, and models are strongly disposed to pick the closest available option rather than object. That is why none_applicable belongs in the response schema, paired with actions that change the candidate set: scroll, open the menu, switch tab, then re-mark. An agent without that escape hatch trades a visibly wrong click for a confidently wrong one.

The economics also change in a way that is worth stating, because the argument for marks is often made purely on accuracy. A coordinate-based agent that misses recovers by taking another screenshot and asking again, so each miss costs a full loop — capture, encode, inference, dispatch — which the previous chapter put at 700 to 1,700 milliseconds and one image's worth of input tokens. Moving from 71% to 96% first-attempt success on a six-step task takes the expected number of model calls from about 8.5 to about 6.3, so the 15 to 40 milliseconds of overlay work pays for itself many times over on latency alone, before counting the tokens.

Density is the other limit. On a page with 34 candidates the badges are unambiguous. On a spreadsheet-like grid with 300 interactive cells the badges overlap, the digits shrink below legibility, and accuracy falls off — well before the numbers get large enough to be interesting as a counting problem. Practical systems cap the set at roughly 30 to 50 and filter to relevance, either by region — mark only what is inside the container the task concerns — or in two passes, where the model first picks a region from a coarsely marked screenshot and then chooses among marks drawn only inside it.

The environments from chapter 1 that lack an accessibility tree get none of this for free. A remote desktop session can still be marked if you run a UI element detector — a model trained to draw boxes around the buttons, fields and menu items in a screenshot, standing in for the enumeration the accessibility tree would have given you — over the screenshot to propose candidate boxes, which restores the discrete choice while leaving you with a box rather than a handle, so the click still goes through coordinates and still needs the crop comparison. That is a real improvement in grounding accuracy and no improvement at all in verifiability, and it is worth being clear with yourself about which of the two problems a given technique is solving.

Everything so far has been about acting on an interface. The other half of what vision models get asked to do is read values off images that nobody will click — charts in a deck, dashboards in a screenshot, a graph in a PDF — and there the failure is not a misplaced click but a number that is simply wrong while looking entirely reasonable.