Where a click lands when the coordinate frame is wrong

The grounding suite was green on the Linux CI runner and red on every laptop in the team. Same commit, same browser build, same fixture page. On the runner the agent clicked the Approve button in row 6; on a MacBook it clicked somewhere in row 4, about two hundred pixels to the left. The first three theories were all about the model — a bad seed, an unlucky screenshot compression, a prompt that needed an example — and none of them survived the observation that the misses were not random. Plotting intended against actual for twelve targets produced a straight line through the origin with a slope of 0.82, and a straight line is not what hallucination looks like.

The pattern in that plot is the whole diagnostic method for this class of bug. A model that genuinely cannot find an element returns points scattered around the page with no relationship to the truth. A coordinate frame that has been misinterpreted returns points related to the truth by an affine transform, which is to say a scale and an offset, and both of those are recoverable from a handful of samples.

Six questions about four numbers#

The list [0.12, 0.34, 0.28, 0.41] is four floats and nothing else. Turning it into a screen position requires six decisions, and the number of ways to be wrong is the product of the options rather than their sum.

The first question is what the numbers are measured in. Values between 0 and 1 are almost certainly normalised against the image dimensions, meaning each one is a fraction of the image's width or height rather than a count of pixels. Integers up to 1000 are usually per-mille — thousandths of the width or height — a convention several model families use because it keeps coordinates in a small token vocabulary. Anything larger is pixels — but pixels of which image, which is a separate question below. A model asked for pixels on a 3024-pixel-wide screenshot that returns 0.72 has not given you a pixel coordinate, and code that does int(value) on it produces a click at x=0.

The second is what the four numbers mean as a group. xyxy gives two opposite corners, xywh gives one corner plus a size, and cxcywh gives a centre plus a size. The list above is consistent with xyxy describing a box 0.16 wide and 0.07 tall, and equally consistent with xywh describing a box starting at (0.12, 0.34) that is 0.28 wide and 0.41 tall — a region five times larger. Both parse cleanly; only one is right, and the wrong one puts your click at the centre of a region that mostly contains other things.

The third is the order of the axes. It is easy to assume x comes first, and several widely used models return [ymin, xmin, ymax, xmax] because that is the convention their training data used. The signature of getting this wrong is unmistakable once you look for it, because the click lands mirrored across the diagonal of the viewport, and elements near that diagonal appear to work.

The fourth is the origin. Screens and image files put (0, 0) at the top-left with y growing downward; PDF user space, OpenGL and several plotting libraries put it at the bottom-left with y growing upward. If a box arrives from a pipeline that touched a PDF renderer, this is live. The signature is a click mirrored vertically, so an element 100 pixels from the top gets a click 100 pixels from the bottom.

The fifth is which rectangle the coordinates are relative to. Image space is the screenshot. Viewport space is the visible part of the page. Page space includes the scroll offset, so a viewport coordinate used as a page coordinate is wrong by exactly window.scrollY. Screen space includes the browser chrome — the toolbar, tabs and address bar drawn around the page, which have nothing to do with the Chrome browser — and, on macOS, the menu bar, so an OS-level click driven by a viewport coordinate lands roughly 100 pixels too high. Each of these is a constant offset, which is why they are easy to mistake for each other and easy to distinguish from the sixth.

The sixth is the unit, and this is where the MacBook came in. A screenshot captured on a display with devicePixelRatio 2 has twice the pixels per CSS pixel in each dimension, so a 1280×800 viewport produces a 2560×1600 image. Browser input APIs — page.mouse.click in Playwright, Input.dispatchMouseEvent in the DevTools Protocol — take CSS pixels. Anything that reasons over the image file is speaking device pixels. Nothing in the type system says so, because both are numbers.

The slope of 0.82#

The MacBook run had a slope, and the number is more informative than its existence. Playwright's deviceScaleFactor was inheriting the developer's display setting of 2 while CI ran headless — the browser running with no visible window and no physical display attached, which is how it runs on a server — at 1, so the laptop screenshot arrived as a 2560×1600 PNG. That image is over the long-edge resize cap most vision APIs apply — commonly around 1568 pixels — so the provider silently downscaled it to 1568×980 before the model ever saw it, and the model, quite reasonably, returned coordinates in the image it was shown. The agent then handed those numbers to page.mouse.click, which reads them as CSS pixels in a 1280-wide viewport. The composite factor is 1280 / 1568, which is 0.816.

Working an actual target through it: the Approve button in row 6 sits at CSS (1102, 636). In the 2560×1600 capture it is at (2204, 1272); after the provider's downscale it is at (1349, 779); dispatched as CSS pixels the click lands at (1349, 779) clamped into the viewport, which on this page put it in row 4 about two hundred pixels left of where it belonged. Near the top-left the same factor is nearly invisible — a target at CSS (60, 40) maps to (73, 49), thirteen pixels off, comfortably inside a 40-pixel-tall button — which is precisely why the login step passed on every machine and the table step failed on half of them.

Recovering the transform takes five samples and no model at all, because you can generate ground truth from the DOM. The fit below is a least-squares one, which means it picks the scale and offset that come closest to explaining all five samples at once and reports the residual, the amount by which the fitted line still misses the points it was fitted to:

import numpy as np

# intended: CSS coords from element.bounding_box(); actual: where the click landed
intended = np.array([[60, 40], [420, 210], [900, 400], [1102, 636], [1240, 760]])
actual   = np.array([[73, 49], [514, 257], [1102, 490], [1349, 779], [1518, 931]])

for axis in (0, 1):
    A = np.c_[intended[:, axis], np.ones(len(intended))]
    (scale, offset), residuals, *_ = np.linalg.lstsq(A, actual[:, axis], rcond=None)
    print(f"axis {axis}: scale={scale:.4f} offset={offset:+.1f} "
          f"residual={float(residuals[0]) if residuals.size else 0:.2f}")

# axis 0: scale=1.2250 offset=+0.0 residual=0.00
# axis 1: scale=1.2250 offset=+0.0 residual=0.00

A residual near zero says the relationship is exactly affine, which rules out the model having guessed. A scale of 1.225 in the direction the clicks were transformed is 1568/1280, the inverse of the factor above, and equal scale on both axes points at a uniform resize rather than a viewport mismatch, which would distort the axes differently unless the aspect ratios happened to match. A non-zero offset with a scale of exactly 1 would have sent you looking at scroll and chrome instead.

That fit is also the single experiment that separates the two candidates when a suite passes in CI and fails on developer machines. The two live suspects are a scale factor — devicePixelRatio, provider downscale, or an OS display-scaling setting like Windows at 150% — and a constant offset from the window position, browser chrome or scroll. Probing one target near the top-left corner and one near the bottom-right separates them in a single run: if both are off by the same vector it is an offset, and if the far one is off by several times the near one it is a scale. Pinning the browser version, which is the reflex answer, tests neither, and would not have moved a number in this trace.

Making it fail in CI instead of on someone's laptop#

The reason this bug ships is that CI runs headless at one scale factor and humans run at another, so the environment that gates the merge is the only environment where the code is correct. The fix is to make the scale factor a test matrix dimension and assert the click lands, which requires the fixture to be able to tell you whether it did.

// fixture.html contains: <button id="target" onclick="this.dataset.hit='1'">Approve</button>
for (const deviceScaleFactor of [1, 2, 3]) {
  test(`grounding round-trips at dpr ${deviceScaleFactor}`, async ({ browser }) => {
    const ctx = await browser.newContext({
      viewport: { width: 1280, height: 800 },
      deviceScaleFactor,
    });
    const page = await ctx.newPage();
    await page.goto(FIXTURE);

    const png = await page.screenshot();               // deviceScaleFactor px
    const point = await groundElement(png, 'the Approve button');  // full pipeline
    await page.mouse.click(point.x, point.y);          // CSS px

    await expect(page.locator('#target')).toHaveAttribute('data-hit', '1');
  });
}

Two properties make this worth the minute it takes to run. It exercises the whole transform chain including the provider's resize, which a unit test on your own scaling code cannot, because your code never sees the resize happen. And it fails loudly at deviceScaleFactor 2 and 3 while passing at 1, which localises the defect to the scale path without anyone reading a stack trace. Adding a second target near the bottom-right corner strengthens it further, since a near-origin target can pass under a wrong scale factor by luck.

The reason to stop returning coordinates#

Getting all six conventions right yields an agent that clicks where the model pointed, which leaves the accuracy of the pointing itself. That is the weaker link. ScreenSpot-Pro, a benchmark of grounding tasks on high-resolution professional software, reported single-digit to low-double-digit success rates for strong general vision models on precise targets — small toolbar icons in CAD and IDE interfaces at 4K — because pixel regression from a downscaled image is a genuinely hard task rather than a prompting failure. Coordinates also carry no identity, so a correct point is only correct for as long as the layout holds, which the previous chapter showed is often under a second.

Both problems have the same shape: the model is being asked to produce a continuous value, and it is being asked to produce it in a frame that your code then has to reconstruct. Removing the continuous value from the interface removes both at once, and that is what numbered marks do.