Incorrect click location

[JGKle]

Whenever I ask it to click on a particular point, it misses - it clicks below and to the right. Below is a screenshot: I visited excalidraw, drew the black dot, changed the color to red, told it to click the black dot. Then I changed it to blue and told it to try it again. As you can see, red and blue are shifted a little to the right and down significantly.

[Jeomon]

Windows-mcp is operating based on the accessibility tree data
what you shown here is a limitation of the MCP server
because the dots has no information about it in the accessibility tree
and whatever elements lies inside a canvas can't be detected by the accessibility tree

[JGKle]

Windows-mcp is operating based on the accessibility tree data

Hmm...so that's what it tried at first, but it said "the clicks didn’t register. I’m not seeing any interactive elements from the accessibility snapshot, so the automation can’t “see” the hamburger/menu as a target. That usually means the app UI isn’t exposing controls to
the automation layer." I'm trying to use it to automate testing of a .net application.

But even so, shouldn't it be able to take a screenshot, find the location of a pixel on the screenshot, and then click that same coordinate? Why would it be "shifted"? Like if it can locate the x,y on the image, shouldn't it be able to just simulate a click at that same x,y?

[Jeomon]

[Jeomon]

Hmm...so that's what it tried at first, but it said "the clicks didn’t register. I’m not seeing any interactive elements from the accessibility snapshot, so the automation can’t “see” the hamburger/menu as a target. That usually means the app UI isn’t exposing controls to the automation layer." I'm trying to use it to automate testing of a .net application.

But even so, shouldn't it be able to take a screenshot, find the location of a pixel on the screenshot, and then click that same coordinate? Why would it be "shifted"? Like if it can locate the x,y on the image, shouldn't it be able to just simulate a click at that same x,y?

When you develop the .NET application, make the application developed accessibility friendly, then you can test it.

Which model you have tried

[JGKle]

Which model you have tried

Codex on gpt-5.2-codex, high reasoning

When you develop the .NET application, make the application developed accessibility friendly, then you can test it.

Got it. The trick was to assign AutomationProperties.Name in axaml like: <Button AutomationProperties.Name="Open" ...... />, or in code: AutomationProperties.SetName(x, "Open"). Then it can find them.

Still, given that it can take screenshots & understand what it sees, why wouldn't it be able to say "I see a dot at pixel x,y, so now I'll click at x,y"? Strange that it ends up shifted. If there's some scaling going on (i.e. screenshot dimensions isn't an exact representation of input/screen dimensions), maybe all it needs is context on how to map them/translate between them. It's clicking reasonably close to the dot, so clearly it understands what to do. And at a given resolution, it always misses by the exact same amount, so there just seems to be some sort of shifting/scaling between the screenshot coordinate system & the input coordinate system.

[Jeomon]

The accessibility tree coordinates are based on logical screen, while the screenshot we have to move to physical coordinates

[JGKle]

Right - so presumably if the LLM were given info on how to map logical to physical coordinates, it could easily calculate how to translate between them & thus where to click

[Jeomon]

yes but that i have already done already but still let me check