Skip to content

[compass] Fix: seat_observe repeats a window's tree while a Chrome drop-down list is open - #29

Merged
skulitom merged 7 commits into
mainfrom
compass/anode-fix-2026-10-08-observe-repeats
Oct 9, 2026
Merged

skulitom merged 7 commits into
mainfrom
compass/anode-fix-2026-10-08-observe-repeats

Conversation

@skulitom

@skulitom skulitom commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #28: merge #28 first. This branch starts from #28's head, because its quick check uses the Hosted helper that #28 adds. Until #28 lands, the diff below also shows #28's commit. After #28 squash-merges, merge main into this branch and the diff shrinks to this fix.

Report

C:\DEV\Compass\ANODE-BUGS.md, B-027 (compass-followup, 2026-10-08, while recording the C-042 web-form guide in the seat). On installed Anode 0.11.0, the followup ran seat_element expand on a Chrome <select> in an --app window. Then seat_observe (maxDepth 16, maxElements 60) returned the window's Pane "Sample sign-up form" nested inside itself at depths 1, 2, 3 and 4. Each copy repeated the caption buttons and the address bar until the element budget ran out ("Tree truncated"). The list's options never appeared. Before the expand, and after the list closed, the same call returned the normal 32-element tree.

Root cause

AccessibilityReader.Observe walks the window breadth-first and trusted the app's tree to have no loops or shared controls. The report shows the window's tree nested inside itself, so with the list open Chrome lists the window, or a pane holding its controls, below itself. The walk followed that loop until the budget ran out, so the rest of the tree, including where the options would be, was never reached. B-014's log shows something similar: Chrome's toolbar and bookmarks listed twice in a normal window.

Not confirmed: public results strip runtime IDs, so the report can't show that Chrome's copies share one. The review judged that they very likely do, about 75% sure. Chromium gives each node a stable [UiaAppendRuntimeId, unique id]. If they don't, the guard still ends the loop after at most one extra copy, since only two windows (the app window and the popup) take part. A seat run settles it; see below.

Fix

The walk reads each child's UIA runtime ID when its parent lists it. A child with an ID already queued is skipped: it isn't queued, described or walked, so it takes no room from real controls. The new skippedRepeats field says how many were skipped, with a line in the text summary.

  • It's a field, not a warning, because nothing was hidden, and seat_wait's missing state needs a tree without warnings.
  • Sibling indexes still count the skipped repeats, so the paths that seat_element follows (first child, then next sibling by index, then a runtime-ID check) still lead to the observed control.
  • A control whose description fails leaves room for a later copy of it.
  • A control whose ID can't be read when queued is read again, and its failure reported, when it is described.
  • A list of children that comes back to a control it already gave ends there. Repeats take no budget, so nothing else would stop it.
  • A null runtime ID no longer throws, in observing or acting.

docs/DESKTOP-TOOLS.md, docs/PROTOCOL.md and the CHANGELOG say this.

How it was tested

  • New quick check desktop repeated elements. It uses a hidden in-process window (never shown) with a UIA fragment tree:

    • the window lists A and D, and A lists B and C;
    • C lists A, its own parent, so its children are A and D again (a loop);
    • D lists a second provider object with B's runtime ID, then E, which says it is its own next sibling.

    It observes with a budget of 7 for 6 controls. It expects A, D, B, C and E once each, no truncation, skippedRepeats: 3, the summary line, no warnings, and a seat_wait-style "missing" match. Then it invokes E through its path [1,1], which counts the skipped repeat before it, and only E must be invoked.

  • Mutation runs (file restored byte for byte after each, then rebuilt):

    • With the guard off, the check fails: A,D,B,C,B,E with truncated: True, the reported symptom in small.
    • With the index counting only queued controls, the check fails: the action reaches the wrong control and is refused as stale.
    • Without the sibling-loop stop (E says it is its own next sibling), the check fails with only A,D and truncated: True, after the 4 s limit.
  • FOCUS test command (scripts\build.ps1 -QuickTest -OutputDirectory artifacts\pkg-build): 97/97 pass at each commit, with [compass] Fix: seat_element set_value reports "performed" on a Chrome drop-down list that ignores it #28's check. Debug selftest --quick: pass.

  • No seat, lease, input, browser or Android command was run, and the installed Anode's anode-control pipe wasn't touched.

  • Second review: a fresh read-only Claude subagent (Codex is at its usage limit until Fri 9 Oct 22:15). First pass, on the first commit: no P1. Two P2s:

    • The "Skipped" warning would have broken seat_wait state=missing.
    • The docs stated Chrome's cause as fact.

    Both are fixed in fed1aea, with its P3s:

    • repeats are caught before queueing;
    • a failed description leaves room for a later copy;
    • null runtime IDs are handled;
    • the stronger test above.

    It also noted that WPF and WinForms build some runtime IDs from 26-bit hash codes, so two distinct controls could rarely share one (about 0.03 to 0.2% chance in a 200 to 500 element walk). That's left as a P3. A re-check of fed1aea found one new P2, fixed in 0acceef. Repeats take no budget, so a provider whose next sibling loops back (X to X) kept the walk going until the 4 s limit and dropped the rest of the tree. A parent's list of children now ends at the first control it gives twice. Its P3s are fixed in the same commit: a re-read ID is checked against those already listed, a failed description frees its ID, and Act compares a null runtime ID as empty. A last look at 0acceef found no new P1/P2. Two P3s are left: two distinct provider objects with one runtime ID under the same parent would end that parent's list early, and the test doesn't cover the dequeue-time re-read or freeing an ID after a failed description.

Still to check in a real seat

  • Follow B-027's steps with C:\DEV\Compass\scratch\compass-followup\c042-media\form\index.html. The tree should list each control once, with skippedRepeats set and no repeated window pane. This confirms or refutes the shared-ID assumption above.
  • Unknown: whether the open list's options (ListItem) now appear in that window's tree, or only in Chrome's separate popup window, whether they are marked offscreen, and whether select on one works through its path.
  • B-014's normal Chrome window: does the toolbar still appear twice? It won't, if the copies share runtime IDs.

🤖 Generated with Claude Code

skulitom and others added 5 commits October 8, 2026 23:24
seat_element set_value on a Chrome <select> returned {"performed":"set_value"}
while the list kept its old choice, so an agent could submit a form with the
wrong option. AccessibilityReader.Act now reads the control back after
SetValue: it waits up to 1.5 s for the value to change (Chrome shows a change
a moment later), fails with a message that says to expand the list and select
the option when the control still reads its old value, and returns what the
control reads when it reformatted the value.

The new quick check "desktop ignored set_value" drives a hidden in-process
ComboBox that applies, ignores, applies late or reformats the value. The
hidden-window plumbing of "desktop unknown control types" moves into a shared
Hosted helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e may have taken (review)

From the second review of B-026's fix:
- The failure quoted up to 200 characters of the control's old value, and the
  pipe server logs every handler error to anode.log, so a field's contents
  could reach the log. The message no longer quotes the control.
- It now says the control may have taken the change and kept its old text (a
  tag field, a value it rewrites, a change still pending), and gives the
  "expand the list" advice only for a ComboBox.
- A COMException, timeout or InvalidOperationException while reading back,
  after the value was sent, now answers "performed" with a note instead of a
  failure that reads as "nothing happened".
- The text summary shows what a reformatting control now reads (sanitised),
  and PROTOCOL.md lists the optional `value`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ck catch and a text field (review)

From the re-check of b884144: the text summary quotes what a reformatting
control reads, or says "(empty)", so app text isn't read as part of Anode's
note. The quick check now also covers a control that goes away after the
value is sent (performed, "could not be read back") and a text field that
ignores the value (no drop-down advice).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t is open (B-027)

With a <select> list open, Chrome reported the window's Pane again below
itself, and AccessibilityReader.Observe's breadth-first walk followed the copy
over and over (depths 1, 2, 3, 4...) until the element budget ran out, so the
list's options never appeared. The walk now remembers each element's UIA
runtime ID and skips one it has already described, without walking it again or
counting it against maxElements, and a warning says how many repeats were
skipped. Sibling indexes still count the skipped repeats, so the paths that
seat_element follows stay valid.

The new quick check "desktop repeated elements" builds a hidden in-process
fragment tree with a loop (C lists its ancestor A) and a control under two
parents (B under A and D), then acts on C through its observed path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…efore queueing (review)

From the second review of B-027's fix:
- The "Skipped N controls" warning made seat_wait's "missing" state, which
  needs a tree without warnings, time out on any window with a repeat. The
  count is now `skippedRepeats` and a summary line, not a warning.
- A repeat is caught when its parent lists it, so it no longer takes a queue
  slot from real controls; a control whose description fails leaves room for
  a later copy; a null runtime ID no longer throws.
- The CHANGELOG and DESKTOP-TOOLS.md describe what Anode does, not Chrome's
  internals, which still need a seat to confirm; PROTOCOL.md lists the field.
- The check now has a second provider object with B's runtime ID before a
  unique control E, acts on E through a path that counts the skipped repeat,
  uses a budget one above the unique count, and asserts that a wait for a
  missing control still matches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

skulitom and others added 2 commits October 8, 2026 23:47
…ad IDs unique (review)

From the re-check of fed1aea: since a skipped repeat takes no budget, a
provider whose next sibling comes back to a control it already gave (X -> X,
or X -> Y -> X) kept the walk going until the 4 s limit, then dropped the
rest of the tree. A parent's list of children now ends at the first control
it gives twice. Also: an ID read only when a control is described is
checked against the ones already listed; a failed description frees that ID
either way; and Act compares a null runtime ID as empty instead of throwing.
The check's E now says it is its own next sibling; without the stop, the
check fails with only A and D listed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…landed as a squash

Resolved with #28's head (1f7d5a1) as the merge base, so the diff against main is still
exactly 1f7d5a1..0acceef. The tree is unchanged from 0acceef.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@skulitom

skulitom commented Oct 9, 2026

Copy link
Copy Markdown
Owner Author

Compass merge: ready to merge, merging now

#28 landed as a squash (72ac2a8), so this branch no longer merged cleanly. I merged the new main into it with #28's head as the merge base, the stacked-PR method, and pushed that ordinary merge commit, 06b7f3a (no force-push). The diff against main is unchanged: it is still exactly 1f7d5a1..0acceef, and the tree is the same as 0acceef's.

Tested by compass-merge on 2026-10-09, on that same tree, so this is the code that lands:

Review (whole diff):

  • Repeats are found by UIA runtime ID. They are skipped at queue time and take no budget.
  • A sibling list that loops back stops the walk at that point, so the 4 s limit is a backstop, not the only stop.
  • Indexes still count the skipped copies, so the paths seat_element follows stay right. The check proves this by invoking E after a repeat.
  • skippedRepeats is a field, not a warning, so seat_wait "missing" still matches.
  • The read-failure path un-marks the ID, so a later copy can still be described.
  • A provider that gives two different controls the same runtime ID would now show only one of them. It is counted in skippedRepeats, and seat_element already treats the runtime ID as identity, so I'm not blocking on it.
  • Not blockers: [compass] Anode after the launch: be where agents look, and show what it is for #27's web-form guide (not merged yet) says the drop-down behaviour is "not yet fixed", and my comment there asks for that line to change. Whether an open list's options appear in a real Chrome seat is still to be checked, as the PR says.

Second opinion: the fixer's fresh Claude subagent reviews (three passes; the last found no P1 or P2). The Codex CLI is at its usage limit until 22:15 today, and Codex's GitHub review hit the same limit here.

Undo: git revert of the squash commit.

@skulitom
skulitom merged commit a504b9a into main Oct 9, 2026
1 check passed
@skulitom
skulitom deleted the compass/anode-fix-2026-10-08-observe-repeats branch October 9, 2026 00:32
skulitom added a commit that referenced this pull request Oct 9, 2026
…down notes

compass-merge's review of #27 (2026-10-09):
- GUIDE-DESKTOP-APP: build before `lease acquire`, so a long build can't
  expire the 120 s lease; mention `--ttl 600`; scale screenshot points back
  to seat pixels before `seat_click`.
- GUIDE-ANDROID, ANDROID, TROUBLESHOOTING: `anode android` needs Anode
  running; only `--json` shows `booted`.
- GUIDE-WEB-FORM: sign in before the agent takes the lease; take the next
  element ID from `seat_wait`'s own reply; scale screenshot points back;
  #28 and #29 fix the `<select>` behaviour in the next release; one
  `seat_type` call types at most about 3,000 characters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant