Playwright/Puppeteer backend

[apireno]

Alternative to the Chrome extension for headless agent workflows.

Use Playwright or Puppeteer as the browser automation backend instead of the Chrome debugger API.

From roadmap: Platform

[m13v]

we went through this exact decision building our desktop automation tool. started with Chrome DevTools Protocol but switched to a hybrid approach - Playwright for browser stuff, plus native accessibility APIs (AXUIElement on macOS) for everything outside the browser. the big win was being able to automate native app UI alongside web content in the same workflow. one thing worth considering early is how you handle element identification across both backends, we ended up building a unified element abstraction in Rust that normalizes selectors regardless of whether the target is a DOM node or a native UI element.

[m13v]

here's the element abstraction we built for this - it wraps both native UI elements and browser nodes into the same interface so the agent code doesn't care which backend is active: https://github.com/mediar-ai/terminator/blob/main/crates/terminator/src/element.rs

[apireno]

@m13v This is really helpful — thanks for sharing the Terminator element abstraction. The trait-based approach (UIElementImpl → UIElement wrapper) is clean, and the key insight about normalizing selectors across backends early is well taken.

DOMShell's current architecture sidesteps some of this by operating at the Accessibility Tree level rather than DOM nodes directly — the AX tree already abstracts away the underlying HTML into semantic roles and a navigable hierarchy. But if we add a Playwright backend alongside the Chrome Extension backend, we'd need exactly this kind of unified element layer to keep the filesystem commands (ls, cd, grep, click) working identically regardless of which backend is driving the browser.

Question: in Terminator's hybrid model, did you find meaningful differences in the AX tree quality between CDP (Chrome DevTools Protocol) and Playwright's accessibility API? Curious whether the tree structure varies enough to cause issues in the normalization layer, or if it's mostly consistent.

We just published a HARNESS.md documenting how to integrate DOMShell as a backend for other frameworks — the Playwright backend would follow the same pattern. Would be great to compare notes as we build this out.

[m13v]

on the CDP vs Playwright AX tree quality - they're mostly consistent for standard web content, both give you the same semantic roles and ARIA attributes. the differences show up in edge cases: CDP exposes some Chrome-internal nodes (like Shadow DOM boundaries) that Playwright's accessibility API abstracts away. for DOMShell's filesystem metaphor that's probably fine since you want the semantic view anyway.

the bigger difference is in dynamic content. CDP gives you raw tree updates in real-time via accessibility events, while Playwright snapshots the tree at query time. if your filesystem commands need to reflect live DOM changes (like an SPA routing), CDP's event stream is more reliable.

HARNESS.md looks clean. the backend abstraction pattern would map well to adding a native macOS accessibility backend too - the AX tree on macOS follows similar role/hierarchy patterns, just with different API surface.

[apireno]

@m13v Really useful breakdown on the CDP vs Playwright AX differences — the Shadow DOM boundary point is exactly the kind of thing that would bite us later if we didn't account for it early.

The real-time events vs snapshot-at-query distinction is key for DOMShell. We currently use CDP's event stream and it's what enables the auto-refresh behavior — the tree invalidates on DOM mutations and rebuilds on the agent's next command. If we add a Playwright backend, we'd need to figure out the right polling strategy to approximate that, or accept slightly stale trees between commands. Probably fine for most workflows but worth documenting as a known tradeoff in the harness.

The macOS native AX backend is an interesting direction. You're right that the role/hierarchy patterns are similar enough that the filesystem abstraction would carry over. The HARNESS.md was written with exactly that kind of extensibility in mind — swap the backend, keep the ls/cd/grep surface. If you ever want to experiment with mapping Terminator's AX traversal to filesystem commands, happy to compare notes.

[m13v]

Yeah the real-time events vs snapshot trade-off is the crux of it. CDP's event stream gives you that instant invalidation, which is great for interactive agents. Playwright snapshots are simpler but you lose that reactivity.

For the hybrid approach, what worked for us: use CDP events as the invalidation signal, but Playwright's snapshot as the actual tree source. That way you get the best of both - you know when the tree changed (CDP mutation events) but you read the tree through Playwright's cleaner API that handles Shadow DOM boundaries correctly. The polling fallback is still worth having for edge cases where CDP events get swallowed (iframes, service workers).