e2e: kitty graphics + pixel mouse verification (three tiers) - #8
e2e: kitty graphics + pixel mouse verification (three tiers)#8pattobrien wants to merge 9 commits into
Conversation
Upgrade the xterm.js backend to a vendored xterm.js master build (vendor/xterm-master: headless bundle + kitty graphics MVP handler from addons/addon-image, built at 904ae93 — APC parser hooks and the kitty storage are unreleased on npm). The backend now tracks DEC mouse modes, answers XTWINOPS 14t/16t with synthetic 8x16 cell pixels (herdr's own fallback), encodes SGR 1006 and SGR-pixels 1016 clicks, and exposes the kitty image storage. New herdr-graphics-xterm tests cover 1016 pixel reports, pane kitty transmits landing in the emulator storage through herdr's host re-encode, and pixel-coordinate clicks moving pane focus. Claude-Session: https://claude.ai/code/session_01SXWJVYuEcms1C41eHKiM71
Two herdr behavior tests driven through @termless/kitty (kitty's real VT parser headless via 'kitty +runpy' batch replay — no OS windows, so the file is untagged): - a pane-emitted kitty-graphics image (APC G a=T, f=32, 1x1 RGBA) is re-emitted by herdr to the host as a rewritten-id q=2 transmit plus a 1x1-cell placement - host mouse contract (1006 enabled, 1016 reset, never enabled) and the pane-side SGR-pixels path: a cell click focuses the unfocused pane and a pane that enabled 1016 receives cell-center pixel coordinates The headless kitty Screen answers CSI 16t with its 10x20px cell size, which is what unlocks both herdr pipelines; the batch-replay bridge only flushes host replies on snapshot queries, so polls touch text(). Claude-Session: https://claude.ai/code/session_01SXWJVYuEcms1C41eHKiM71
Verify kitty graphics actually render: the herdr client's PTY byte stream is mirrored into a full browser xterm.js Terminal + complete ImageAddon inside headless Chromium (playwright-core, system Chrome channel), and the test counts saturated-red pixels on the addon's compositing canvas where a pane's 32x32 red RGBA transmit must land (herdr re-encodes it a=t o=z + a=p; the addon inflates and composites it). Node keeps the control plane — input, CSI 16t answering, SDK asserts — so the browser page stays a passive replay sink. Vendor gains the two browser IIFE bundles + xterm.css, the committed esbuild entries, and build.sh pinning the upstream commit for reproducible regeneration. Claude-Session: https://claude.ai/code/session_01SXWJVYuEcms1C41eHKiM71
Real-kitty companion to the protocol-level graphics tests: herdr runs in an actual kitty OS window and the test asserts red pixels rasterize on screen for a pane-emitted kitty-graphics PNG. The window is an os-panel fixture (src/kitty-panel.ts) launched via kitty remote control with layer=top and focus-policy=not-allowed — occluded kitty windows freeze their screencapture frames, so the panel must stay top-layer and is never focused or raised. Control flows through the herdr SDK socket and kitty get-text; capture via screencapture -l on the panel's platform window id, decoded with pngjs. Assertions are channel-dominance based (tolerant of color management): zero red before, and after display a dense contiguous red block roughly the image's size with nothing red elsewhere. Tagged e2e-kitty; excluded from the default run. Claude-Session: https://claude.ai/code/session_01SXWJVYuEcms1C41eHKiM71
…ests Consolidation pass over the three graphics test tiers: - Fixture-managed resources everywhere: seam-wrapped and backend-specific herdr sessions, the chromium browser page, and the kitty os-panel + its herdr server are all acquired via fixtures with onCleanup, which runs on test timeout — timed-out tests no longer leak running herdr servers (previously in-body try/finally was skipped on timeout). The shared herdr fixture gains a herdrBackend parameter fixture (test.override per file); herdr.ts exports the socket-wait and stop/delete helpers so the panel fixture shares them. - Real test image instead of synthetic solid-red squares: committed tests/e2e/data/graphics/test-icon.png (Chrome app icon at 64px — real multi-hue imagery with genuine alpha; the kitty icon is single-hue). Assertions are image-aware: multi-hue-cluster density in the image region, and blend-through at verified-transparent points (composited output equals the themed pane background — the asset stores black under its transparency, so dropped alpha rasterizes black and fails). The 16KB RGBA payload crosses herdr's compression threshold, so the re-emit exercises chunking + o=z zlib on every run (asserted). - Protocol-seam tier now reconstructs the image from the captured transmission (chunk concat, o=z inflate, f= interpretation) and asserts a lossless RGBA round-trip against the committed asset; the seam records payload bytes to make that possible. - Every tier writes per-run visual artifacts to test-results/graphics/ (reconstructed PNG + screen SVG, chromium full-terminal element screenshot themed from kitty.conf background, kitty window capture) in addition to the report annotations. - xterm.ts merge resolution: the vendored-master rewrite keeps the protocol-seam branch's onOutput launch-option tap, folded into the backend's single pty.onData handler alongside mode tracking and raw-stream capture. Claude-Session: https://claude.ai/code/session_01SXWJVYuEcms1C41eHKiM71
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 740c76c323
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| x: Math.floor((originCol - 1) * cell.width), | ||
| y: Math.floor((originRow - 1) * cell.height), |
There was a problem hiding this comment.
When a pane starts at (0, 0), originCol and originRow already identify its zero-based content cell after adding the border inset; subtracting one here shifts the sampled rectangle a full cell up and left. The color-density check may pass from the remaining overlap, but the right and bottom transparency samples then land on opaque image pixels, making this default Chromium test fail or validate unrelated pixels. Multiply the origins directly by the cell dimensions.
Useful? React with 👍 / 👎.
| cp "$HERE/entries/BrowserKitty.ts" "$CLONE/addons/addon-image/src/BrowserKitty.ts" | ||
|
|
||
| cd "$CLONE" | ||
| npm install |
There was a problem hiding this comment.
When a developer regenerates these committed bundles, the script invokes npm install and later executes ./node_modules/.bin/esbuild directly, bypassing the repository's required package-management and executable wrappers. Use vp install and the corresponding vp exec invocation so the rebuild follows the project's unified tool lifecycle.
AGENTS.md reference: AGENTS.md:L76-L90
Useful? React with 👍 / 👎.
|
|
||
| const IMAGE = loadTestImage(); | ||
|
|
||
| const test = base.override("herdrBackend", () => createTermlessBackend("kitty")); |
There was a problem hiding this comment.
On hosts without the optional system kitty installation, bare vp test now includes this untagged file and unconditionally launches the kitty +runpy backend, so fixture setup fails before the tests run. The configuration excludes only e2e-kitty-tagged tests, while the suite documentation describes this backend as optional; tag or otherwise exclude this system-kitty test from the default headless tier.
Useful? React with 👍 / 👎.
1 participant
Adds kitty-graphics and pixel-mouse coverage to the emulator-backed e2e suite as three complementary tiers, consolidated from spike branches
patt/e2e-gfx-{protocol-seam,xterm-master,termless-kitty}(kept as reference).Tiers
CSI 16t; asserts a lossless RGBA round-trip by reconstructing the transmitted image (chunked,o=zzlib) and comparing to the source asset.904ae93, reproducible viavendor/xterm-master/build.sh) + full ImageAddon in system Chrome via playwright-core; asserts rendered pixels and alpha blend-through against the kitty theme background (#1E1E2E).e2e-kittytag, opt-in): os-panel window (layer=top, focus-policy=not-allowed, never focused), screencapture by window id; asserts kitty's actual rasterization + blend-through.Also
data/graphics/test-icon.png), not a synthetic square.onCleanup— timed-out tests no longer leak herdr servers (verified by timeout simulation; previously 21 leaked sessions from spike runs).CSI 16tis the only host query, graphics gated onexperimental.kitty_graphics+ cell size, re-emit rewrites ids withq=2.test-results/graphics/(kitty capture, chromium frame, reconstructed PNG + screen SVG).https://claude.ai/code/session_01SXWJVYuEcms1C41eHKiM71