Skip to content

feat(linux): auto-sample X11 cursor telemetry when portal metadata is missing - #1

Open
Jasonsu22 wants to merge 2 commits into
mainfrom
cursor/x11-cursor-telemetry-fallback-19b6
Open

Jasonsu22 wants to merge 2 commits into
mainfrom
cursor/x11-cursor-telemetry-fallback-19b6

Conversation

@Jasonsu22

@Jasonsu22 Jasonsu22 commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner

Summary

On X11 + Xvfb hosts the ScreenCast portal often has no METADATA cursor mode, so editable-cursor takes used to write no sidecar. This PR samples the pointer automatically during record (Electron screen.getCursorScreenPoint(), then xdotool / xinput / xev / libinput) and writes version-2 <video>.cursor.json next to the take. openscreen export --auto-zoom then works with no extra import step.

Also adds openscreen cursor import for an external ffmpeg/x11grab CSV track.

Follow-up on this branch addresses CodeRabbit’s review of the first revision: discard stops the X11 sampler, fallback samples are compacted by pause ranges, the sampler clock starts at capture start, cursor import CSV/write guards are stricter, XI2 rest no longer accumulates, click watchers fail over on child exit, and the browser fallback keeps the source it just picked.

Related issue

Fixes getopenscreen#1415

Type of change

  • Bug fix
  • Feature
  • Enhancement
  • Documentation
  • Refactor / maintenance
  • Performance
  • Security

Release impact

  • Patch
  • Minor
  • Major / breaking change
  • No release note needed

Desktop impact

  • Windows
  • macOS
  • Linux
  • Installer / packaging
  • Not platform-specific

Repro host (from getopenscreen#1415)

Item Value
Host Grok Bot cloud Linux (grok-bot-vm-316768464)
OS Debian 13 (trixie), kernel 6.12.94+ x86_64
Session X11 DISPLAY=:2, Xvfb 1920×1200×24
Portal ScreenCast METADATA unavailable (AvailableSourceTypes=0)
GPU Mesa llvmpipe (CPU export fallback)
OpenScreen tested Linux AppImage 2.1.2

Behavior

  • Auto path: X11 (incl. Xvfb) — PipeWire fail → Electron/xdotool/xinput/xev/libinput; native Linux capture with empty cursor track fills sidecar via parallel X11 sampler
  • CLI: openscreen cursor import --csv mouse.csv --clicks clicks.txt --video take.mp4
  • Env: OPENSCREEN_DISABLE_X11_CURSOR=1 / OPENSCREEN_FORCE_X11_CURSOR=1

Testing

  • Unit tests for sampler / sidecar import / CLI parser / factory (vitest)
  • tsc --noEmit and tsc -p tsconfig.test.json --noEmit
  • Targeted vitest after the CodeRabbit follow-up
  • Maintainer smoke on a normal Linux desktop (should be no-op when METADATA works)
  • Optional: X11/Xvfb agent host record → sidecar present → export --auto-zoom

Notes

Upstream PR (same branch): getopenscreen#1416

Open in Web Open in Cursor 

… missing

On X11 (including Xvfb cloud agents) the ScreenCast portal often has no
METADATA cursor mode, so editable-cursor takes used to write no sidecar.
Sample the pointer via Electron/xdotool/xev during record, write the
version-2 <video>.cursor.json next to the take, and add
`openscreen cursor import` for an external CSV. export --auto-zoom then
works with no extra step.
Stop the X11 sampler on discard, compact pause ranges into the fallback
sidecar, start the sampler clock at capture start, harden cursor import
CSV parsing/writes, trim XI2 rest, fail over click watchers on exit,
and pass the browser-fallback source through instead of re-reading
getSelectedSource().
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Linux X11/Xvfb: auto-sample cursor telemetry when ScreenCast METADATA is unavailable

2 participants