Linux X11/Xvfb: auto-sample cursor telemetry when ScreenCast METADATA is unavailable #1415

Summary

On some Linux cloud-agent desktops (X11 + Xvfb), the xdg-desktop-portal ScreenCast interface reports empty source/cursor modes (AvailableSourceTypes=0 / no METADATA cursor mode). OpenScreen’s live editable-cursor capture therefore cannot write a .cursor.json sidecar even though the screen can still be recorded and the pointer can be sampled separately.

Agents can still:

1. Record with ffmpeg x11grab (draw_mouse off) or OpenScreen’s non-METADATA path
2. Sample pointer positions + clicks (Electron screen.getCursorScreenPoint(), xdotool / xev / xinput / libinput)
3. Drive openscreen export --auto-zoom from a v2 <video>.mp4.cursor.json sidecar

We implemented an automatic fallback on our fork and would like to contribute it upstream.

Environment (repro host)

Proposed behavior

When ScreenCast METADATA cursor capture is unavailable on X11:

• Automatically sample the pointer during record
• Write <recording>.mp4.cursor.json (v2: timeMs, normalized cx/cy, move|click|mouseup) next to the video
• So openscreen export --auto-zoom needs no manual import

Optional CLI for external ffmpeg tracks: openscreen cursor import --csv … --clicks … --video …

Env knobs: OPENSCREEN_DISABLE_X11_CURSOR=1, OPENSCREEN_FORCE_X11_CURSOR=1

Workaround today (without the PR)

ffmpeg x11grab + CSV/clicks → converter → sidecar → openscreen export --auto-zoom

Ask

1. Accept auto X11 sampling when METADATA is missing (primary path for agent hosts)
2. Document the sidecar / optional CLI import path
3. Confirm preferred schema fields beyond { version: 2, samples: [...] }

Happy to adjust the PR to maintainer preference.

Related

• CLI: openscreen export --auto-zoom, openscreen record --cursor editable-overlay
• Fork implementation: feat(linux): auto-sample X11 cursor telemetry when portal metadata is missing Jasonsu22/openscreen#1