feat(webxr): XR input recording, replay, and hand/controller path trace

[yanziz-nvidia]

Motivation

During IsaacTeleop sessions it is hard to diagnose whether hand/controller poses are drifting, latency is accumulating, or whether a bug is in the live path vs. the recorded replay path. This PR adds two troubleshooting tools — a rolling path-trace visualization and an XR input recorder — to make these problems visible and reproducible without needing a hardware setup to repro.

Changes

New files

• xrInputRecorder.ts — frame-indexed recorder; monkey-patches XRFrame.getPose, getJointPose, and XRSession.inputSources to capture and replay grip/aim poses, hand joint arrays, and gamepad button/axis state; JSON save/load
• RecorderContext.tsx / RecorderComponent.tsx — React context + null-rendering R3F component that drives recorder.beginFrame() at priority -1001 (before CloudXR SDK at -1000); live frame count forwarded every 30 frames via onFrameRecord
• TraceVisualization.tsx — rolling 500-point path trace (6 px screen-space dots, sizeAttenuation:false for WebXR stereo) for grip and wrist; queries live XR poses in idle, recorder.currentFrame during recording/replay

Modified files

• index.html / CloudXR2DUI.tsx — adds Hand/Controller Trace select (In-XR Comfort group) and Recording Controls select (Troubleshooting group); both default hidden, persisted to localStorage, URL-param overrideable (showTraceInXR, showRecordingControls)
• CloudXRUI.tsx — conditional Recording section (Record/Stop, Play/Stop, Save, Load) in the in-XR overlay, shown only when showRecordingControls is enabled
• App.tsx — wraps AppContent in RecorderProvider; mounts RecorderComponent and TraceVisualization in the R3F canvas
• params.ts / resolve.test.ts — registers showTraceInXR and showRecordingControls URL params

How to try it

Path trace

1. Open the web client settings panel (before CONNECT)
2. Under In-XR Comfort → Hand/Controller Trace, select Show
3. Connect and enter XR — colored dots trail your hands/controllers in real time

Record → Save → Replay

1. Under Troubleshooting → Recording Controls, select Show
2. Connect and enter XR
3. In the XR overlay panel, scroll to the Recording section:
   • Press Record → move controllers/hands → press Stop (frame count shown live)
   • Press Save → downloads isaacteleop-input-recording.json
4. Reload the page, enter XR again, enable Recording Controls
5. Press Load → pick the saved file
6. Press Play → poses and button presses replay exactly

Testing

• Set Hand/Controller Trace → Show, connect → trace dots appear and fade old-to-bright
• Set Hand/Controller Trace → Hide → dots disappear; setting persists after reload
• Enable Recording Controls → Record/Stop/Play/Load/Save appear in XR overlay
• Record session → frame counter increments live (not stuck at 0)
• Save → isaacteleop-input-recording.json downloads
• Reload, Load file, Play → poses and button state replay correctly
• Switch to hand tracking → trace follows wrist joints
• npx tsc --noEmit — no new type errors
• npm test (jest) — all tests pass

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added XR input recording, replay, and downloadable recording controls.
  • Added optional hand and controller trace visualization for troubleshooting and comfort settings.
  • Added performance metrics for rendering, streaming, and latency.
  • Improved teleoperation controls, countdowns, connection handling, and headset status updates.

• Bug Fixes

  • Improved startup capability checks, disconnect recovery, and error messages.
  • Normalized teleoperation links for more reliable session setup.

• UI Improvements

  • Added settings to show or hide recording controls and XR traces.
  • Updated connection status and recording state displays.

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ac13dff1-8dcd-4aed-9815-06dbe76c2e1f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The CloudXR WebXR client adds XR input recording and replay, JSON import/export, live and recorded hand/controller trace visualization, and configurable in-XR controls. App startup, capability checks, session handling, teleoperation messaging, OOB control, metrics reporting, and disconnect behavior are updated. Configuration and HTML settings now persist trace and recording-control options through runtime configuration.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant User
  participant CloudXR3DUI
  participant RecorderContext
  participant XRInputRecorder
  participant RecorderComponent
  User->>CloudXR3DUI: Start recording
  CloudXR3DUI->>RecorderContext: startRecord()
  RecorderContext->>XRInputRecorder: startRecording()
  RecorderComponent->>XRInputRecorder: beginFrame(XRFrame)
  XRInputRecorder-->>RecorderContext: recorded frame count
  User->>CloudXR3DUI: Stop, replay, or save recording
  CloudXR3DUI->>RecorderContext: invoke recording action
  RecorderContext->>XRInputRecorder: stop, replay, or export recording

Loading

sequenceDiagram
  participant AppContent
  participant CloudXRSession
  participant MessageChannel
  participant HeadsetControlChannel
  AppContent->>CloudXRSession: enter immersive session
  AppContent->>MessageChannel: send start or reset command
  MessageChannel-->>AppContent: receive teleoperation data
  CloudXRSession-->>AppContent: report stream status and performance
  AppContent->>HeadsetControlChannel: send status and metrics

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: XR input recording/replay plus hand/controller path tracing.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch yanziz/xr-input-recording

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 8

🧹 Nitpick comments (1)

deps/cloudxr/webxr_client/src/TraceVisualization.tsx (1)

180-254: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Per-frame allocations in the idle live-pose fallback path.

Every XR frame when not recording/replaying, this block allocates livePoses, liveJoints, and a new pose/joint object for every tracked source and joint (up to 25 joints × 2 hands). The file's own doc comment states geometry is pre-allocated specifically to "run at full XR frame rate," but this fallback path directly contradicts that by generating GC pressure at 72-120Hz — a known cause of frame judder in VR.

Consider reusing a small set of persistent scratch objects (mutate in place) instead of allocating fresh objects/arrays each frame.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deps/cloudxr/webxr_client/src/TraceVisualization.tsx` around lines 180 - 254,
Eliminate per-frame allocations in the live fallback within useFrame by adding
persistent, reusable scratch pose objects, joint arrays, and frame containers
outside the callback. Mutate and reset these structures in place for each XR
frame, including clearing absent poses/joints and tracking hasJoints, then
assign the reused container to frame. Avoid creating objects or arrays inside
the input-source and joint loops.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deps/cloudxr/webxr_client/src/App.tsx`:
- Around line 159-171: Wire the JSON load flow end-to-end: destructure
onLoadRecording from useRecorder() in AppContent, pass it into CloudXR3DUI, add
the corresponding onLoadRecording prop to CloudXRUIProps, and render a Load
button in the recording panel that invokes it alongside the existing Save
control.

In `@deps/cloudxr/webxr_client/src/index.html`:
- Around line 1066-1069: Escape the raw greater-than character in the config
guidance text by replacing “>=” with “>=” within the config-text div.

In `@deps/cloudxr/webxr_client/src/RecorderComponent.tsx`:
- Around line 53-57: Update the visibility logic in the recorder frame handling:
since the comparison assigned to isVisible always produces a boolean, remove the
dead ?? true fallback and explicitly default missing sessions to the intended
visibility behavior. Adjust the session visibility expression and the
recorder.beginFrame call so a null session is handled consistently with the
intended default.

In `@deps/cloudxr/webxr_client/src/TraceVisualization.tsx`:
- Around line 172-176: Assign distinct colors to the leftWrist and rightWrist
channels in the trace channel configuration, matching the per-hand color
distinction already used by leftGrip and rightGrip. Update one of the duplicate
"`#ff4422`" values while preserving the makeTraceChannel setup.
- Around line 256-274: Update updateTrace so slot.points.visible is set to false
whenever showTrace is false or frame/pose is unavailable; only append data,
update geometry, and show the points when all required inputs exist, ensuring
stale trace dots are hidden during tracking loss.

In `@deps/cloudxr/webxr_client/src/xrInputRecorder.ts`:
- Around line 333-338: Validate the parsed recording structure in static
importJSON, not just version: ensure frames exists and is an array before
returning the Recording, and throw a descriptive error for malformed input. Keep
this validation inside importJSON so RecorderContext.onLoadRecording can handle
invalid files before the render loop starts.
- Around line 455-466: The replay branch of _onGetPose only handles gripSpace,
so recorded target-ray poses are not replayed. Extend the input-source matching
to recognize src.targetRaySpace and return makeFakePose using
_currentReplay.poses.leftAim or rightAim according to src.handedness, while
preserving the existing grip-space behavior and fallback to _origGetPose.
- Around line 248-315: Gate the recording branch in beginFrame with connected,
matching the existing replay behavior and RecorderComponent.tsx intent. Update
the condition around the recording logic so frames are captured only when
connected; preserve replay handling and ensure disconnected recording does not
append entries or update _currentEntry.

---

Nitpick comments:
In `@deps/cloudxr/webxr_client/src/TraceVisualization.tsx`:
- Around line 180-254: Eliminate per-frame allocations in the live fallback
within useFrame by adding persistent, reusable scratch pose objects, joint
arrays, and frame containers outside the callback. Mutate and reset these
structures in place for each XR frame, including clearing absent poses/joints
and tracking hasJoints, then assign the reused container to frame. Avoid
creating objects or arrays inside the input-source and joint loops.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 9e79ad35-4730-4f66-8dd5-6a7006db1d76

📥 Commits

Reviewing files that changed from the base of the PR and between 8b53bea and 7e5366f.

📒 Files selected for processing (10)

• deps/cloudxr/webxr_client/src/App.tsx
• deps/cloudxr/webxr_client/src/CloudXR2DUI.tsx
• deps/cloudxr/webxr_client/src/CloudXRUI.tsx
• deps/cloudxr/webxr_client/src/RecorderComponent.tsx
• deps/cloudxr/webxr_client/src/RecorderContext.tsx
• deps/cloudxr/webxr_client/src/TraceVisualization.tsx
• deps/cloudxr/webxr_client/src/config/params.ts
• deps/cloudxr/webxr_client/src/config/resolve.test.ts
• deps/cloudxr/webxr_client/src/index.html
• deps/cloudxr/webxr_client/src/xrInputRecorder.ts

[yanziz-nvidia]

try it at https://nvidia.github.io/IsaacTeleop/preview/pr-765/client/

[yanziz-nvidia]

Reviewed by yanziz-reviewer-bot

Summary

Adds an XR input recorder/replayer (monkey-patching XRFrame.prototype.getPose, getJointPose, and XRSession.inputSources) and a rolling 500-point hand/controller path-trace visualization, wired through a new RecorderProvider context and two opt-in 2D-panel dropdowns. All CI checks pass; no new TypeScript errors per npx tsc --noEmit.

Legend: 🚫 Blocker · 💡 Suggestion · 🔍 Nit

	Finding
🚫	None
💡	deps/cloudxr/webxr_client/src/TraceVisualization.tsx:3376 — The BufferGeometry, PointsMaterial, and Points objects produced by makeTraceChannel are never disposed. R3F's <primitive> wrapper does not call .dispose() on unmount; GPU VBOs and shader programs leak until page reload. Add a useEffect that returns a cleanup calling ch.points.geometry.dispose(); (ch.points.material as PointsMaterial).dispose() over all four channels in traceRef.current.
🔍	deps/cloudxr/webxr_client/src/CloudXRUI.tsx:2405 — showTrace is declared in CloudXRUIProps and destructured with a default value but is never referenced anywhere in CloudXR3DUI's render tree. The actual trace toggle is consumed entirely by TraceVisualization. Remove it from the interface and destructuring to keep the prop surface honest. deps/cloudxr/webxr_client/src/xrInputRecorder.ts:7027 — importJSON validates only r.version === 1 and Array.isArray(r.frames). A corrupt JSON file with frames missing the poses or gamepads keys will be accepted silently and throw a TypeError during replay when _currentReplay.poses.leftGrip is read. Add a minimal shape check on at least r.frames[0]?.poses so corruption surfaces at load time with a useful message rather than mid-replay.

Actionables (for bots — copy-paste-ready for AI)

Fix if it makes sense in context — these are agent-generated suggestions, not human-vetted obligations. Skip anything that's wrong, already addressed, or not worth the churn.

• deps/cloudxr/webxr_client/src/TraceVisualization.tsx:3376 — Add a useEffect that returns a cleanup calling .geometry.dispose() and (ch.points.material as PointsMaterial).dispose() over all four traceRef.current channels to prevent GPU buffer leaks on unmount.
• deps/cloudxr/webxr_client/src/CloudXRUI.tsx:2405 — Remove the unused showTrace prop from the CloudXRUIProps interface and from CloudXR3DUI's destructured parameter list; it is never read in the render output.
• deps/cloudxr/webxr_client/src/xrInputRecorder.ts:7027 — After the Array.isArray(r.frames) check in importJSON, add a guard validating r.frames[0]?.poses and r.frames[0]?.gamepads exist so malformed recordings throw an actionable error at load time rather than a cryptic TypeError during replay.

[sgrizan-nv]

These are nice features, thanks!