Skip to content

voice: make passive ptt_start the canonical hands-free narration fix#326405

Draft
accnops wants to merge 9 commits into
microsoft:mainfrom
accnops:arthurcnops/voice-narration-handsfree-fix
Draft

voice: make passive ptt_start the canonical hands-free narration fix#326405
accnops wants to merge 9 commits into
microsoft:mainfrom
accnops:arthurcnops/voice-narration-handsfree-fix

Conversation

@accnops

@accnops accnops commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Problem

In hands-free mode the client opens a passive barge-in mic that keeps streaming while it waits for the user to actually speak. Auto-listen does the same. Those ptt_start events made the backend latch the session as "user is speaking" for the whole open-mic window, and the window ends silently (the mic is aborted with no ptt_end), so the latch never cleared. Client-driven narration requested during that window was NACK'd busy and stranded. A few reconnect and unblock paths also lost or failed to replay narrations that were deferred while the session was busy.

#326378 fixed the same symptom from the frontend by force-ending the open turn (synthesizing a ptt_end) before narrating. That works, but it overloads ptt_end to mean "stop for playback" and can cut a live barge-in short.

Solution

Address the root cause instead of forcing an end. The client tags every silent open-mic turn (barge-in and auto-listen) with passive: true on ptt_start; the backend sets user_is_speaking = not passive, so a passive turn never latches. Deliberate presses stay non-passive and still latch, which correctly defers narration while the user is actually speaking.

Because this makes the passive flag the canonical fix, it reverts the #326378 frontend workaround (now on main): _userSpeechActive, _isActivelyDictating(), _finishPtt('immediate'), the _prepareForPlayback(endOpenTurn) parameter, canRequestNarration, and the endPassiveTurnFirst block in _narrate.

Supporting changes on the branch:

  • Adopt the server session id and clear the resuming flag on (re)init.
  • Replay deferred narrations on reconnect.
  • Retry the specific narration id named by narration_unblocked instead of the latest.
  • Client-side telemetry when a narration is deferred or dropped.
  • Preserve a held deliberate press in _prepareForPlayback: a non-passive press latched user_is_speaking, so its narration was deferred; aborting it here would send no ptt_end and strand the latch. Its natural release sends ptt_end, clears the guard, and drives the narration_unblocked retry. Passive open-mic turns (which never latched) are still torn down to free the mic for narration audio. _prepareForPlayback now returns whether the slot is ready.
  • Don't play or double-speak buffered narration over a held deliberate press. _flushDeferredResponse peeks the buffered response, prepares the slot, and only deletes/enqueues it when the slot is actually ready; a preserved press keeps the buffer intact and reports retained plus the buffered reply's transcript. _activateShownSession then skips a fresh narration only for the SAME reply that buffer will replay on release (matched by normalized transcript). An unrelated confirmation or a newer, different response for the session still narrates, because a stale buffered response can outlive its turn (a thinking transition clears _deferredNarrations, not _deferredResponses). The buffer plays on release via the periodic safety-net; the pending indicator stays until then, so nothing is lost.
  • Clear the passive flag when a barge-in listen is promoted to a real press, so a promoted press latches locally and its narration is preserved rather than torn down.

Tests

voiceSessionController.test.ts and voiceClientService.test.ts:

  • Auto-listen opens a passive mic turn; a deliberate press opens a non-passive one.
  • A held deliberate press awaiting narration is preserved (no abort, stays held).
  • A held deliberate press keeps buffered narration deferred instead of playing over the press.
  • A buffered reply retained under a held press is not re-narrated (no duplicate on release).
  • A different pending reply is still narrated when an unrelated buffer is retained under a held press.
  • Promoting a passive barge-in listen to a real press clears the passive flag so playback preserves the press.
  • A passive open-mic turn is torn down for playback since it never latched.
  • Deferred-narration replay on reconnect, unblock-by-id retry, and the deferred/dropped telemetry paths.

Full voiceClient suites pass (controller 19, client service 17, glow 3). ESLint clean; typecheck clean (only the 8 pre-existing native-module errors unrelated to this change).

Risks

  • Depends on the backend understanding passive (yolo#56284). This must reach GA before this lands. If it merges first, the backend ignores passive and, with #326378 reverted, hands-free narration would strand again. Kept as a draft for that reason.
  • Reverts merged frontend code from #326378; anyone relying on the forced-ptt_end behavior would see it change to the passive model.

Non-goals

  • No change to how narration text is phrased or how audio is echoed.
  • No new dependencies, settings, or public API surface.
  • The narrow async cold-mic acquisition race is pre-existing and out of scope; with VAD it self-corrects via the backend speech edge.
  • Making a promoted barge-in press visible on the wire. Clearing the passive flag on promotion is local state only; the turn's ptt_start was already sent as passive: true, so the backend still treats it as passive and does not latch. With VAD, real speech during the press cancels any narration via the speech edge. Without VAD there is no speech edge, so a narration requested during that brief promoted-press window can be accepted and play over the user. A wire-level promotion that latches the backend for the same turn is deferred (it needs a protocol change and carries regression risk on the working deliberate-press path).
  • Clearing a session's pending-reply indicator after the periodic safety-net (rather than a focus) replays its buffered audio. The safety-net flush currently ignores the flush result, so the indicator can linger and a later refocus can re-narrate the reply. This is pre-existing (unchanged by this PR) and tracked as a follow-up.

Copilot AI review requested due to automatic review settings July 17, 2026 20:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Improves voice narration recovery during hands-free listening and reconnects.

Changes:

  • Marks passive barge-in PTT sessions.
  • Restores session and deferred narration state after reconnects.
  • Adds narration defer/drop telemetry and tests.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
voiceClientService.test.ts Tests PTT serialization and session recovery.
common/voiceClientService.ts Extends the PTT API with passive.
voiceTelemetry.ts Defines narration lifecycle telemetry.
voiceSessionController.ts Replays and revalidates deferred narrations.
browser/voiceClientService.ts Serializes passive PTT and adopts server session IDs.
micCaptureService.ts Preserves passive state through microphone acquisition.

Comment thread src/vs/workbench/contrib/chat/test/browser/voiceClient/voiceClientService.test.ts Outdated
Comment thread src/vs/workbench/contrib/chat/browser/voiceClient/voiceClientService.ts Outdated
Comment thread src/vs/workbench/contrib/chat/browser/voiceClient/voiceSessionController.ts Outdated
Comment thread src/vs/workbench/contrib/chat/browser/voiceClient/voiceSessionController.ts Outdated
Comment thread src/vs/workbench/contrib/chat/browser/voiceClient/voiceSessionController.ts Outdated
Comment thread src/vs/workbench/contrib/chat/browser/voiceClient/voiceTelemetry.ts Outdated
@accnops
accnops force-pushed the arthurcnops/voice-narration-handsfree-fix branch from c06bdc7 to fb29d8c Compare July 17, 2026 22:18
@accnops accnops changed the title voice: recover client-driven narration starved in hands-free mode voice: make passive ptt_start the canonical hands-free narration fix Jul 17, 2026
@meganrogge meganrogge added this to the 1.130.0 milestone Jul 18, 2026
@accnops
accnops force-pushed the arthurcnops/voice-narration-handsfree-fix branch 4 times, most recently from a85eb07 to 3a98729 Compare July 18, 2026 08:34
Arthur Cnops and others added 9 commits July 18, 2026 09:39
… user speaking

Thread a `passive` flag immutably through mic-start into the ptt_start
frame instead of reading controller state at emit time, which could
race with the async getUserMedia acquire in the cold-capture path.

- micCaptureService: pttDown(turnId, passive = false) now captures
  passive per-press and fires it on onPttStart (Emitter<boolean>) at
  both the warm and post-acquire fire sites.
- voiceClientService: sendPttStart(turnId, passive = false) serializes
  `passive: true` only when true, keeping the wire frame byte-identical
  for real presses / old clients.
- voiceSessionController: onPttStart forwards the captured passive to
  sendPttStart; _startBargeInListen is the only caller passing
  passive=true (verified single true caller).

Safe because barge-in promotion (a real press during a passive listen)
does not re-send ptt_start, so the passive flag captured at listen-open
time remains correct for the whole turn.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 803dc35b-1aee-4354-85c7-df6076293a49
`_isResuming` was set on reconnect but never cleared, and a failed
resume refused to adopt the server's fresh session_init id, leaving
the client believing it still owned a dead session and stalling
reconnect. On both `session_init` and `session_resumed`, always adopt
`msg.session_id` and clear `_isResuming`; also reset `_isResuming` in
`_cleanup()` for terminal disconnects.

Adopting a fresh id on a failed resume is strictly more correct than
the previous refusal.

Extended the test createTestWindow harness to bind setInterval and
clearInterval to the real window target (native timer methods are
branded and throw Illegal invocation when called with a Proxy as
this), which was needed to drive ws.onopen in the new tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 803dc35b-1aee-4354-85c7-df6076293a49
`_deferredNarrations` (busy/interrupted narrations awaiting a
narration_unblocked nudge) was never replayed on reconnect - only
_pendingNarrationRetries was. The narration_unblocked nudge that would
otherwise trigger a retry is itself lost with the dropped socket, so a
narration deferred right before a disconnect could be silently lost
forever.

On session_init/session_resumed, now also replay _deferredNarrations by
reusing _retryDeferredNarration for each entry - the same revalidation
already used for the live narration_unblocked path: it re-checks
_currentNarratable before firing and reuses the same narration_id when
the text is unchanged, so backend idempotency returns accepted without
re-synth for anything already spoken. Entries no longer warranted or no
longer the shown session are dropped without speaking, same as before.

_retryDeferredNarration now returns whether it actually sent a retry
(mirrors _narrate's return) so the reconnect handler can gate entering
auto-listen the same way it already does for _pendingNarrationRetries -
a narration about to play should not be torn down by a listen that
starts moments later.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 803dc35b-1aee-4354-85c7-df6076293a49
`_retryDeferredNarration` ignored the `narration_id` carried by
`narration_unblocked` and always retried whatever was currently
deferred for the session key. `_deferredNarrations` is latest-wins per
session key, so if a second narration for the same session went busy
before the first one's `narration_unblocked` arrived, the stale nudge
would retry the newer (possibly still-busy) entry early instead of
waiting for its own unblock.

`_retryDeferredNarration` now takes an optional `unblockedNarrationId`;
when it does not match the currently deferred entry's id, the retry is
skipped - the correct nudge for that entry will arrive separately once
its own guard clears. The reconnect replay path (no unblock event, no
id to compare against) is unaffected and keeps retrying whatever is
currently deferred.

Low risk: retrying a superseded id was already an idempotent no-op
server-side, so this only removes an unnecessary/early retry attempt.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 803dc35b-1aee-4354-85c7-df6076293a49
…ent-side

Add voiceNarrationDeferred/voiceNarrationDropped telemetry events
(voiceTelemetry.ts) following the existing publicLog2 pattern used
elsewhere in voiceSessionController.ts (owner meganrogge, SystemMetaData/
FeatureInsight classification). Only `kind` (response|confirmation) and a
`reason` enum are logged - never narration text.

Wired into the four client-side branches where a narration is deferred or
dropped without ever playing:
- _handleNarrationAck: 'invalid' disposition -> dropped (reason: invalid);
  'busy' disposition -> deferred (reason: busy)
- _handleNarrationInterrupted -> deferred (reason: interrupted)
- _retryDeferredNarration: no longer warranted -> dropped (reason: stale);
  session no longer shown -> dropped (reason: session_changed)

No dedicated unit test: voiceSessionController has no unit harness (per
task scope, not building a new one). Verified via typecheck/eslint on the
touched files and the unaffected voiceClientService.test.ts suite (15/15
passing).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 803dc35b-1aee-4354-85c7-df6076293a49
Extend the passive open-mic flag to the auto-listen path (not just barge-in)
so every hands-free turn opened without a deliberate user press sends
ptt_start with passive:true. The backend then leaves user_is_speaking clear
on those silent turns, so a client-driven narration request is accepted
instead of being NACK'd busy: user_speaking and stranded.

This supersedes the force-ptt_end workaround from microsoft#326378: a passive turn
never latches in the first place, so there is no latch to force-clear. Remove
that mechanism (_userSpeechActive, _isActivelyDictating, the _finishPtt
'immediate' variant, _prepareForPlayback(endOpenTurn), and canRequestNarration)
and restore _narrate to unconditionally prepare for playback on a sent request.

Depends on backend passive handling (yolo#56284): must be live in GA before
this lands, or silent hands-free turns latch again on the old backend.

Adds controller tests: auto-listen opens a passive mic turn; a deliberate
press stays non-passive.
…playback

A deliberate (non-passive) ptt_start latches the backend's user_is_speaking,
so its narration request is NACK'd busy and deferred. _prepareForPlayback
previously aborted any held turn via _finishPtt('auto'), which sends no
ptt_end and strands that latch (the later physical release no-ops because
_pttHeld is already false). Track the current turn's passiveness and skip the
abort for a held deliberate press, leaving its natural release to send
ptt_end and clear the guard. Passive open-mic turns (auto-listen / barge-in),
which never latch, are still torn down to free the mic for narration audio.
When a deliberate (non-passive) press is held, its natural release drives the
narration_unblocked retry, so the playback slot must not be torn down under it
(that sends no ptt_end and strands the backend latch). Three changes keep a
held press intact without dropping or duplicating its buffered reply:

- _flushDeferredResponse peeks the slot via _prepareForPlayback() (now returns
  a boolean) and only deletes/enqueues the buffered chunks once the slot is
  actually free; a held deliberate press leaves the buffer intact and reports
  `retained` plus the buffered reply's transcript. Passive open-mic turns
  (which never latched) are still torn down.
- _activateShownSession skips a fresh narration only for the SAME reply the
  retained buffer will replay on release (matched by normalized transcript);
  otherwise it would double up with that buffer. An unrelated confirmation or a
  newer, different response for the session still narrates, because a stale
  buffered response can outlive its turn (thinking clears _deferredNarrations,
  not _deferredResponses). The pending indicator stays until the buffer plays.
- Clear _pttCurrentTurnPassive when a passive barge-in listen is promoted to a
  deliberate press, so playback prep preserves the promoted press.
@accnops
accnops force-pushed the arthurcnops/voice-narration-handsfree-fix branch from 3a98729 to 5bdc624 Compare July 18, 2026 08:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants