Add voice overlay: dictate into any app from a floating button - #68
Open
hagope wants to merge 12 commits into
Open
Add voice overlay: dictate into any app from a floating button#68hagope wants to merge 12 commits into
hagope wants to merge 12 commits into
Conversation
A draggable mic button drawn over other apps. Tap it and it becomes stop/cancel: stop types the transcript into whatever text field has input focus, cancel discards it. Reaching another app's text field requires an accessibility service — an overlay alone cannot. The overlay window is deliberately non-focusable so the target field keeps input focus while the buttons are tapped. Text is spliced in at the cursor with ACTION_SET_TEXT, falling back to clipboard paste for fields that reject it (some web views), and to the clipboard entirely when no editable field is focused, so a dictation is never silently lost. Engine state has to be flushed between dictations. speech-core's VoicePipeline never clears pending_utterances_ on stop()/start(), and the turn detector holds mid-utterance audio until VAD sees end-of-speech. Since the overlay stops the mic mid-sentence and reuses one long-lived pipeline, that leftover audio surfaced inside the *next* dictation. Each session now pushes silence to close the open utterance (which also captures the sentence tail) and drains the engine before going idle. A session gate drops results arriving outside their dictation. An explicit reset() upstream would let the drain go away. - app/.../overlay/OverlayBubbleService.kt — bubble, mic, session state - app/.../overlay/DictationAccessibilityService.kt — text insertion - app/.../overlay/TextInsertion.kt — cursor/selection splicing, tested - app/.../overlay/VoiceOverlayActivity.kt — permission setup screen Documented in README.md and all nine translations.
The wait after Stop returned as soon as speechActive and partialText were both clear — but both are clear in the window between SpeechEnded and TranscriptionCompleted, so it could return before the result existed, commit an empty transcript, and report "Nothing heard". Short phrases that finish before the first partial is emitted fall in the same hole. Flushing silence to close the utterance made this fire far more often: it makes SpeechEnded arrive promptly and reliably, which is exactly the moment the old condition bailed out. Wait for the engine to fall quiet instead. Every event it emits pushes the quiet window back, so the wait cannot return mid-flight, and a genuine no-speech case still resolves after the settle window rather than hanging to the timeout.
Trying the overlay meant leaving the app for a third-party one, which is a poor first run: if nothing appears, it is unclear whether the overlay, the accessibility grant, or that particular app is at fault. The setup screen now carries a plain EditText to dictate into. Being an ordinary text field is the point — it exercises the same focused-field path as any other app, so a successful dictation here confirms both the accessibility service and the non-focusable overlay window. Showing the overlay no longer sends the app to the background, since the field to test against is on this screen; Home still dismisses it. The screen scrolls now that the keyboard covers part of it.
The instructions already sit above the box, so the in-field hint was repeating them inside the thing it was describing.
An empty text field often reports its placeholder through the node's text — Telegram's composer returns "Message" — so dictation was being appended to the placeholder and every message started with "Message". Hint detection uses the node's own isShowingHintText, falling back to comparing text against hintText for apps that populate the hint but never set the flag. Selection offsets describe the hint in that state, so they are ignored once it is treated as empty.
…e node isShowingHintText and hintText cover standard EditTexts but not custom editors: Telegram's composer still prefixed every message with "Message". Placeholder detection now also treats text matching contentDescription as empty when the caret is unset, which is the shape a custom editor reports for an empty field. Since this has to be inferred per app, the service now records what the targeted node actually reported — class, package, text, hintText, isShowingHintText, contentDescription, selection, and what was treated as existing text — and the setup screen can show it. Guessing twice was one time too many.
Two attempts at detecting placeholders both failed on Telegram, whose composer still prefixed every message with "Message". The approach was the problem, not the heuristics: ACTION_SET_TEXT has to rebuild the field's entire value, so it must be told what is already there, and an app whose empty field reports a placeholder as its text will always poison that read. ACTION_PASTE lets the app insert at its own cursor. It never has to be told the current contents, so an empty field stays empty underneath and no placeholder can reach the result. Paste is now primary with SET_TEXT as the fallback for fields that reject it, inverting the previous order. The user's clipboard is saved and restored around the paste, after a delay — restoring synchronously races the target app reading the clip. Trade-off: the app controls placement, so the joining-space logic no longer applies when appending to existing text. A missing space is a smaller problem than a corrupted prefix, and the SET_TEXT fallback still applies spacing where it runs.
It sits under the thumb for most right-handed use, and keeps clear of the left-edge back gesture. The initial position starts at the right edge rather than sliding there on first layout.
Insertion goes through the clipboard, so the dictated text was left sitting there whenever the user had nothing on it beforehand — as readable to any other app as anything else they had copied. It is now cleared after the paste, or the user's previous clip is restored if there was one. The no-field-focused fallback still leaves the text on the clipboard deliberately, since pasting it manually is the entire point of that path.
Every clipboard write raises the system's "copied" chip, and routing all insertion through paste meant one on every dictation. Paste is only needed when an empty field might be reporting a placeholder as its text. When the node says it is empty, or declares a hint, the contents are known and ACTION_SET_TEXT can write directly — no clipboard, no chip. Fields whose text no placeholder signal explains still go through paste, keeping the Telegram fix intact. The clip is also marked sensitive so Android 13+ cannot preview the dictation in the chip on the paths that still use it.
Solid green and red circles shouted over whatever app was underneath. The mark now carries the colour on the same dark circle as the idle bubble, so the overlay stays quiet while still reading as stop and cancel at a glance. Glyphs go up 18sp to 20sp to hold their weight without the filled background behind them.
speech-core force-splits any utterance past 15 s (agent_config.h max_utterance_duration, not overridden here), so a long dictation arrives as several utterances queued behind each other. The wait after Stop could not survive that. Silence cannot distinguish "finished" from "busy": transcribing a segment emits nothing for its whole duration, so a wait keyed only on quiet gave up mid-decode, committed what it had, and closed the session — after which the real result was dropped as belonging to a finished dictation. The longer the audio, the more reliably it lost the tail. The wait now consults the pipeline's own state, and the timeout scales with how long was recorded rather than sitting at a flat 4 s, capped at 30 s. The drain uses the same busy check.
Member
|
@hagope Is this PR ready for review as a merge candidate? At a high level, I think a few design points may still need another pass:
Are these planned follow-ups, or should we review the current shape as final? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a voice overlay to the demo app: a draggable mic button drawn over other apps. Tap it and it becomes ■ stop / ✕ cancel — stop types the transcript into whatever text field currently has input focus, cancel discards it.
Reaching another app's text field requires an accessibility service; an overlay alone cannot. Three grants are needed (microphone, display-over-other-apps, accessibility), each behind its own system screen, and
VoiceOverlayActivityshows which are still missing. That screen also carries a scratch text field, so the overlay can be tried without leaving the app — useful on first run, where a failure otherwise gives no clue whether the overlay, the grant, or the target app is at fault.Manually tested on a Samsung S26 (Android 16), dictating into third-party apps.
Insertion: two paths, chosen per field
Worth review attention, because the obvious approach is the wrong one in a way that is not obvious.
ACTION_SET_TEXThas to rebuild the field's entire value, so it must be told what is already there. Apps whose empty field reports a placeholder as itstextpoison that read — Telegram's composer returns"Message", so every dictation arrived as"Message <what you said>". Two rounds of placeholder detection (isShowingHintText, thenhintText/contentDescriptionmatching) both failed against it, because nothing the node exposed distinguished placeholder from content.ACTION_PASTElets the app insert at its own cursor and is never told the contents, so a placeholder structurally cannot reach the result. But every clipboard write raises the system's "copied" chip, which is noise on every single dictation.So the method is chosen per field:
ACTION_SET_TEXTwrites directly, with cursor-aware splicing. No clipboard, no chip.ACTION_PASTE, which cannot be poisoned.SET_TEXTremains the fallback if a field rejects paste.On the paste path the clip is marked sensitive so Android 13+ cannot preview the dictation, and afterwards the user's previous clip is restored, or the clipboard cleared if there was none — dictation should not linger there any more than anything else typed.
Trade-off: on the paste path the app controls placement, so the joining-space logic does not apply. A missing space is a smaller problem than a corrupted prefix.
Other design notes
findFocus(FOCUS_INPUT)on the service,rootInActiveWindow, and each window, then falls back to a depth-bounded walk for a focused editable node — which catches Compose fields and web views that never report input focus the standard way.Engine state between dictations
Reaches into
speech-corebehaviour, so flagging it:VoicePipelinenever clearspending_utterances_onstop()/start(), and the turn detector holds mid-utterance audio until VAD sees end-of-speech. The overlay stops the mic mid-sentence and reuses one long-lived pipeline, so leftover audio surfaced inside the next dictation — including audio the user had cancelled. Filed upstream as soniqo/speech-core#120.Handled app-side: a silence flush closes the open utterance (also capturing the sentence tail), a drain waits for the engine to fall quiet before a new dictation can start, and a session gate drops results arriving outside their own dictation. An explicit
reset()upstream would let the drain go away.DictationActivitylooks structurally exposed to the same leak (one pipeline, mic toggled on and off). Left untouched rather than changed speculatively.Commits
speechActiveandpartialTextwere both clear, but both are clear betweenSpeechEndedandTranscriptionCompleted, so it could return before the result existed and report "Nothing heard". Now waits for the engine to fall quiet.Test plan
./gradlew :app:testDebugUnitTest— 15 pass, including 13TextInsertionTestcases covering cursor clamping, selection replacement, unknown/out-of-range offsets, joining-space rules, and placeholder handling./gradlew :sdk:test— passes./gradlew :app:assembleDebug— passes:sdk:connectedAndroidTestnot run — no device available in this environment; this change does not touch the SDKDocs
README.mdplus all nine translations (zh, ja, ko, es, de, fr, hi, pt, ru).