Skip to content

fix: make eye tracking work and fix issues found in device testing - #5

Merged
manupawickramasinghe merged 13 commits into
mainfrom
fix/device-test-findings
Oct 1, 2026
Merged

fix: make eye tracking work and fix issues found in device testing#5
manupawickramasinghe merged 13 commits into
mainfrom
fix/device-test-findings

Conversation

@manupawickramasinghe

@manupawickramasinghe manupawickramasinghe commented Oct 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Fixes the issues found by testing the PR stack (#1–#4) on real hardware, plus two requested changes: landscape-only and an eye-tracking pointer.

This PR is stacked on #4 (docs/modernization-guide). Merge #1 → #2 → #3 → #4 first, retargeting each next PR to main, then retarget this one to main.

Critical fixes

  • Eye tracking stopped on the 2nd frame, every time. MPImage.close() recycled the bitmap that CameraTracker reuses for every frame. MediaPipe now gets a per-frame copy.
  • Release (R8) Vision builds could not start tracking. R8 stripped protobuf-lite fields (Field platform_ … not found) and then broke Flogger's caller lookup. Added keep rules for MediaPipe, protobuf-lite messages, Flogger and ONNX Runtime.
  • Switch keys were unreliable. Android consumed the first Space/Enter after any touch (leaving touch mode), and a focused button swallowed Escape. Keys are now handled in pre-IME dispatch on a focusable root.
  • CI never built the app. setup-android@v3 failed on the obsolete tools package before Gradle ran.

Other fixes

  • Camera: restarts on return to the app and keeps its calibration, including across app restarts until stopped in Settings. The screen stays on while tracking. The permission prompt no longer pauses selection, and one press now retries after a failure.
  • Persistence: the access mode is saved, and pause state survives rotation.
  • Status and messages: camera status is now accurate (off / starting / no face / tracking, plus the provider in use). Tracker failures are logged. After repeated camera denials the button opens App info.
  • Rebuilds and layout: UI rebuilds keep scroll position and Settings open; rows don't misalign on wrapped labels.
  • Settings and speech: value readouts, readable voice names with a spoken preview, an "Install voices" button, and a muted-media warning.
  • Delete: removes Sinhala combining signs one at a time.
  • Scanning: row/column scanning, with a pinned message preview on narrow screens.
  • Translations: Greek is fully translated (56 strings were English), and new strings are added to every pack. The validator now rejects untranslated strings and lost {placeholders}.
  • Telemetry: ONNX Runtime's launch-time telemetry provider is removed. MediaPipe's datatransport logger has to stay (FaceLandmarker fails without it), but it cannot upload without INTERNET; this is documented.
  • Build: Vision builds fail when the face model is missing or its checksum is wrong. The NDK pin moves to 28.2.13676358.

Requested changes

  • Landscape only (sensorLandscape). Wide screens use a two-column board, so the whole keyboard and phrases fit without scrolling. Flipping the device restarts the camera, and calibration is tied to the screen size and rotation.
  • Eye-tracking pointer drawn at the calibrated gaze point, with a ring that fills as dwell progresses.

Verified on device

The original bugs were reproduced on a Redmi Note 13 (Android 16) and a Lenovo Tab M11 (Android 15). The phone was disconnected before the fixes, so every fix here was verified on the tablet only, debug and R8 release:

  • Tracking runs continuously, with no recycled-bitmap warnings and no tracker failures.
  • The R8 release tracks for 30 s+.
  • Camera restart works on return from the background.
  • Mode and pause survive rotation.
  • The touch → Space, focused → Escape and paused → Space sequences each passed 3/3.
  • Row scanning types the scanned key.
  • Landscape layout fits on the tablet.
  • Lint is clean, and the pack and calibration validators pass.
  • CI: native, Android Lite and Android Vision jobs all pass (the Android jobs never ran before).

Code-only (not exercised on a device)

  • The 180° flip (adb can't rotate a sensorLandscape app).
  • The pointer on screen and calibration persistence. Both need a person in front of the camera.

Not tested

  • These fixes on the phone, including landscape on a short phone screen and HyperOS key handling.
  • Gaze accuracy.
  • NNAPI.
  • TalkBack and Switch Access.
  • Real switch hardware (keys were injected with adb).
  • Audio quality.
  • Long-run thermals.

The Greek strings and the new Android strings in other languages are machine-drafted and need native-speaker review.

🤖 Generated with Claude Code

https://claude.ai/code/session_014Csi3Kyo14x29WV5eKMdNM

manupawickramasinghe and others added 4 commits October 1, 2026 17:40
@manupawickramasinghe @claude
fix: make eye tracking work and harden switch, lifecycle and layout b…
1a06ae3 
…ehavior

Device testing of feat/camera-inference on a Redmi Note 13 (Android 16) and a
Lenovo Tab M11 (Android 15) found that eye tracking stopped on every second frame
and that release builds could not start tracking at all.

Vision
- MPImage.close() recycled the bitmap CameraTracker reuses for every frame, so the
  next frame failed. MediaPipe now gets a per-frame copy.
- R8 broke MediaPipe (protobuf-lite fields, Flogger stack lookup). Add keep rules
  for MediaPipe, protobuf-lite messages, Flogger and ONNX Runtime.
- Tracker failures are logged with their exception and reported with
  language-pack keys; a failed tracker is cleared so one press retries.
- Remove ONNX Runtime's launch-time telemetry provider and disable its telemetry.
  MediaPipe's datatransport logger must stay (FaceLandmarker needs it); INTERNET
  remains removed.
- Fail Vision builds when face_landmarker.task is missing or its SHA-256 differs.

App
- Handle Escape/Space/Enter before focus handling (pre-IME on a focusable root):
  Android consumed the first switch press after a touch, and focused buttons
  swallowed Escape.
- Row/column scanning with a pinned message preview.
- Restart the camera on return, keep calibration across backgrounding and app
  restarts, keep the screen on while tracking, don't pause for the permission
  prompt, persist the access mode and keep pause state across rotation.
- Landscape-only, with a two-column board on wide screens so it fits without
  scrolling; restart the camera when the device is flipped.
- Eye-tracking pointer that follows the calibrated gaze and shows dwell progress.
- Honest camera status (off / starting / no face / tracking + provider), App info
  link after a permanent camera denial, Settings value readouts, readable voice
  names with a preview, an Install voices button and a muted-media warning.
- UI rebuilds keep the scroll position and Settings; rows no longer misalign on
  wrapped labels; Delete removes combining signs one at a time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Csi3Kyo14x29WV5eKMdNM
@manupawickramasinghe @claude
fix(i18n): complete Greek, localize new UI strings and reject untrans…
8682dc1 
…lated text

Greek had 56 of 70 UI strings in English. Translate them and add the new camera,
calibration, scanning and voice strings to all six packs. The validator now fails
on strings left identical to English (with an allowlist for words spelled the
same) and on translations that drop a {placeholder}. Android-specific strings and
the Greek UI are machine-drafted and need native-speaker review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Csi3Kyo14x29WV5eKMdNM
@manupawickramasinghe @claude
ci: fix Android SDK setup and align the NDK pin
1c9970c 
android-actions/setup-android@v3 installs 'tools platform-tools' by default and
current cmdline-tools reject the obsolete 'tools' package, so both Android jobs
failed before Gradle ran. Request only platform-tools, and install NDK
28.2.13676358 to match app/build.gradle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Csi3Kyo14x29WV5eKMdNM
@manupawickramasinghe @claude
docs: record device validation and describe the new behavior
4671987 
Add the 1 October 2026 device validation (both test devices, debug and R8
release), document landscape mode, row/column scanning, switch keys, the gaze
pointer, camera lifecycle and calibration persistence, the dependency telemetry
that remains, and the Vision model build check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Csi3Kyo14x29WV5eKMdNM
Copilot AI balanced review requested due to automatic review settings October 1, 2026 12:11
Copilot AI reviewed Oct 1, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

manupawickramasinghe and others added 4 commits October 1, 2026 17:44
@manupawickramasinghe @claude
fix(build): only require the face model when packaging Vision
948d0c7 
Lite builds reach preVisionDebugBuild through the CMake build shared by both
flavors, so the model check failed CI's Lite job, which downloads no model. Gate
it on mergeVision*Assets instead. Verified locally: Lite builds and lints without
the model; Vision fails at verifyVisionModel without it and builds with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Csi3Kyo14x29WV5eKMdNM
@manupawickramasinghe @claude
docs: attribute each validation result to the device that ran it
fb5540e 
The phone was disconnected before the fixes were made, so it only ran the original
branch and the one-line diagnostic patch; every fix was verified on the tablet.
Say so, and list the fixes on the phone as untested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Csi3Kyo14x29WV5eKMdNM
@manupawickramasinghe @claude
feat: camera preview, calibration that waits for the face, Settings page
7bb84f0 
Testing on the tablet showed calibration failing with "not accurate enough", so
the app stayed in touch mode and the pointer never appeared. There was also no
way to see what the camera saw.

- Live camera preview (what the model receives, with iris and eye-corner dots) in
  the header and as the first calibration step: calibration starts once the face
  has been found for a second.
- Nine calibration targets with a progress ring; time only counts while the face
  is tracked; per-target medians. The fit reports its error and the app accepts
  anything under 25% of the screen, telling the user the typical error.
- Capture near 640x480 instead of 320x240, detection confidence 0.5, and logged
  diagnostics (detection rate, feature spread, per-target samples, fit error).
- Smooth the pointer in screen space after mapping, and ease it between samples.
- Settings is its own two-column page; wide screens fold the navigation row into
  the header.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Csi3Kyo14x29WV5eKMdNM
@manupawickramasinghe @claude
feat(vision): zoom in on the face, add head turn to gaze, find the ri…
a65be4b 
…ght rotation

On the tablet the tracker never found a face (0 frames with eyes), so calibration
never started and there was no pointer.

- Follow the face with the camera's sensor crop (up to 3x): the eyes fill more of
  the frame at full sensor detail, and the crop only re-centres when the face
  drifts, which keeps the face frame steady. It zooms out when the face is lost.
- Use face-priority exposure (with +0.5 EV) so a bright lamp behind the user does
  not leave the face too dark to detect.
- If no face is found, try the other image rotations, since some devices report
  the wrong sensor orientation; log the rotation that works.
- Gaze features are now iris position plus a head-turn proxy; calibration fits any
  number of features (tests cover 2 and 4) and stored calibrations record theirs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Csi3Kyo14x29WV5eKMdNM
@manupawickramasinghe manupawickramasinghe mentioned this pull request Oct 1, 2026
Merged
manupawickramasinghe and others added 5 commits October 1, 2026 20:58
@manupawickramasinghe @claude
fix(vision): software face zoom, head pointer before calibration, cam…
8dead8a 
…era restart

The face was found on the tablet, but setting a sensor crop that followed it made
the MediaTek MT6768 ISP fail ("support no arbitrary cropping"), crash the camera
HAL and stop tracking with CameraDevice error 4.

- Capture at ~1280x960 and crop around the face in software, easing the crop
  after the face; no SCALER_CROP_REGION is used any more.
- Show the round pointer as soon as a face is tracked: before calibration it
  follows head turns from the starting pose, afterwards the calibrated gaze.
  Dwell selection stays limited to Eye tracking mode.
- Restart the camera after a camera-service failure (up to 3 times a minute).
- Make the rotation search conservative (a false detection on the ceiling locked
  the wrong rotation): more time and less evidence for the sensor rotation, eight
  consecutive faces for any other, and implausible eye/head readings rejected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Csi3Kyo14x29WV5eKMdNM
@manupawickramasinghe @claude
feat: visible pointer, blink to select, larger keyboard, per-axis gaz…
266160c 
…e features

- Fix the pointer never showing: an operator-precedence bug treated "no
  calibration" as a changed screen size, and every blink or dropped frame
  re-centred the head pointer and hid it for ~1.5 s. Dropouts now keep the last
  gaze for 500 ms and the pointer shows whenever a face is tracked.
- Blink to select (on by default, 600 ms): using MediaPipe's eyeBlink blendshapes,
  closing both eyes for 0.6-2.5 s presses the button under the pointer, or acts as
  the switch while scanning. Natural blinks and resting the eyes are ignored.
- Wide landscape board: message beside its actions, one phrase row, and a
  full-width keyboard in real rows that fills the remaining height (keys ~2.5x
  larger on the tablet). English uses QWERTY rows; other packs are balanced into
  at most four rows.
- Calibration measured on the tablet: horizontal eye position worked (r = 0.98)
  but the lid-relative vertical feature had no signal. Features are now eyeX,
  eyeY against the eye corners, head x/y, blendshape look x/y and lid opening,
  fitted per axis; calibration dots stay inside the system bars with the text and
  Cancel between rows.
- Debug builds accept a DEBUG_GAZE broadcast to drive the pointer and blinks over
  adb, so pointer, dwell and blink can be tested without a face.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Csi3Kyo14x29WV5eKMdNM
@manupawickramasinghe @claude
feat: lock the pointer onto buttons and give it momentum
1c28431 
- Magnetic lock: when the pointer rests on a button, or within 28 dp of one, for
  120 ms it locks on, turns green and settles at the button's centre. It stays
  locked until the gaze has been outside the button plus a margin (a quarter of
  its smaller side, at least 16 dp) for 150 ms. Dwell and blink act on the locked
  button, so jitter no longer resets a dwell; a blink presses the button that was
  locked when the eyes closed.
- Momentum: the pointer is a slightly under-damped spring integrated every display
  frame (Choreographer), stiffer while locked, instead of an exponential ease at
  the 20 Hz tick.

Verified on the tablet with injected gaze: a point in the gap between two buttons
locked onto the nearer one; after a jump the pointer was mid-glide, then on target.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Csi3Kyo14x29WV5eKMdNM
@manupawickramasinghe @claude
feat(vision): lock the camera onto the face
fe889cd 
Once the face has been steady for three frames the zoomed software crop freezes
on it instead of easing after every movement. It re-frames only when the eyes
near the edge of the frame (outside the middle half) or the eye distance changes
by more than about a third, gliding to the new framing and locking again, and it
holds the lock for 3 s through blinks and brief turns before zooming out. The
preview shows a green border while locked, and lock/re-frame/release are logged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Csi3Kyo14x29WV5eKMdNM
@manupawickramasinghe @claude
Merge docs/modernization-guide into fix/device-test-findings
06f9e9c 
Brings in PR #4's CI fix and its rewritten validation record. The workflow keeps
this branch's version (same platform-tools fix, NDK 28.2.13676358 to match
app/build.gradle). validation.md keeps PR #4's PR 1-4 section as written and adds
this PR's device validation after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Csi3Kyo14x29WV5eKMdNM
@manupawickramasinghe
manupawickramasinghe changed the base branch from docs/modernization-guide to main October 1, 2026 19:23
@manupawickramasinghe
manupawickramasinghe merged commit 8125914 into main Oct 1, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@manupawickramasinghe