From 0d2cc68eb68b915686485afdb1666457041c3116 Mon Sep 17 00:00:00 2001 From: "M.N Wickramasinghe" Date: Thu, 1 Oct 2026 13:53:49 +0530 Subject: [PATCH 1/3] docs: document architecture, hardware, development and validation --- .github/dependabot.yml | 8 +++ .github/workflows/android.yml | 44 ++++++++++++ README.md | 84 +++++++++++++++++++++-- docs/README.md | 7 ++ docs/architecture.md | 42 ++++++++++++ docs/development.md | 45 ++++++++++++ docs/getting-started.md | 51 ++++++++++++++ docs/hardware.md | 59 ++++++++++++++++ docs/languages.md | 26 +++++++ docs/security.md | 31 +++++++++ docs/third-party.md | 17 +++++ docs/validation.md | 22 ++++++ examples/language-pack/es.json | 122 +++++++++++++++++++++++++++++++++ 13 files changed, 553 insertions(+), 5 deletions(-) create mode 100644 .github/dependabot.yml create mode 100644 .github/workflows/android.yml create mode 100644 docs/README.md create mode 100644 docs/architecture.md create mode 100644 docs/development.md create mode 100644 docs/getting-started.md create mode 100644 docs/hardware.md create mode 100644 docs/languages.md create mode 100644 docs/security.md create mode 100644 docs/third-party.md create mode 100644 docs/validation.md create mode 100644 examples/language-pack/es.json diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000..032236a --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,8 @@ +version: 2 +updates: + - package-ecosystem: gradle + directory: / + schedule: {interval: weekly} + - package-ecosystem: github-actions + directory: / + schedule: {interval: weekly} diff --git a/.github/workflows/android.yml b/.github/workflows/android.yml new file mode 100644 index 0000000..27ca578 --- /dev/null +++ b/.github/workflows/android.yml @@ -0,0 +1,44 @@ +name: Android and native checks +on: + push: + branches: [main] + pull_request: + workflow_dispatch: +permissions: + contents: read +concurrency: + group: android-${{ github.ref }} + cancel-in-progress: true +jobs: + native: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-java@v4 + with: + distribution: temurin + java-version: '17' + - run: cmake -S . -B build/host -DZEKALS_SANITIZERS=ON && cmake --build build/host -j2 && ctest --test-dir build/host --output-on-failure + - run: bash scripts/check-calibration.sh && python3 scripts/validate_packs.py + android: + runs-on: ubuntu-latest + strategy: + matrix: + flavor: [Lite, Vision] + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-java@v4 + with: + distribution: temurin + java-version: '17' + - uses: android-actions/setup-android@v3 + - uses: gradle/actions/setup-gradle@v4 + - run: sdkmanager 'platforms;android-35' 'build-tools;35.0.0' 'ndk;28.0.13004108' 'cmake;3.22.1' + - if: matrix.flavor == 'Vision' + run: python3 scripts/download_model.py + - run: ./gradlew :app:assemble${{ matrix.flavor }}Debug :app:lint${{ matrix.flavor }}Debug --no-daemon + - uses: actions/upload-artifact@v4 + with: + name: zekals-${{ matrix.flavor }}-debug + path: app/build/outputs/apk/**/*.apk + retention-days: 14 diff --git a/README.md b/README.md index 6cfc6d7..9dcee67 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,82 @@ # zekALS Android -Native Android companion to [zekALS](https://github.com/Zektopic/zekals-containerized). -Development is organized into focused pull requests for native input processing, -accessible communication and speech, optional camera inference, and documentation. +**Native, local communication with accessible controls and offline speech.** -This initial branch records the repository purpose. The implementation and its -validation notes are in the linked pull requests; merge their stack in order. +zekALS Android is the mobile companion to +[zekALS](https://github.com/Zektopic/zekals-containerized). It provides a message +editor, phrase boards, multilingual character pages, and speech using installed +Android voices. Touch, keyboards, accessibility services and single switches work +independently of camera tracking. + +## Choose the appropriate build + +| Build | Intended use | Included components | +| --- | --- | --- | +| **Lite** | Older phones, low memory devices, touch/switch communication | Native UI, offline Android TTS, C++ dwell/filter core; no camera or ML libraries | +| **Vision** | Optional experimental webcam gaze input | Lite functionality plus Camera2 capture, C++ image conversion, MediaPipe and optional ONNX adapters | + +Lite targets ARM64, ARMv7 and x86_64. Vision targets ARM64 and x86_64. Android 8.0 +(API 26) or newer is required. CPU is the default inference choice. GPU and legacy +NNAPI acceleration are optional and depend on the model, runtime and device; +NNAPI selection is not proof of NPU execution. + +## Features + +- Native controls with large touch targets, font scaling, high contrast, visible + selection, pause, undo, fixed scrolling controls and access-mode settings. +- English, French, Simplified Chinese, Italian and Sinhala packs; Greek retained. + Language assets share the desktop format and can be extended without Java edits. +- Installed offline voices, language matching, quality-based voice ordering, + speed control and immediate stop. No cloud API key or bundled voice is required. +- A C++/JNI core with validated buffers, strided YUV conversion, rotation/mirroring, + time-based filtering and selection that fires once per entry. +- Vision captures the latest frame, caps inference at 10 Hz, reduces it under + thermal pressure, cancels stale input and calibrates before gaze selection. +- No application internet permission, background camera service, conversation + logging or message backup. Models are a deliberate build-time installation. + +## Build + +Install JDK 17 and the Android SDK, including API 35, Build Tools 35.0.0, +NDK 28.0.13004108 and CMake 3.22.1. Set `ANDROID_HOME` to the SDK directory. + +```bash +./gradlew :app:assembleLiteDebug :app:lintLiteDebug +``` + +For Vision, first download the pinned model and then build: + +```bash +python3 scripts/download_model.py +./gradlew :app:assembleVisionDebug :app:lintVisionDebug +``` + +Install the chosen APK from `app/build/outputs/apk/`. Debug APKs are development +artifacts. A production release requires your signing configuration and physical +device acceptance testing; signing keys are never stored in this repository. + +## Documentation + +- [Installation and operation](docs/getting-started.md) +- [Accessibility and speech](docs/accessibility.md) +- [Hardware, models and performance](docs/hardware.md) +- [Architecture and lifecycle](docs/architecture.md) +- [Language packs](docs/languages.md) +- [Native C++ interface](docs/native-core.md) +- [Security and privacy](docs/security.md) +- [Development and validation](docs/development.md) +- [Validation record and limitations](docs/validation.md) +- [Third-party licenses](docs/third-party.md) + +## Status + +The communication foundation has produced a debug APK for all three baseline +architectures and passed Android lint. The expanded Lite/Vision source is staged +for review; the final Vision build and physical device testing are still required. +Webcam gaze is experimental and needs individual calibration and user/caregiver +trials. No universal NPU support, medical efficacy or measured end-to-end speedup +is claimed. A natural Sinhala voice must be supplied by a compatible installed +engine; none is bundled. + +Application code is [MIT licensed](LICENSE). Runtime libraries, camera models and +installed voices retain their own licenses. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..928f05e --- /dev/null +++ b/docs/README.md @@ -0,0 +1,7 @@ +# Documentation + +Start with [setup](getting-started.md) and [access controls](accessibility.md). +See [hardware](hardware.md), [architecture](architecture.md), +[languages](languages.md), [native interface](native-core.md), +[security](security.md), [development](development.md), +[validation](validation.md), and [third-party notices](third-party.md). diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..0ad64a9 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,42 @@ +# Architecture and lifecycle + +`MainActivity` owns accessible Android views, composition/undo state, speech and +access modes. `LanguagePack` reads bounded local JSON assets. Android TTS owns voice +models; the app filters for installed offline voices matching the pack language. +UI preferences are shared preferences; message text never goes into preferences, +files, logs or automatic view-state saving. Rotation retains the message through +an in-process non-configuration instance, not a persistent bundle. + +`NativeCore` passes caller-owned arrays/direct buffers across JNI. `native/core.cpp` +implements smoothing, dwell and image conversion. No long-lived native pointer is +exposed to Java. Bounds and strides are checked before pixel access. Native tests +cover invalid samples, reacquisition, selection locking and strided conversion. + +The Lite source set supplies a small camera-disabled implementation and excludes +camera dependencies/permissions. Vision supplies `CameraTracker`, `MediaPipeEstimator` +and `OnnxEstimator`. Camera/model construction and inference share one worker +thread, including the GPU delegate's thread affinity. It consumes latest images, +closes them deterministically, rate-limits processing and coalesces UI callbacks. +Provider failures invalidate input or use an explicit CPU fallback. + +The activity owns a generation counter for camera sessions. Late results from a +stopped/replaced session are ignored. A sample's capture receipt time expires after +500 ms. Stop and activity backgrounding release the camera, ImageReader, model and +worker. The app has no foreground camera service or hidden background recording. + +Calibration fits a two-output affine transform from five normalized feature +samples. It rejects singular systems and high fitting residuals. This is a basic +webcam iris estimator and does not compensate robustly for all head motion or +clinical conditions. The native dwell state fires once per target entry. After +activation, gaze must leave the activated rectangle before another action, even +when a keyboard page rebuilds its Android views. + +All speech and access controls stay available without a camera. Scan mode brings +controls into view; the scroll and pause controls stay outside the scrolling +content. The message editor uses system IME support. Native accessibility semantics +come from real Buttons, EditText and live-region TextViews. + +Runtime fallback keeps the same model family and coordinate meaning: ONNX retries +its model on CPU; MediaPipe retries its model on CPU. If ONNX still fails, tracking +stops so an existing calibration cannot silently be reused with a different model. +An unavailable custom ONNX model can select MediaPipe at startup, before calibration. diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..96aaf8c --- /dev/null +++ b/docs/development.md @@ -0,0 +1,45 @@ +# Development and testing + +```bash +cmake -S . -B build/host -DZEKALS_SANITIZERS=ON +cmake --build build/host -j2 +ctest --test-dir build/host --output-on-failure +bash scripts/check-calibration.sh +python3 scripts/validate_packs.py +./gradlew :app:assembleLiteDebug :app:lintLiteDebug +python3 scripts/download_model.py +./gradlew :app:assembleVisionDebug :app:lintVisionDebug +``` + +The native tests exercise production core functions under ASan and UBSan. +Calibration tests compile the production pure-Java class and check affine mapping +and rejection of degenerate samples. Language validation uses real bundled assets. +CI builds both flavors and uploads debug artifacts without signing/publishing a +release. Gradle dependencies and wrapper distribution are version-pinned. + +A ptrace-based sandbox can prevent LeakSanitizer initialization. In that environment +`ASAN_OPTIONS=detect_leaks=0 ctest ...` still checks address/undefined behavior but +does not perform leak detection. Record the distinction. Do not disable checks in +normal CI to conceal a failure. + +Physical-device and accessibility tests remain necessary: TalkBack/Switch Access, +real switches, font scaling, IMEs, voice installation/loss, denied permission, +backgrounding, rotation, calibration, tracking loss and battery/thermal behavior. +Use synthetic text and do not record private conversations in bug reports. + +## Review sequence + +1. Native C++ core, JNI and reproducible Android build foundation. +2. Accessible native communication, language assets and offline speech. +3. Lite/Vision separation, camera lifecycle, calibration and inference providers. +4. Documentation, CI, dependency updates and explicit validation record. + +Merge in order and rerun CI after retargeting stacked PRs. A successful source +compile is distinct from APK packaging, model initialization and device testing. + +## Planned improvements + +Add emulator UI/instrumentation coverage, provider-specific device testing, +calibration validation targets independent of fitting targets, custom phrase +import/export, native-speaker review, and sustained resource benchmarks. A direct +modern vendor NPU adapter can replace the legacy NNAPI option after validation. diff --git a/docs/getting-started.md b/docs/getting-started.md new file mode 100644 index 0000000..465f8f4 --- /dev/null +++ b/docs/getting-started.md @@ -0,0 +1,51 @@ +# Installation and operation + +## Development setup + +Use JDK 17 and Android SDK packages `platforms;android-35`, `build-tools;35.0.0`, +`ndk;28.0.13004108` and `cmake;3.22.1`. The Gradle wrapper pins 8.11.1 and verifies +the distribution SHA-256. Android Gradle Plugin is pinned to 8.9.2. Open the root +in Android Studio or set `JAVA_HOME` and `ANDROID_HOME` for command-line builds. +Windows uses `gradlew.bat`; Python scripts are platform-independent. + +Choose Lite for communication without camera libraries. Run +`./gradlew :app:assembleLiteDebug :app:lintLiteDebug`. The APK is under +`app/build/outputs/apk/lite/debug/`. Install with `adb install -r ` after +explicitly authorizing the development computer on the Android device. + +For Vision, run `python3 scripts/download_model.py`, then +`./gradlew :app:assembleVisionDebug :app:lintVisionDebug`. The downloader verifies +the pinned face model before atomic installation in `app/src/vision/assets/`. +Do not rename arbitrary models to this filename. Model weights are not committed. +The model is not included in Lite assets or downloaded while someone communicates. + +## First use + +1. Set the device media volume to a comfortable level. +2. Choose the language using the language button. Enter a short phrase and Speak. +3. If no voice is available, install a matching offline voice through Android's + text-to-speech settings. Return to the app and choose Voice in Settings. +4. Set font size, contrast, dwell time and scanning interval with the user. +5. Test Clear and Undo, Pause and Stop before enabling automatic input. + +The editor uses the installed Android keyboard/IME. The on-screen Chinese board +contains common characters, not a complete Pinyin dictionary. Sinhala has separate +letters/combining marks and a join control; validate typing and deletion with a +native speaker. Phrase buttons insert text without unexpectedly speaking it. + +## Optional camera + +Use Vision. In Settings, choose an inference request (CPU first), then Camera: +start/stop. The app requests camera permission only on this explicit action. No +microphone or internet permission is requested. A denied permission leaves the +other controls available. The front camera is used without a preview or recording. + +Start Calibrate and look at each of five targets for three seconds. Sampling starts +after a settling interval. Lost tracking, too few samples, degenerate input or +large fitting error rejects the calibration. Try the large phrase buttons first. +Recalibrate after repositioning or activity recreation. Keyboard-page changes +preserve calibration within the same viewport. + +Pause/Stop and alternate input remain available. Camera and speech stop when the +activity leaves the foreground. Resume and restart the camera deliberately after +returning. NNAPI requires the optional custom model described in the hardware guide. diff --git a/docs/hardware.md b/docs/hardware.md new file mode 100644 index 0000000..ce801aa --- /dev/null +++ b/docs/hardware.md @@ -0,0 +1,59 @@ +# Hardware, models and performance + +## Capability matrix + +| Target | Implementation | Validation status | +| --- | --- | --- | +| ARMv7 Android | Lite native UI, speech and C++ core | Baseline APK compiled; device testing pending | +| ARM64 Android | Lite or Vision | Baseline JNI/APK compiled; final Vision/device testing pending | +| x86_64 Android | Lite or Vision/emulator | Baseline JNI/APK compiled; final Vision/emulator testing pending | +| CPU inference | MediaPipe Face Landmarker | Source integrated; Android camera/model test pending | +| GPU inference | Actual MediaPipe GPU delegate, CPU fallback | Depends on EGL/driver/runtime; hardware test pending | +| GPU/NPU via NNAPI | Optional ONNX Runtime NNAPI request | Custom compatible model required; device assignment and accuracy unverified | +| Qualcomm QNN / vendor NPU SDKs | No direct adapter bundled | Future work; do not advertise native vendor support | + +NNAPI is deprecated by Android and is an opt-in compatibility path. It can route +operations to different devices or fall back. The app requests the CPU-disabled +NNAPI flag and falls back to ONNX CPU when session creation/execution fails. +An unavailable or incompatible ONNX asset falls back to the default MediaPipe CPU +model when that model is installed. Inspect the active provider display; a selector +label alone is not proof of accelerated execution. See +[Android NNAPI](https://developer.android.com/ndk/guides/neuralnetworks) and +[ONNX NNAPI](https://onnxruntime.ai/docs/execution-providers/NNAPI-ExecutionProvider.html). + +## Resource use + +Lite excludes both camera inference dependencies. Vision uses an ImageReader with +two buffers and `acquireLatestImage`; it closes every acquired image. Preprocessing +reuses a direct RGBA buffer and a Bitmap. C++ handles plane strides, rotation and +mirroring. Inference is capped at 10 Hz (5 Hz under moderate-or-higher thermal +pressure on supported Android versions). UI delivery replaces pending gaze work +instead of building a queue. Samples older than 500 ms cannot select controls. + +The custom ONNX adapter reuses its tensor/pixel arrays, but resizing and runtime +outputs may allocate. MediaPipe/runtime allocations are outside the small C++ core. +Measure actual RSS, GC, inference latency, power and temperature; do not infer +whole-app allocation behavior from the native helper's allocation-free design. + +## Custom ONNX contract + +Put a reviewed `gaze.onnx` and its lowercase SHA-256 text in `gaze.sha256` under +`app/src/vision/assets/`. The adapter verifies the asset and bounds model size to +128 MiB. It expects exactly one float32 `[1,3,H,W]` RGB input, values in `[0,1]`, +with fixed dimensions between 16 and 1024, and one `[1,3]` output containing +normalized x, y and confidence. Confidence below 0.7 invalidates the sample. +The input has already been rotated upright and mirrored for the front camera. + +No compatible trained ONNX gaze model is bundled. Arbitrary face/iris models do +not satisfy this contract. Review licenses, training population, preprocessing, +quantization and accuracy before using a custom model. Keep its runtime libraries +compatible with the device and record their versions. + +## Device acceptance + +Record build flavor, ABI, Android version, model checksum, runtime/delegate, actual +capture resolution, median/p95 frame-to-input latency, peak memory and ten-minute +thermal behavior. Test camera denial, disconnect, app backgrounding, rotation, +tracking loss, closed eyes, screen resizing, calibration error and sustained speech. +Include people with the intended access needs in evaluation. Physical-device +performance and gaze accuracy have not been measured in this development session. diff --git a/docs/languages.md b/docs/languages.md new file mode 100644 index 0000000..33333f3 --- /dev/null +++ b/docs/languages.md @@ -0,0 +1,26 @@ +# Language and voice expansion + +Packs live in `app/src/main/assets/languages/`. English, French, Simplified Chinese, +Italian, Sinhala and Greek use the desktop schema version 1. The first five contain +complete desktop UI keys; Greek retains some English fallback. Android adds +`cameraControl`, `inference` and `cancelCalibration`. Translations require native +speaker/user review before being marked reviewed. + +Copy the example in `examples/language-pack/es.json`, change its identity/locale, +translate UI strings and phrases, and define keyboard rows and extra pages. The +example is incomplete and is not shipped as Spanish support. Set `direction` for +right-to-left packs. Keep files below 64 KiB and bounded keys/phrases as documented +by `scripts/validate_packs.py`. Run that script and rebuild; asset discovery does +not require Java edits. Do not place markup or executable actions inside keys. + +Use the system IME for unrestricted Chinese entry. The virtual board deliberately +contains a small set of common Hanzi. Sinhala combining marks show a dotted-circle +label while inserting only the actual mark; Join letters inserts a zero-width +joiner. Deletion uses Android ICU grapheme boundaries, which must be checked on the +minimum supported OS and with native conjunct examples. + +The pack's locale selects matching installed Android voices. Piper metadata is +shared documentation for the desktop and is not a mobile neural model. Natural +voice quality depends on the installed Android engine/voice. Sinhala requires a +Sinhala-capable installed offline voice; none is bundled. Do not silently choose +English or send text to a network voice when a requested voice is absent. diff --git a/docs/security.md b/docs/security.md new file mode 100644 index 0000000..80822f0 --- /dev/null +++ b/docs/security.md @@ -0,0 +1,31 @@ +# Security and privacy + +The application requests no internet permission; the manifest removes it even if a +dependency proposes it. Lite requests no camera permission. Vision requests camera +access only after an explicit start action, and camera hardware is optional. +There is no microphone access, WebView, web server, cloud API key or JavaScript +bridge. The launcher Activity is the only exported component. + +Messages and undo are in memory. Android view-state message saving is disabled; +backups and device transfer are excluded using both legacy and modern Android +configuration. Preferences contain access/language/voice settings only. Process +death clears the draft. Screen capture, a trusted accessibility service or an +installed TTS engine can observe on-screen/spoken text and remain separate trust +boundaries. The chosen voice must be local and already installed. + +The JNI boundary validates buffer dimensions, capacities, row/pixel strides, +rotation and finite samples. Camera buffers remain owned by Android and are used +only while the Image is open. The native interface exposes no allocation handles. +Native code still needs review and fuzzing; passing sanitizers does not prove that +all memory errors are absent. + +Models are packaged deliberately after checksum verification. Checksums establish +artifact identity, not model safety or accuracy. Installed TTS engines, models, +vendor drivers and runtime native libraries need independent provenance and +license review. An optional ONNX model is copied into app-private cache for runtime +loading and removed on normal estimator close; no camera frames are cached. + +Do not commit signing keys, personal phrases, model binaries or recordings. Debug +builds are not store releases. Before distribution, inspect merged manifests, +native-library page alignment, dependency advisories, signing and the exact target +OS. Report security issues privately with synthetic data and no message content. diff --git a/docs/third-party.md b/docs/third-party.md new file mode 100644 index 0000000..854537b --- /dev/null +++ b/docs/third-party.md @@ -0,0 +1,17 @@ +# Third-party notices + +The application's MIT license does not relicense dependencies, voices or models. +Preserve notices and review exact artifact licenses before redistribution. + +- Android SDK/NDK and Gradle are development tools with their own terms. +- MediaPipe Tasks code is Apache-2.0; face-landmarker weights have their upstream + model distribution terms. The reviewed artifact URL/checksum is in the downloader. +- ONNX Runtime is MIT licensed; bundled provider dependencies retain their notices. +- Android voice engines and downloaded voices are installed separately and retain + vendor/model terms. This app does not provide rights to redistribute them. +- Language assets originate in the companion desktop project and remain subject + to native-speaker review. No personal voice or private phrase dataset is bundled. + +References: [MediaPipe Face Landmarker Android guide](https://ai.google.dev/edge/mediapipe/solutions/vision/face_landmarker/android), +[ONNX Runtime Android](https://onnxruntime.ai/docs/tutorials/mobile/deploy-android.html), +[Android TextToSpeech](https://developer.android.com/reference/android/speech/tts/TextToSpeech). diff --git a/docs/validation.md b/docs/validation.md new file mode 100644 index 0000000..3076ae9 --- /dev/null +++ b/docs/validation.md @@ -0,0 +1,22 @@ +# Validation record + +October 2026 development session, Linux x86_64 host. + +| Check | Evidence | +| --- | --- | +| Baseline Android communication app | Gradle `assembleDebug` and `lintDebug` succeeded with AGP 8.9.2 / JDK 17 | +| Baseline ABIs | C++/JNI compiled for arm64-v8a, armeabi-v7a and x86_64 and included in the debug APK | +| Native core | Host ASan/UBSan tests passed before environment restrictions | +| Staged native recheck | Passed with leak detection disabled because the restricted runner uses ptrace | +| Updated Lite Java source | Compiled against the actual Android API 35 SDK; generated flavor constant supplied for the offline compile | +| Calibration | Production Java affine mapping and degeneracy tests passed | +| Language packs | Six bundled packs pass the standard-library validator | +| Final Lite/Vision Gradle builds | Outstanding: network and original checkout writes became unavailable mid-session | +| Vision model / GPU / NNAPI execution | Not tested on Android hardware or emulator | +| Physical access/voice/thermal tests | Outstanding | + +The baseline APK predates the new camera/flavor integration and is not a verified +Vision build. Do not distribute it as such. The final sources need a full build, +lint, dependency review and emulator/device testing once environment access is +restored. No Android gaze accuracy, NPU execution or whole-app speed claim has +been established by the native tests. diff --git a/examples/language-pack/es.json b/examples/language-pack/es.json new file mode 100644 index 0000000..3134754 --- /dev/null +++ b/examples/language-pack/es.json @@ -0,0 +1,122 @@ +{ + "schemaVersion": 1, + "code": "es", + "name": "Español", + "locale": "es-ES", + "direction": "ltr", + "reviewStatus": "example", + "keyboard": [ + [ + "Q", + "W", + "E", + "R", + "T", + "Y", + "U", + "I", + "O", + "P" + ], + [ + "A", + "S", + "D", + "F", + "G", + "H", + "J", + "K", + "L", + "Z" + ], + [ + "X", + "C", + "V", + "B", + "N", + "M" + ] + ], + "keyboardPages": [], + "phrases": [ + "Sí", + "No", + "Gracias", + "Necesito ayuda" + ], + "ui": { + "tagline": "Your words, your pace.", + "settings": "Settings", + "pause": "Pause selection", + "resume": "Resume selection", + "messageTitle": "What would you like to say?", + "messageLabel": "Your message", + "messageHint": "Your message stays on this device.", + "textPlaceholder": "Type here, choose a phrase, or use the keyboard below.", + "speak": "Leer mensaje", + "stop": "Detener voz", + "undo": "Undo", + "clear": "Clear message", + "phrases": "Everyday phrases", + "phraseHint": "Choose a phrase, then speak.", + "keyboard": "Make it yours", + "numbers": "Numbers & symbols", + "letters": "Letters", + "space": "Space", + "backspace": "Delete last character", + "done": "Done", + "settingsHint": "Choose the controls that work best for you. Escape pauses automatic selection.", + "accessMode": "Access mode", + "manual": "Touch / keyboard", + "dwell": "Pointer dwell", + "scan": "Single switch", + "gaze": "Eye tracking", + "textSize": "Text size", + "contrast": "Contrast", + "voice": "Speaking voice", + "calibration": "Eye tracking setup", + "calibrate": "Calibrate", + "large": "Large", + "standard": "Standard", + "highContrast": "High contrast", + "dark": "Dark", + "dwellTime": "Dwell time", + "scanTime": "Scan interval", + "speechSpeed": "Speech speed", + "noVoice": "No local voice installed", + "installVoice": "Install an offline voice for this language in device settings.", + "emptyMessage": "Write a message first.", + "ready": "Ready when you are.", + "speaking": "Speaking…", + "stopped": "Speech stopped.", + "cleared": "Message cleared. Undo restores it.", + "paused": "Automatic selection paused. Touch and keyboard still work.", + "speechFailed": "Speech could not start. Check your device voice and volume.", + "pointerReady": "Pointer ready · camera unavailable", + "cameraReady": "Camera connected", + "scanHelp": "Single switch: press Space to choose the highlighted button. Touch and keyboard controls always work.", + "speechEngine": "Speech engine", + "deviceVoice": "Device voice", + "neuralVoice": "Local neural voice", + "speechInfo": "Use an installed offline voice, or configure local neural speech. Sinhala voice availability depends on the engine.", + "moreKeys": "More characters", + "messageFull": "Your message is full. Speak or clear it before adding more.", + "manualHint": "Use a mouse, touch, or keyboard. Change access mode in Settings.", + "dwellHint": "Hold the pointer over a button to choose it. Move away before choosing again.", + "scanHint": "Press Space to choose the highlighted button. Escape pauses scanning.", + "gazeHint": "Calibrate in Settings before choosing buttons with your eyes.", + "calibrated": "Calibration saved for this window. Try the large phrase buttons first.", + "needCamera": "Connect the camera service before calibration.", + "lookTarget": "Look at the target and hold still", + "calibrationLost": "Tracking lost during calibration. Please retry.", + "scrollUp": "Subir", + "scrollDown": "Bajar", + "joinLetters": "Unir letras" + }, + "speech": { + "locale": "es-ES", + "piperVoice": null + } +} From f45a77cd6de392cbe77d40fc7011543ebd2ef3f1 Mon Sep 17 00:00:00 2001 From: "M.N Wickramasinghe" Date: Thu, 1 Oct 2026 18:31:26 +0530 Subject: [PATCH 2/3] fix(ci): avoid the obsolete Android SDK tools package --- .github/workflows/android.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/android.yml b/.github/workflows/android.yml index 27ca578..6e60d59 100644 --- a/.github/workflows/android.yml +++ b/.github/workflows/android.yml @@ -32,6 +32,8 @@ jobs: distribution: temurin java-version: '17' - uses: android-actions/setup-android@v3 + with: + packages: platform-tools - uses: gradle/actions/setup-gradle@v4 - run: sdkmanager 'platforms;android-35' 'build-tools;35.0.0' 'ndk;28.0.13004108' 'cmake;3.22.1' - if: matrix.flavor == 'Vision' From f59c1ee48140307d3a8169f98739973a1a8511cd Mon Sep 17 00:00:00 2001 From: "M.N Wickramasinghe" Date: Thu, 1 Oct 2026 18:35:06 +0530 Subject: [PATCH 3/3] docs(validation): record final Android build results --- docs/validation.md | 33 ++++++++++++++++----------------- 1 file changed, 16 insertions(+), 17 deletions(-) diff --git a/docs/validation.md b/docs/validation.md index 3076ae9..4b452bf 100644 --- a/docs/validation.md +++ b/docs/validation.md @@ -1,22 +1,21 @@ # Validation record -October 2026 development session, Linux x86_64 host. +October 2026 development session, Linux x86_64 host. These checks cover the PR 1–4 Android stack currently open for review. Results from the later Android device-fix PR are identified separately because it has not merged into this stack. -| Check | Evidence | +| Check | Result | | --- | --- | -| Baseline Android communication app | Gradle `assembleDebug` and `lintDebug` succeeded with AGP 8.9.2 / JDK 17 | -| Baseline ABIs | C++/JNI compiled for arm64-v8a, armeabi-v7a and x86_64 and included in the debug APK | -| Native core | Host ASan/UBSan tests passed before environment restrictions | -| Staged native recheck | Passed with leak detection disabled because the restricted runner uses ptrace | -| Updated Lite Java source | Compiled against the actual Android API 35 SDK; generated flavor constant supplied for the offline compile | -| Calibration | Production Java affine mapping and degeneracy tests passed | -| Language packs | Six bundled packs pass the standard-library validator | -| Final Lite/Vision Gradle builds | Outstanding: network and original checkout writes became unavailable mid-session | -| Vision model / GPU / NNAPI execution | Not tested on Android hardware or emulator | -| Physical access/voice/thermal tests | Outstanding | +| Android Lite debug | `assembleLiteDebug` passed; arm64-v8a, armeabi-v7a and x86_64 JNI libraries built | +| Android Vision debug | `assembleVisionDebug` passed; arm64-v8a and x86_64 JNI libraries built with the checksum-verified MediaPipe face model | +| Android release/R8 | `assembleLiteRelease` and `assembleVisionRelease` passed on the host; outputs are unsigned APKs | +| Android debug lint | Lite and Vision pass with zero errors and two warnings each (data extraction and one untranslated emergency string) | +| Native C++ | Host ASan/UBSan build and CTest contract pass | +| Calibration | Production Java affine mapping and degeneracy checks pass | +| Language packs | All six bundled packs pass the Python validator | +| Android PR GitHub CI | PR #4's native, Lite and Vision jobs pass after limiting SDK setup to `platform-tools` | +| Android PR #5 | Its own GitHub CI passes. Its description reports manual validation on a Lenovo Tab M11 (Android 15); those results apply to PR #5's changes, which are not in this PR 1–4 branch yet | -The baseline APK predates the new camera/flavor integration and is not a verified -Vision build. Do not distribute it as such. The final sources need a full build, -lint, dependency review and emulator/device testing once environment access is -restored. No Android gaze accuracy, NPU execution or whole-app speed claim has -been established by the native tests. +The debug Lite APK is approximately 92 KB and the debug Vision APK approximately 98 MiB on this build. The Vision artifact is much larger because it combines camera inference runtimes, their native libraries and the model; users who do not need camera tracking should install Lite. Both release APK outputs are unsigned and are not ready to distribute. + +The host does not have a connected Android device. This session has not tested eye-tracking accuracy, actual Android CPU/GPU/NNAPI execution, camera lifecycle, switch hardware, TalkBack, translation quality or thermal behavior. PR #5 records additional device work, but its fixes must be reviewed and merged before attributing those results to the release stack. No Android build establishes that every NPU, GPU, CPU, x86 or ARM device works. Run the app with intended users and target devices before deployment. + +The earlier baseline APK predates the Lite/Vision flavors and is not a Vision build. See the getting-started and hardware guides for build commands, model integrity and per-device provider limitations.