docs: document architecture, hardware, development and validation

.github/dependabot.yml

@@ -0,0 +1,8 @@
+version: 2
+updates:
+  - package-ecosystem: gradle
+    directory: /
+    schedule: {interval: weekly}
+  - package-ecosystem: github-actions
+    directory: /
+    schedule: {interval: weekly}

.github/workflows/android.yml

@@ -0,0 +1,46 @@
+name: Android and native checks
+on:
+  push:
+    branches: [main]
+  pull_request:
+  workflow_dispatch:
+permissions:
+  contents: read
+concurrency:
+  group: android-${{ github.ref }}
+  cancel-in-progress: true
+jobs:
+  native:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-java@v4
+        with:
+          distribution: temurin
+          java-version: '17'
+      - run: cmake -S . -B build/host -DZEKALS_SANITIZERS=ON && cmake --build build/host -j2 && ctest --test-dir build/host --output-on-failure
+      - run: bash scripts/check-calibration.sh && python3 scripts/validate_packs.py
+  android:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        flavor: [Lite, Vision]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-java@v4
+        with:
+          distribution: temurin
+          java-version: '17'
+      - uses: android-actions/setup-android@v3
+        with:
+          packages: platform-tools
+      - uses: gradle/actions/setup-gradle@v4
+      - run: sdkmanager 'platforms;android-35' 'build-tools;35.0.0' 'ndk;28.0.13004108' 'cmake;3.22.1'
+      - if: matrix.flavor == 'Vision'
+        run: python3 scripts/download_model.py
+      - run: ./gradlew :app:assemble${{ matrix.flavor }}Debug :app:lint${{ matrix.flavor }}Debug --no-daemon
+      - uses: actions/upload-artifact@v4
+        with:
+          name: zekals-${{ matrix.flavor }}-debug
+          path: app/build/outputs/apk/**/*.apk
+          retention-days: 14

README.md

@@ -1,8 +1,82 @@
 # zekALS Android
 
-Native Android companion to [zekALS](https://github.com/Zektopic/zekals-containerized).
-Development is organized into focused pull requests for native input processing,
-accessible communication and speech, optional camera inference, and documentation.
+**Native, local communication with accessible controls and offline speech.**
 
-This initial branch records the repository purpose. The implementation and its
-validation notes are in the linked pull requests; merge their stack in order.
+zekALS Android is the mobile companion to
+[zekALS](https://github.com/Zektopic/zekals-containerized). It provides a message
+editor, phrase boards, multilingual character pages, and speech using installed
+Android voices. Touch, keyboards, accessibility services and single switches work
+independently of camera tracking.
+
+## Choose the appropriate build
+
+| Build | Intended use | Included components |
+| --- | --- | --- |
+| **Lite** | Older phones, low memory devices, touch/switch communication | Native UI, offline Android TTS, C++ dwell/filter core; no camera or ML libraries |
+| **Vision** | Optional experimental webcam gaze input | Lite functionality plus Camera2 capture, C++ image conversion, MediaPipe and optional ONNX adapters |
+
+Lite targets ARM64, ARMv7 and x86_64. Vision targets ARM64 and x86_64. Android 8.0
+(API 26) or newer is required. CPU is the default inference choice. GPU and legacy
+NNAPI acceleration are optional and depend on the model, runtime and device;
+NNAPI selection is not proof of NPU execution.
+
+## Features
+
+- Native controls with large touch targets, font scaling, high contrast, visible
+  selection, pause, undo, fixed scrolling controls and access-mode settings.
+- English, French, Simplified Chinese, Italian and Sinhala packs; Greek retained.
+  Language assets share the desktop format and can be extended without Java edits.
+- Installed offline voices, language matching, quality-based voice ordering,
+  speed control and immediate stop. No cloud API key or bundled voice is required.
+- A C++/JNI core with validated buffers, strided YUV conversion, rotation/mirroring,
+  time-based filtering and selection that fires once per entry.
+- Vision captures the latest frame, caps inference at 10 Hz, reduces it under
+  thermal pressure, cancels stale input and calibrates before gaze selection.
+- No application internet permission, background camera service, conversation
+  logging or message backup. Models are a deliberate build-time installation.
+
+## Build
+
+Install JDK 17 and the Android SDK, including API 35, Build Tools 35.0.0,
+NDK 28.0.13004108 and CMake 3.22.1. Set `ANDROID_HOME` to the SDK directory.
+
+```bash
+./gradlew :app:assembleLiteDebug :app:lintLiteDebug
+```
+
+For Vision, first download the pinned model and then build:
+
+```bash
+python3 scripts/download_model.py
+./gradlew :app:assembleVisionDebug :app:lintVisionDebug
+```
+
+Install the chosen APK from `app/build/outputs/apk/`. Debug APKs are development
+artifacts. A production release requires your signing configuration and physical
+device acceptance testing; signing keys are never stored in this repository.
+
+## Documentation
+
+- [Installation and operation](docs/getting-started.md)
+- [Accessibility and speech](docs/accessibility.md)
+- [Hardware, models and performance](docs/hardware.md)
+- [Architecture and lifecycle](docs/architecture.md)
+- [Language packs](docs/languages.md)
+- [Native C++ interface](docs/native-core.md)
+- [Security and privacy](docs/security.md)
+- [Development and validation](docs/development.md)
+- [Validation record and limitations](docs/validation.md)
+- [Third-party licenses](docs/third-party.md)
+
+## Status
+
+The communication foundation has produced a debug APK for all three baseline
+architectures and passed Android lint. The expanded Lite/Vision source is staged
+for review; the final Vision build and physical device testing are still required.
+Webcam gaze is experimental and needs individual calibration and user/caregiver
+trials. No universal NPU support, medical efficacy or measured end-to-end speedup
+is claimed. A natural Sinhala voice must be supplied by a compatible installed
+engine; none is bundled.
+
+Application code is [MIT licensed](LICENSE). Runtime libraries, camera models and
+installed voices retain their own licenses.

docs/README.md

@@ -0,0 +1,7 @@
+# Documentation
+
+Start with [setup](getting-started.md) and [access controls](accessibility.md).
+See [hardware](hardware.md), [architecture](architecture.md),
+[languages](languages.md), [native interface](native-core.md),
+[security](security.md), [development](development.md),
+[validation](validation.md), and [third-party notices](third-party.md).

docs/architecture.md

@@ -0,0 +1,42 @@
+# Architecture and lifecycle
+
+`MainActivity` owns accessible Android views, composition/undo state, speech and
+access modes. `LanguagePack` reads bounded local JSON assets. Android TTS owns voice
+models; the app filters for installed offline voices matching the pack language.
+UI preferences are shared preferences; message text never goes into preferences,
+files, logs or automatic view-state saving. Rotation retains the message through
+an in-process non-configuration instance, not a persistent bundle.
+
+`NativeCore` passes caller-owned arrays/direct buffers across JNI. `native/core.cpp`
+implements smoothing, dwell and image conversion. No long-lived native pointer is
+exposed to Java. Bounds and strides are checked before pixel access. Native tests
+cover invalid samples, reacquisition, selection locking and strided conversion.
+
+The Lite source set supplies a small camera-disabled implementation and excludes
+camera dependencies/permissions. Vision supplies `CameraTracker`, `MediaPipeEstimator`
+and `OnnxEstimator`. Camera/model construction and inference share one worker
+thread, including the GPU delegate's thread affinity. It consumes latest images,
+closes them deterministically, rate-limits processing and coalesces UI callbacks.
+Provider failures invalidate input or use an explicit CPU fallback.
+
+The activity owns a generation counter for camera sessions. Late results from a
+stopped/replaced session are ignored. A sample's capture receipt time expires after
+500 ms. Stop and activity backgrounding release the camera, ImageReader, model and
+worker. The app has no foreground camera service or hidden background recording.
+
+Calibration fits a two-output affine transform from five normalized feature
+samples. It rejects singular systems and high fitting residuals. This is a basic
+webcam iris estimator and does not compensate robustly for all head motion or
+clinical conditions. The native dwell state fires once per target entry. After
+activation, gaze must leave the activated rectangle before another action, even
+when a keyboard page rebuilds its Android views.
+
+All speech and access controls stay available without a camera. Scan mode brings
+controls into view; the scroll and pause controls stay outside the scrolling
+content. The message editor uses system IME support. Native accessibility semantics
+come from real Buttons, EditText and live-region TextViews.
+
+Runtime fallback keeps the same model family and coordinate meaning: ONNX retries
+its model on CPU; MediaPipe retries its model on CPU. If ONNX still fails, tracking
+stops so an existing calibration cannot silently be reused with a different model.
+An unavailable custom ONNX model can select MediaPipe at startup, before calibration.

docs/development.md

@@ -0,0 +1,45 @@
+# Development and testing
+
+```bash
+cmake -S . -B build/host -DZEKALS_SANITIZERS=ON
+cmake --build build/host -j2
+ctest --test-dir build/host --output-on-failure
+bash scripts/check-calibration.sh
+python3 scripts/validate_packs.py
+./gradlew :app:assembleLiteDebug :app:lintLiteDebug
+python3 scripts/download_model.py
+./gradlew :app:assembleVisionDebug :app:lintVisionDebug
+```
+
+The native tests exercise production core functions under ASan and UBSan.
+Calibration tests compile the production pure-Java class and check affine mapping
+and rejection of degenerate samples. Language validation uses real bundled assets.
+CI builds both flavors and uploads debug artifacts without signing/publishing a
+release. Gradle dependencies and wrapper distribution are version-pinned.
+
+A ptrace-based sandbox can prevent LeakSanitizer initialization. In that environment
+`ASAN_OPTIONS=detect_leaks=0 ctest ...` still checks address/undefined behavior but
+does not perform leak detection. Record the distinction. Do not disable checks in
+normal CI to conceal a failure.
+
+Physical-device and accessibility tests remain necessary: TalkBack/Switch Access,
+real switches, font scaling, IMEs, voice installation/loss, denied permission,
+backgrounding, rotation, calibration, tracking loss and battery/thermal behavior.
+Use synthetic text and do not record private conversations in bug reports.
+
+## Review sequence
+
+1. Native C++ core, JNI and reproducible Android build foundation.
+2. Accessible native communication, language assets and offline speech.
+3. Lite/Vision separation, camera lifecycle, calibration and inference providers.
+4. Documentation, CI, dependency updates and explicit validation record.
+
+Merge in order and rerun CI after retargeting stacked PRs. A successful source
+compile is distinct from APK packaging, model initialization and device testing.
+
+## Planned improvements
+
+Add emulator UI/instrumentation coverage, provider-specific device testing,
+calibration validation targets independent of fitting targets, custom phrase
+import/export, native-speaker review, and sustained resource benchmarks. A direct
+modern vendor NPU adapter can replace the legacy NNAPI option after validation.

docs/getting-started.md

@@ -0,0 +1,51 @@
+# Installation and operation
+
+## Development setup
+
+Use JDK 17 and Android SDK packages `platforms;android-35`, `build-tools;35.0.0`,
+`ndk;28.0.13004108` and `cmake;3.22.1`. The Gradle wrapper pins 8.11.1 and verifies
+the distribution SHA-256. Android Gradle Plugin is pinned to 8.9.2. Open the root
+in Android Studio or set `JAVA_HOME` and `ANDROID_HOME` for command-line builds.
+Windows uses `gradlew.bat`; Python scripts are platform-independent.
+
+Choose Lite for communication without camera libraries. Run
+`./gradlew :app:assembleLiteDebug :app:lintLiteDebug`. The APK is under
+`app/build/outputs/apk/lite/debug/`. Install with `adb install -r <apk>` after
+explicitly authorizing the development computer on the Android device.
+
+For Vision, run `python3 scripts/download_model.py`, then
+`./gradlew :app:assembleVisionDebug :app:lintVisionDebug`. The downloader verifies
+the pinned face model before atomic installation in `app/src/vision/assets/`.
+Do not rename arbitrary models to this filename. Model weights are not committed.
+The model is not included in Lite assets or downloaded while someone communicates.
+
+## First use
+
+1. Set the device media volume to a comfortable level.
+2. Choose the language using the language button. Enter a short phrase and Speak.
+3. If no voice is available, install a matching offline voice through Android's
+   text-to-speech settings. Return to the app and choose Voice in Settings.
+4. Set font size, contrast, dwell time and scanning interval with the user.
+5. Test Clear and Undo, Pause and Stop before enabling automatic input.
+
+The editor uses the installed Android keyboard/IME. The on-screen Chinese board
+contains common characters, not a complete Pinyin dictionary. Sinhala has separate
+letters/combining marks and a join control; validate typing and deletion with a
+native speaker. Phrase buttons insert text without unexpectedly speaking it.
+
+## Optional camera
+
+Use Vision. In Settings, choose an inference request (CPU first), then Camera:
+start/stop. The app requests camera permission only on this explicit action. No
+microphone or internet permission is requested. A denied permission leaves the
+other controls available. The front camera is used without a preview or recording.
+
+Start Calibrate and look at each of five targets for three seconds. Sampling starts
+after a settling interval. Lost tracking, too few samples, degenerate input or
+large fitting error rejects the calibration. Try the large phrase buttons first.
+Recalibrate after repositioning or activity recreation. Keyboard-page changes
+preserve calibration within the same viewport.
+
+Pause/Stop and alternate input remain available. Camera and speech stop when the
+activity leaves the foreground. Resume and restart the camera deliberately after
+returning. NNAPI requires the optional custom model described in the hardware guide.

docs/hardware.md

@@ -0,0 +1,59 @@
+# Hardware, models and performance
+
+## Capability matrix
+
+| Target | Implementation | Validation status |
+| --- | --- | --- |
+| ARMv7 Android | Lite native UI, speech and C++ core | Baseline APK compiled; device testing pending |
+| ARM64 Android | Lite or Vision | Baseline JNI/APK compiled; final Vision/device testing pending |
+| x86_64 Android | Lite or Vision/emulator | Baseline JNI/APK compiled; final Vision/emulator testing pending |
+| CPU inference | MediaPipe Face Landmarker | Source integrated; Android camera/model test pending |
+| GPU inference | Actual MediaPipe GPU delegate, CPU fallback | Depends on EGL/driver/runtime; hardware test pending |
+| GPU/NPU via NNAPI | Optional ONNX Runtime NNAPI request | Custom compatible model required; device assignment and accuracy unverified |
+| Qualcomm QNN / vendor NPU SDKs | No direct adapter bundled | Future work; do not advertise native vendor support |
+
+NNAPI is deprecated by Android and is an opt-in compatibility path. It can route
+operations to different devices or fall back. The app requests the CPU-disabled
+NNAPI flag and falls back to ONNX CPU when session creation/execution fails.
+An unavailable or incompatible ONNX asset falls back to the default MediaPipe CPU
+model when that model is installed. Inspect the active provider display; a selector
+label alone is not proof of accelerated execution. See
+[Android NNAPI](https://developer.android.com/ndk/guides/neuralnetworks) and
+[ONNX NNAPI](https://onnxruntime.ai/docs/execution-providers/NNAPI-ExecutionProvider.html).
+
+## Resource use
+
+Lite excludes both camera inference dependencies. Vision uses an ImageReader with
+two buffers and `acquireLatestImage`; it closes every acquired image. Preprocessing
+reuses a direct RGBA buffer and a Bitmap. C++ handles plane strides, rotation and
+mirroring. Inference is capped at 10 Hz (5 Hz under moderate-or-higher thermal
+pressure on supported Android versions). UI delivery replaces pending gaze work
+instead of building a queue. Samples older than 500 ms cannot select controls.
+
+The custom ONNX adapter reuses its tensor/pixel arrays, but resizing and runtime
+outputs may allocate. MediaPipe/runtime allocations are outside the small C++ core.
+Measure actual RSS, GC, inference latency, power and temperature; do not infer
+whole-app allocation behavior from the native helper's allocation-free design.
+
+## Custom ONNX contract
+
+Put a reviewed `gaze.onnx` and its lowercase SHA-256 text in `gaze.sha256` under
+`app/src/vision/assets/`. The adapter verifies the asset and bounds model size to
+128 MiB. It expects exactly one float32 `[1,3,H,W]` RGB input, values in `[0,1]`,
+with fixed dimensions between 16 and 1024, and one `[1,3]` output containing
+normalized x, y and confidence. Confidence below 0.7 invalidates the sample.
+The input has already been rotated upright and mirrored for the front camera.
+
+No compatible trained ONNX gaze model is bundled. Arbitrary face/iris models do
+not satisfy this contract. Review licenses, training population, preprocessing,
+quantization and accuracy before using a custom model. Keep its runtime libraries
+compatible with the device and record their versions.
+
+## Device acceptance
+
+Record build flavor, ABI, Android version, model checksum, runtime/delegate, actual
+capture resolution, median/p95 frame-to-input latency, peak memory and ten-minute
+thermal behavior. Test camera denial, disconnect, app backgrounding, rotation,
+tracking loss, closed eyes, screen resizing, calibration error and sustained speech.
+Include people with the intended access needs in evaluation. Physical-device
+performance and gaze accuracy have not been measured in this development session.

docs/languages.md

@@ -0,0 +1,26 @@
+# Language and voice expansion
+
+Packs live in `app/src/main/assets/languages/`. English, French, Simplified Chinese,
+Italian, Sinhala and Greek use the desktop schema version 1. The first five contain
+complete desktop UI keys; Greek retains some English fallback. Android adds
+`cameraControl`, `inference` and `cancelCalibration`. Translations require native
+speaker/user review before being marked reviewed.
+
+Copy the example in `examples/language-pack/es.json`, change its identity/locale,
+translate UI strings and phrases, and define keyboard rows and extra pages. The
+example is incomplete and is not shipped as Spanish support. Set `direction` for
+right-to-left packs. Keep files below 64 KiB and bounded keys/phrases as documented
+by `scripts/validate_packs.py`. Run that script and rebuild; asset discovery does
+not require Java edits. Do not place markup or executable actions inside keys.
+
+Use the system IME for unrestricted Chinese entry. The virtual board deliberately
+contains a small set of common Hanzi. Sinhala combining marks show a dotted-circle
+label while inserting only the actual mark; Join letters inserts a zero-width
+joiner. Deletion uses Android ICU grapheme boundaries, which must be checked on the
+minimum supported OS and with native conjunct examples.
+
+The pack's locale selects matching installed Android voices. Piper metadata is
+shared documentation for the desktop and is not a mobile neural model. Natural
+voice quality depends on the installed Android engine/voice. Sinhala requires a
+Sinhala-capable installed offline voice; none is bundled. Do not silently choose
+English or send text to a network voice when a requested voice is absent.