docs: Deprecate ONNX legacy config and modernize Vision ML for INFERENCE

[javoweb-fc]

Description

This PR updates the developer reference documentation for the INFERENCE data event function to align with the latest on-device Vision ML runtime capabilities and contracts, while keeping underlying backend engines transparent to developers.

Key Changes

1. User-Friendly Abstraction: Removed direct references to underlying engine internals in user-facing text.
2. Strict File-Extension-Based Auto-Detection: Documented that model type auto-detection is determined strictly by the file extension (.tflite for Vision ML). Clarified that config-level parameters do not affect or override detection, but must still match the model type.
3. Streamlined Model Loading (Form Reference Files Only): Removed references to the central Model Catalog, catalog IDs, and resolution priority. Documentation now focuses exclusively on loading custom models via Form Reference Files.
4. ONNX Support Deprecation: Marked the legacy/ONNX-based configuration as Deprecated and added guidance to update to the newer config-based model definitions.
5. Execution Modes: Documented two paths: Vision ML and Legacy Vision ML (Deprecated).
6. Performance and Device Warnings: Added a prominent callout warning about computational overhead, device memory footprint, and battery drain.
7. Updated Examples: Updated the standard Vision ML example to use the modern config syntax.

JIRA / Issue Reference

Ref: FLCRM-20635

Notes

This is the first of two stacked PRs splitting the original #76. Part 2 (Generative LLM support) is stacked on top of this branch and targets it directly.

[Copilot]

Pull request overview

Note

Copilot couldn't run its full agentic review because no GitHub Actions runner was available. Make sure your repository has a runner available to run Copilot's review, or add a copilot-setup-steps.yml file specifying one with the runs-on attribute. See the docs for more details.

Updates the INFERENCE data event documentation to reflect the modern on-device Vision ML runtime, deprecate the legacy ONNX-based mode, and clarify model loading/detection behavior for developers.

Changes:

• Reframes INFERENCE as on-device Vision ML with clearer execution modes and a prominent device resource warning.
• Documents strict file-extension-based model auto-detection and focuses model loading on form reference files.
• Deprecates the legacy ONNX path and refreshes the primary example to the modern config syntax.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.

[HenryTabima]

LGTM

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 8 comments.

[pedrodotavila]

Android INFERENCE() usage notes (from fulcrum-android / FLCRM-21207)

Thanks for modernizing this page. From the current Android implementation, a few usage details and doc gaps stood out that would help form authors avoid incorrect assumptions.

What Android currently supports (Vision ML)

Request shape (modern nested config):

INFERENCE({
  model: 'my-model.tflite',   // form reference filename (or modelName)
  photo_id: event.value.id,
  form_id: FORMID(),          // optional; defaults to current form context
  form_name: '...',           // optional alternative to form_id
  config: {
    size: 640,                // required for ML; must be > 0
    format: 'chw' | 'hwc',    // optional
    inputType: 'float' | 'int8', // optional (legacy alias: type)
    mean: [0.485, 0.456, 0.406], // optional; exactly 3 numbers if provided
    std:  [0.229, 0.224, 0.225], // optional; exactly 3 numbers if provided
    labels: ['class_a', 'class_b'] // optional inline labels
  }
}, callback)

Legacy flat options (size / format / type at top level) still parse on Android, but the nested config form is preferred.

Gating / prerequisites:

• Plan attribute for data-event inference must be enabled.
• Model is resolved from form reference files (and KMP model storage when available).
• Photo must exist locally (photo_id + available photo attachment).

---

Gaps / inaccuracies vs this PR’s docs

1) Labels are not documented (important for FLCRM-21207 / 21208)

Android now resolves class labels as:

1. Inline config.labels (non-null wins), else
2. Form reference file labels.txt (newline-separated, one label per line; blank lines ignored), else
3. No labels (class indices only)

Missing/unreadable labels.txt is non-fatal — inference still succeeds.

When labels are resolved, Android includes them on the top-level result:

result.labels // string[] | undefined
// e.g. ["pole", "transformer", "crossarm"]
// index i maps to detection class index i

Suggested docs additions:

• Document config.labels
• Document companion reference file labels.txt
• Document precedence + non-fatal missing file behavior
• Show mapping example: result.labels?.[detectionClassIndex]

2) Result payload shape does not match result.outputs.detections

The example/callback docs currently describe:

result.outputs.detections // [{ box, score, class }, ...]

On Android Vision ML today, the payload is closer to:

{
  original: { width, height, orientation },
  resized:  { width, height, orientation },
  outputs: {
    output_0: { shape: [1, n, 4], value: FloatArray /* boxes */ },
    output_1: { shape: [1, n],    value: FloatArray /* scores */ },
    output_2: { shape: [1, n],    value: FloatArray /* class indices */ },
    output_3: { shape: [1],       value: FloatArray /* count */ }
  },
  labels?: string[] // when resolved (inline or labels.txt)
}

Box values are [x1, y1, x2, y2] in resized-image space (not [x, y, width, height] as currently documented).

A more accurate consumer sketch:

const boxes   = result.outputs.output_0.value; // length n*4
const scores  = result.outputs.output_1.value; // length n
const classes = result.outputs.output_2.value; // length n
const count   = result.outputs.output_3.value[0];
const labels  = result.labels; // optional

for (let i = 0; i < count; i++) {
  const x1 = boxes[i * 4], y1 = boxes[i * 4 + 1];
  const x2 = boxes[i * 4 + 2], y2 = boxes[i * 4 + 3];
  const score = scores[i];
  const classIndex = classes[i];
  const className = labels?.[classIndex] ?? String(classIndex);
}

If outputs.detections is the intended public contract going forward, that should be called out as a planned/normalized shape (and platforms should converge). As written, the example will not work against current Android output.

3) Supported extensions / engines

Docs currently list only .tflite. Android still accepts .ort via the legacy ONNX path (deprecated, but present). Worth either:

• documenting .ort under Legacy Vision ML, or
• explicitly stating Android still accepts .ort temporarily while deprecating it.

Also: Android accepts both inputType and legacy type for dtype (float / int8). Legacy docs mention uint8; Android’s modern path uses int8/float.

4) Auto-detection wording is slightly stronger than Android behavior

Docs say model type is determined strictly by file extension and config does not override detection. On Android, request config can influence routing when metadata is ambiguous/wrong:

• presence of ML signals (config.size / photo_id) vs LLM signals (prompt / systemPrompt) can override stored model-type metadata.

For this Vision-only docs PR that may be fine, but “strictly by extension only” is not fully accurate on Android once LLM modes are in play (stacked PR).

5) Defaults

Docs previously implied defaults for size/format/type/mean/std. Android currently treats size as required for ML (errors if missing/≤0). format defaults to HWC when not 'chw'; dtype defaults to float when not 'int8'. Mean/std are optional and only applied when length === 3.

6) Plan / availability note

Besides Elite/Enterprise, Android also requires the plan data-event inference flag to be enabled; otherwise INFERENCE() returns a disabled data-event error.

---

Suggested minimal doc updates for this PR

1. Add Labels section (config.labels + labels.txt + precedence).
2. Fix callback/result docs to match actual output_0..3 tensors or document a normalized detections API as future work.
3. Correct box format to [x1,y1,x2,y2] if documenting current Android tensors.
4. Mention optional result.labels and class-index mapping.
5. Clarify .ort deprecation status and inputType vs legacy type.
6. Update the example to either consume output_* correctly or note that detections is illustrative/future.

Happy to align Android packaging with a cleaner result.outputs.detections + result.labels contract if product wants that as the public API.

Refs:

• Android PR: https://github.com/fulcrumapp/fulcrum-android/pull/4009
• Jira: https://fulcrumapp.atlassian.net/browse/FLCRM-21207
• KMP labels support: https://fulcrumapp.atlassian.net/browse/FLCRM-21208

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 4 comments.