From 85af3316f1e3f088593e083947fee8c89ba5b4ee Mon Sep 17 00:00:00 2001 From: Gonzalo Tixilima Date: Thu, 9 Jul 2026 17:15:21 -0500 Subject: [PATCH 01/10] docs: add Generative LLM support for INFERENCE data event Ref: FLCRM-20635 --- .../data-events-inference.md | 64 +++++++++++++++++-- 1 file changed, 58 insertions(+), 6 deletions(-) diff --git a/docs/DATA EVENTS/data-events-reference/data-events-inference.md b/docs/DATA EVENTS/data-events-reference/data-events-inference.md index a66fa07..d22b012 100644 --- a/docs/DATA EVENTS/data-events-reference/data-events-inference.md +++ b/docs/DATA EVENTS/data-events-reference/data-events-inference.md @@ -13,25 +13,28 @@ next: ## Description -The `INFERENCE` function performs on-device machine learning inference using a specified model. It supports computer vision tasks (such as object detection) directly on the mobile device. +The `INFERENCE` function performs on-device machine learning or generative AI inference using a specified model. It supports computer vision tasks (such as object detection) and generative text tasks (such as summarization, assistant chats, or text classification) directly on the mobile device. **THIS FUNCTION WORKS ON MOBILE DEVICES, BUT NOT IN THE WEB RECORD EDITOR** > ⚠️ **Device Resource & Battery Usage Warning** > On-device model inference is highly resource-intensive and will consume substantial battery and memory. Requirements scale directly with the size of the loaded model. +> +> **Generative LLMs** are especially demanding; consider limiting them to modern flagship devices and/or documenting minimum device requirements (RAM/SoC) for your users. ## Execution Modes -The execution mode determines how the system runs the model. It supports two modes: +The execution mode determines how the system runs the model. It supports three modes: 1. **Vision ML**: Used for on-device computer vision tasks (such as object detection). -2. **Legacy Vision ML**: Legacy format. Migrate to the new format. **Support for ONNX is deprecated. Please upgrade to modern configurations.** +2. **Generative LLM**: Used for on-device generative text tasks (such as summarization, assistant chats, or text classification). +3. **Legacy Vision ML**: Legacy format. Migrate to the new format. **Support for ONNX is deprecated. Please upgrade to modern configurations.** > ⚠️ **Model Type Auto-Detection** > > The model type is determined **strictly by the file extension** of the model file passed to `options.model`. > -> Auto-detection is **not** determined or overridden by the parameters passed inside `options.config`. However, **the parameters in `options.config` must match the auto-detected model type** (e.g., providing a `size` parameter for a Vision ML model). +> Auto-detection is **not** determined or overridden by the parameters passed inside `options.config`. However, **the parameters in `options.config` must match the auto-detected model type** (e.g., providing a `size` parameter for a Vision ML model, or a `prompt` parameter for a Generative LLM). --- @@ -47,10 +50,11 @@ The system detects the correct machine learning engine to use based on the file | File Extension | Detected Model Type | Typical Use Cases | | :--- | :--- | :--- | | **`.tflite`** | **Vision ML** | Object detection | +| **`.gguf`**, **`.litertlm`**, **`.task`** | **Generative LLM** | Text generation, text summarization, assistant chats, text classification | ### Model Loading -If you bundle custom models as form reference files (e.g., `yolov5.tflite`), pass the exact filename (including extension) as the `options.model` string. +If you bundle custom models as form reference files (e.g., `yolov5.tflite` or `gemma.gguf`), pass the exact filename (including extension) as the `options.model` string. --- @@ -78,7 +82,26 @@ If you bundle custom models as form reference files (e.g., `yolov5.tflite`), pas --- -### Mode 2: Legacy Vision ML (ONNX - Deprecated) +### Mode 2: Generative LLM (for `.gguf`, `.litertlm`, and `.task` models) +*Used for running on-device generative AI large language models.* + +* `options` object: + * `photo_id` string (optional) - Omit for text-only LLM tasks. Provide the identifier of the photo to include for multimodal LLMs. + * `config` object (required) - Configuration for the generative text engine: + * `prompt` string (optional*) - The input instruction prompt. + * `systemPrompt` string (optional*) - System instructions to guide the model's behavior, tone, or role. + * `temperature` number (optional) - Controls randomness in generation. Must be non-negative. + * `topK` number (optional) - Restricts sampling to the top K most likely tokens. Must be a positive integer. + * `topP` number (optional) - Restricts sampling to cumulative probability P. Must be non-negative. + * `maxTokens` number (optional) - Maximum number of tokens to generate. Must be a positive integer. + * `contextSize` number (optional) - Context window size. Must be a positive integer. + * `stopTokens` array (optional) - Array of non-empty strings representing tokens that halt generation. + + * **Note:** At least one of `prompt` or `systemPrompt` must be provided. + +--- + +### Mode 3: Legacy Vision ML (ONNX - Deprecated) *Deprecated. Use Modern Vision ML config-based schemas instead.* * `options` object: @@ -99,6 +122,7 @@ If you bundle custom models as form reference files (e.g., `yolov5.tflite`), pas * `box` array - The bounding box coordinates `[x, y, width, height]`. * `score` number - The confidence score for the detection. * `class` number - The detected class index. + * **For Generative LLM**: A `result.outputs` object containing `result.outputs.text` (the generated text response) and a `result.modelType` of `'LLM'`. --- @@ -132,9 +156,37 @@ ON('add-photo', 'photos', (event) => { }); ``` +### Example 2: Modern Generative LLM +```javascript +// Use an on-device LLM to summarize notes when a record is saved +ON('save-record', () => { + const notes = VALUE('notes'); + if (!notes) return; + + INFERENCE({ + model: 'gemma-4-e2b.litertlm', + config: { + systemPrompt: 'You are an assistant. Summarize the user text in one short sentence.', + prompt: notes, + temperature: 0.7, + maxTokens: 100 + } + }, (error, result) => { + if (error) { + ALERT('Summarization failed: ' + error.message); + return; + } + + // Access the generated response text + SETVALUE('summary', result.outputs.text); + }); +}); +``` + ## Usage The `INFERENCE` function is typically used in applications requiring offline, local, or low-latency intelligence on-device: * **Object Detection**: Verify image contents, detect equipment, or perform safety audits offline without any internet connection. +* **On-Device LLMs**: Perform smart form calculations, generate field summaries, suggest translations, or parse unstructured user text instantly in the field. **Note:** This feature is only available with Elite and Enterprise plans. Check out [our plans page](https://www.fulcrumapp.com/pricing/) for more information. From a68a3ead1da961eb1ce671591b402777bab1fc9c Mon Sep 17 00:00:00 2001 From: Gonzalo Tixilima Date: Thu, 9 Jul 2026 17:46:52 -0500 Subject: [PATCH 02/10] docs: drop gguf support from Generative LLM mode Ref: FLCRM-20635 --- .../data-events-reference/data-events-inference.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/DATA EVENTS/data-events-reference/data-events-inference.md b/docs/DATA EVENTS/data-events-reference/data-events-inference.md index d22b012..d4dafd5 100644 --- a/docs/DATA EVENTS/data-events-reference/data-events-inference.md +++ b/docs/DATA EVENTS/data-events-reference/data-events-inference.md @@ -50,11 +50,11 @@ The system detects the correct machine learning engine to use based on the file | File Extension | Detected Model Type | Typical Use Cases | | :--- | :--- | :--- | | **`.tflite`** | **Vision ML** | Object detection | -| **`.gguf`**, **`.litertlm`**, **`.task`** | **Generative LLM** | Text generation, text summarization, assistant chats, text classification | +| **`.litertlm`**, **`.task`** | **Generative LLM** | Text generation, text summarization, assistant chats, text classification | ### Model Loading -If you bundle custom models as form reference files (e.g., `yolov5.tflite` or `gemma.gguf`), pass the exact filename (including extension) as the `options.model` string. +If you bundle custom models as form reference files (e.g., `yolov5.tflite` or `gemma.litertlm`), pass the exact filename (including extension) as the `options.model` string. --- @@ -82,7 +82,7 @@ If you bundle custom models as form reference files (e.g., `yolov5.tflite` or `g --- -### Mode 2: Generative LLM (for `.gguf`, `.litertlm`, and `.task` models) +### Mode 2: Generative LLM (for `.litertlm` and `.task` models) *Used for running on-device generative AI large language models.* * `options` object: From b3cf095ce774ba7141cebfff2fa4310557540819 Mon Sep 17 00:00:00 2001 From: Gonzalo Tixilima Date: Thu, 6 Aug 2026 13:50:28 -0500 Subject: [PATCH 03/10] docs: update INFERENCE LLM and labels documentation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../data-events-inference.md | 47 +++++++++---------- 1 file changed, 22 insertions(+), 25 deletions(-) diff --git a/docs/DATA EVENTS/data-events-reference/data-events-inference.md b/docs/DATA EVENTS/data-events-reference/data-events-inference.md index d4dafd5..0a58e5f 100644 --- a/docs/DATA EVENTS/data-events-reference/data-events-inference.md +++ b/docs/DATA EVENTS/data-events-reference/data-events-inference.md @@ -20,15 +20,16 @@ The `INFERENCE` function performs on-device machine learning or generative AI in > ⚠️ **Device Resource & Battery Usage Warning** > On-device model inference is highly resource-intensive and will consume substantial battery and memory. Requirements scale directly with the size of the loaded model. > -> **Generative LLMs** are especially demanding; consider limiting them to modern flagship devices and/or documenting minimum device requirements (RAM/SoC) for your users. +> **Generative LLMs and SLMs** are especially demanding; consider limiting them to modern flagship devices and/or documenting minimum device requirements (RAM/SoC) for your users. +> +> Generative LLM and SLM support is currently **beta**. Contact [product@fulcrumapp.com](mailto:product@fulcrumapp.com) if you are interested in testing it. ## Execution Modes -The execution mode determines how the system runs the model. It supports three modes: +The execution mode determines how the system runs the model. It supports two modes: 1. **Vision ML**: Used for on-device computer vision tasks (such as object detection). -2. **Generative LLM**: Used for on-device generative text tasks (such as summarization, assistant chats, or text classification). -3. **Legacy Vision ML**: Legacy format. Migrate to the new format. **Support for ONNX is deprecated. Please upgrade to modern configurations.** +2. **Generative LLM**: Used for on-device generative text tasks (such as summarization, assistant chats, or text classification). This includes supported small language models (SLMs). > ⚠️ **Model Type Auto-Detection** > @@ -50,11 +51,11 @@ The system detects the correct machine learning engine to use based on the file | File Extension | Detected Model Type | Typical Use Cases | | :--- | :--- | :--- | | **`.tflite`** | **Vision ML** | Object detection | -| **`.litertlm`**, **`.task`** | **Generative LLM** | Text generation, text summarization, assistant chats, text classification | +| **`.gguf`**, **`.litertlm`**, **`.task`** | **Generative LLM** | Text generation, text summarization, assistant chats, text classification | ### Model Loading -If you bundle custom models as form reference files (e.g., `yolov5.tflite` or `gemma.litertlm`), pass the exact filename (including extension) as the `options.model` string. +If you bundle custom models as form reference files (e.g., `yolov5.tflite` or `gemma.litertlm`), pass the exact filename (including extension) as the `options.model` string. Form reference files are resolved for offline use after synchronization. --- @@ -79,11 +80,18 @@ If you bundle custom models as form reference files (e.g., `yolov5.tflite` or `g * `inputType` string (optional) - The data type of the input layer. Either `'int8'` or `'float'`. * `mean` array (optional) - An array of exactly 3 numbers for normalizing the input data (e.g. `[0.485, 0.456, 0.406]`). * `std` array (optional) - An array of exactly 3 numbers for normalization standard deviations (e.g. `[0.229, 0.224, 0.225]`). + * `labels` array (optional) - Inline class labels. When provided, these take precedence over a `labels.txt` reference file, including when set to an empty array (`[]`). + +#### Class labels + +Vision ML models can use a `labels.txt` file supplied as a form reference file. The file must be UTF-8 text with one class label per line. The parser supports CRLF, LF, and CR line endings, trims surrounding whitespace, and ignores blank lines. The order of the remaining labels maps to the model's class indexes. + +The inline `config.labels` array takes precedence over `labels.txt`. If no inline labels are provided, the runtime uses `labels.txt` when it is available. A missing, unreadable, or empty file is non-fatal; inference continues without resolved labels. Resolved labels are returned in `result.labels`. --- -### Mode 2: Generative LLM (for `.litertlm` and `.task` models) -*Used for running on-device generative AI large language models.* +### Mode 2: Generative LLM (for `.gguf`, `.litertlm`, and `.task` models) +*Used for running on-device generative AI large language models and SLMs.* * `options` object: * `photo_id` string (optional) - Omit for text-only LLM tasks. Provide the identifier of the photo to include for multimodal LLMs. @@ -95,22 +103,8 @@ If you bundle custom models as form reference files (e.g., `yolov5.tflite` or `g * `topP` number (optional) - Restricts sampling to cumulative probability P. Must be non-negative. * `maxTokens` number (optional) - Maximum number of tokens to generate. Must be a positive integer. * `contextSize` number (optional) - Context window size. Must be a positive integer. - * `stopTokens` array (optional) - Array of non-empty strings representing tokens that halt generation. - - * **Note:** At least one of `prompt` or `systemPrompt` must be provided. ---- - -### Mode 3: Legacy Vision ML (ONNX - Deprecated) -*Deprecated. Use Modern Vision ML config-based schemas instead.* - -* `options` object: - * `photo_id` string (required) - * `size` number (required) - * `format` string (optional) - Either `'hwc'` or `'chw'`. - * `type` string (optional) - Either `'uint8'` or `'float'`. - * `mean` array (optional) - * `std` array (optional) + * **Note:** At least one of `prompt` or `systemPrompt` must be provided. --- @@ -122,6 +116,7 @@ If you bundle custom models as form reference files (e.g., `yolov5.tflite` or `g * `box` array - The bounding box coordinates `[x, y, width, height]`. * `score` number - The confidence score for the detection. * `class` number - The detected class index. + * **For Vision ML with labels**: A `result.labels` array containing the resolved class labels. The label at an index corresponds to the detection's `class` value. * **For Generative LLM**: A `result.outputs` object containing `result.outputs.text` (the generated text response) and a `result.modelType` of `'LLM'`. --- @@ -149,9 +144,11 @@ ON('add-photo', 'photos', (event) => { } const detections = result.outputs.detections; + const labels = result.labels || []; // Process detected objects... - SETVALUE('class_result', `Detected ${detections.length} object(s)!`); + const firstLabel = detections.length > 0 ? labels[detections[0].class] : null; + SETVALUE('class_result', firstLabel || `Detected ${detections.length} object(s)!`); }); }); ``` @@ -187,6 +184,6 @@ ON('save-record', () => { The `INFERENCE` function is typically used in applications requiring offline, local, or low-latency intelligence on-device: * **Object Detection**: Verify image contents, detect equipment, or perform safety audits offline without any internet connection. -* **On-Device LLMs**: Perform smart form calculations, generate field summaries, suggest translations, or parse unstructured user text instantly in the field. +* **On-Device LLMs and SLMs**: Perform smart form calculations, generate field summaries, suggest translations, or parse unstructured user text instantly in the field. This capability is beta; contact [product@fulcrumapp.com](mailto:product@fulcrumapp.com) if you are interested in testing it. **Note:** This feature is only available with Elite and Enterprise plans. Check out [our plans page](https://www.fulcrumapp.com/pricing/) for more information. From 68fcf8190091165699d90a73e4b98e7d1f7bad36 Mon Sep 17 00:00:00 2001 From: Gonzalo Tixilima Date: Thu, 6 Aug 2026 13:52:51 -0500 Subject: [PATCH 04/10] docs: use SLM terminology for inference models Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../data-events-inference.md | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/DATA EVENTS/data-events-reference/data-events-inference.md b/docs/DATA EVENTS/data-events-reference/data-events-inference.md index 0a58e5f..fb70386 100644 --- a/docs/DATA EVENTS/data-events-reference/data-events-inference.md +++ b/docs/DATA EVENTS/data-events-reference/data-events-inference.md @@ -20,22 +20,22 @@ The `INFERENCE` function performs on-device machine learning or generative AI in > ⚠️ **Device Resource & Battery Usage Warning** > On-device model inference is highly resource-intensive and will consume substantial battery and memory. Requirements scale directly with the size of the loaded model. > -> **Generative LLMs and SLMs** are especially demanding; consider limiting them to modern flagship devices and/or documenting minimum device requirements (RAM/SoC) for your users. +> **SLMs** are especially demanding; consider limiting them to modern flagship devices and/or documenting minimum device requirements (RAM/SoC) for your users. > -> Generative LLM and SLM support is currently **beta**. Contact [product@fulcrumapp.com](mailto:product@fulcrumapp.com) if you are interested in testing it. +> SLM support is currently **beta**. Contact [product@fulcrumapp.com](mailto:product@fulcrumapp.com) if you are interested in testing it. ## Execution Modes The execution mode determines how the system runs the model. It supports two modes: 1. **Vision ML**: Used for on-device computer vision tasks (such as object detection). -2. **Generative LLM**: Used for on-device generative text tasks (such as summarization, assistant chats, or text classification). This includes supported small language models (SLMs). +2. **SLM**: Used for on-device generative text tasks (such as summarization, assistant chats, or text classification). > ⚠️ **Model Type Auto-Detection** > > The model type is determined **strictly by the file extension** of the model file passed to `options.model`. > -> Auto-detection is **not** determined or overridden by the parameters passed inside `options.config`. However, **the parameters in `options.config` must match the auto-detected model type** (e.g., providing a `size` parameter for a Vision ML model, or a `prompt` parameter for a Generative LLM). +> Auto-detection is **not** determined or overridden by the parameters passed inside `options.config`. However, **the parameters in `options.config` must match the auto-detected model type** (e.g., providing a `size` parameter for a Vision ML model, or a `prompt` parameter for an SLM). --- @@ -51,7 +51,7 @@ The system detects the correct machine learning engine to use based on the file | File Extension | Detected Model Type | Typical Use Cases | | :--- | :--- | :--- | | **`.tflite`** | **Vision ML** | Object detection | -| **`.gguf`**, **`.litertlm`**, **`.task`** | **Generative LLM** | Text generation, text summarization, assistant chats, text classification | +| **`.gguf`**, **`.litertlm`**, **`.task`** | **SLM** | Text generation, text summarization, assistant chats, text classification | ### Model Loading @@ -90,11 +90,11 @@ The inline `config.labels` array takes precedence over `labels.txt`. If no inlin --- -### Mode 2: Generative LLM (for `.gguf`, `.litertlm`, and `.task` models) -*Used for running on-device generative AI large language models and SLMs.* +### Mode 2: SLM (for `.gguf`, `.litertlm`, and `.task` models) +*Used for running on-device generative text models.* * `options` object: - * `photo_id` string (optional) - Omit for text-only LLM tasks. Provide the identifier of the photo to include for multimodal LLMs. + * `photo_id` string (optional) - Omit for text-only SLM tasks. Provide the identifier of the photo to include for multimodal SLMs. * `config` object (required) - Configuration for the generative text engine: * `prompt` string (optional*) - The input instruction prompt. * `systemPrompt` string (optional*) - System instructions to guide the model's behavior, tone, or role. @@ -117,7 +117,7 @@ The inline `config.labels` array takes precedence over `labels.txt`. If no inlin * `score` number - The confidence score for the detection. * `class` number - The detected class index. * **For Vision ML with labels**: A `result.labels` array containing the resolved class labels. The label at an index corresponds to the detection's `class` value. - * **For Generative LLM**: A `result.outputs` object containing `result.outputs.text` (the generated text response) and a `result.modelType` of `'LLM'`. + * **For SLM**: A `result.outputs` object containing `result.outputs.text` (the generated text response). --- @@ -153,9 +153,9 @@ ON('add-photo', 'photos', (event) => { }); ``` -### Example 2: Modern Generative LLM +### Example 2: Modern SLM ```javascript -// Use an on-device LLM to summarize notes when a record is saved +// Use an on-device SLM to summarize notes when a record is saved ON('save-record', () => { const notes = VALUE('notes'); if (!notes) return; @@ -184,6 +184,6 @@ ON('save-record', () => { The `INFERENCE` function is typically used in applications requiring offline, local, or low-latency intelligence on-device: * **Object Detection**: Verify image contents, detect equipment, or perform safety audits offline without any internet connection. -* **On-Device LLMs and SLMs**: Perform smart form calculations, generate field summaries, suggest translations, or parse unstructured user text instantly in the field. This capability is beta; contact [product@fulcrumapp.com](mailto:product@fulcrumapp.com) if you are interested in testing it. +* **On-Device SLMs**: Perform smart form calculations, generate field summaries, suggest translations, or parse unstructured user text instantly in the field. This capability is beta; contact [product@fulcrumapp.com](mailto:product@fulcrumapp.com) if you are interested in testing it. **Note:** This feature is only available with Elite and Enterprise plans. Check out [our plans page](https://www.fulcrumapp.com/pricing/) for more information. From f46cea33d8a76a51ceaccc5768f4adae8887752b Mon Sep 17 00:00:00 2001 From: Gonzalo Tixilima Date: Thu, 6 Aug 2026 13:53:35 -0500 Subject: [PATCH 05/10] docs: limit SLM formats to LiteRT support Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../data-events-reference/data-events-inference.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/DATA EVENTS/data-events-reference/data-events-inference.md b/docs/DATA EVENTS/data-events-reference/data-events-inference.md index fb70386..6e673e0 100644 --- a/docs/DATA EVENTS/data-events-reference/data-events-inference.md +++ b/docs/DATA EVENTS/data-events-reference/data-events-inference.md @@ -51,7 +51,9 @@ The system detects the correct machine learning engine to use based on the file | File Extension | Detected Model Type | Typical Use Cases | | :--- | :--- | :--- | | **`.tflite`** | **Vision ML** | Object detection | -| **`.gguf`**, **`.litertlm`**, **`.task`** | **SLM** | Text generation, text summarization, assistant chats, text classification | +| **`.litertlm`**, **`.task`** | **SLM** | Text generation, text summarization, assistant chats, text classification | + +Public support is limited to LiteRT (`.tflite`) and LiteRT-LM (`.litertlm` and `.task`) model formats. ### Model Loading @@ -90,7 +92,7 @@ The inline `config.labels` array takes precedence over `labels.txt`. If no inlin --- -### Mode 2: SLM (for `.gguf`, `.litertlm`, and `.task` models) +### Mode 2: SLM (for `.litertlm` and `.task` models) *Used for running on-device generative text models.* * `options` object: From fde881efc63a6abfa73f6b257969005fbbe515de Mon Sep 17 00:00:00 2001 From: Gonzalo Tixilima Date: Thu, 6 Aug 2026 13:56:15 -0500 Subject: [PATCH 06/10] docs: clarify SLM sampling and output docs Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../data-events-reference/data-events-inference.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/DATA EVENTS/data-events-reference/data-events-inference.md b/docs/DATA EVENTS/data-events-reference/data-events-inference.md index 6e673e0..c9e803d 100644 --- a/docs/DATA EVENTS/data-events-reference/data-events-inference.md +++ b/docs/DATA EVENTS/data-events-reference/data-events-inference.md @@ -102,7 +102,7 @@ The inline `config.labels` array takes precedence over `labels.txt`. If no inlin * `systemPrompt` string (optional*) - System instructions to guide the model's behavior, tone, or role. * `temperature` number (optional) - Controls randomness in generation. Must be non-negative. * `topK` number (optional) - Restricts sampling to the top K most likely tokens. Must be a positive integer. - * `topP` number (optional) - Restricts sampling to cumulative probability P. Must be non-negative. + * `topP` number (optional) - Restricts sampling to cumulative probability P. Must be between 0 and 1. * `maxTokens` number (optional) - Maximum number of tokens to generate. Must be a positive integer. * `contextSize` number (optional) - Context window size. Must be a positive integer. @@ -119,7 +119,7 @@ The inline `config.labels` array takes precedence over `labels.txt`. If no inlin * `score` number - The confidence score for the detection. * `class` number - The detected class index. * **For Vision ML with labels**: A `result.labels` array containing the resolved class labels. The label at an index corresponds to the detection's `class` value. - * **For SLM**: A `result.outputs` object containing `result.outputs.text` (the generated text response). + * **For SLM**: The generated text is returned in the top-level `result.outputs.text` property. --- From f36e706a9dd4491b6c8950295dbb02d0c2fc9418 Mon Sep 17 00:00:00 2001 From: Gonzalo Tixilima Date: Thu, 6 Aug 2026 14:00:39 -0500 Subject: [PATCH 07/10] docs: remove runtime implementation details Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- docs/DATA EVENTS/data-events-reference/data-events-inference.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/DATA EVENTS/data-events-reference/data-events-inference.md b/docs/DATA EVENTS/data-events-reference/data-events-inference.md index c9e803d..5a2ef35 100644 --- a/docs/DATA EVENTS/data-events-reference/data-events-inference.md +++ b/docs/DATA EVENTS/data-events-reference/data-events-inference.md @@ -53,8 +53,6 @@ The system detects the correct machine learning engine to use based on the file | **`.tflite`** | **Vision ML** | Object detection | | **`.litertlm`**, **`.task`** | **SLM** | Text generation, text summarization, assistant chats, text classification | -Public support is limited to LiteRT (`.tflite`) and LiteRT-LM (`.litertlm` and `.task`) model formats. - ### Model Loading If you bundle custom models as form reference files (e.g., `yolov5.tflite` or `gemma.litertlm`), pass the exact filename (including extension) as the `options.model` string. Form reference files are resolved for offline use after synchronization. From cfaf20047ba65bd3e5f627edf4e9f426e37907e7 Mon Sep 17 00:00:00 2001 From: Gonzalo Tixilima Date: Thu, 6 Aug 2026 14:04:29 -0500 Subject: [PATCH 08/10] docs: clarify labels.txt reference file usage Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../data-events-inference.md | 21 ++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/docs/DATA EVENTS/data-events-reference/data-events-inference.md b/docs/DATA EVENTS/data-events-reference/data-events-inference.md index 5a2ef35..a33c1a8 100644 --- a/docs/DATA EVENTS/data-events-reference/data-events-inference.md +++ b/docs/DATA EVENTS/data-events-reference/data-events-inference.md @@ -57,6 +57,8 @@ The system detects the correct machine learning engine to use based on the file If you bundle custom models as form reference files (e.g., `yolov5.tflite` or `gemma.litertlm`), pass the exact filename (including extension) as the `options.model` string. Form reference files are resolved for offline use after synchronization. +For Vision ML, upload `labels.txt` as a separate form reference file alongside the `.tflite` model. The filename must be exactly `labels.txt`; do not pass it as `options.model`. When present, it is loaded automatically for that model. + --- ## Parameters @@ -84,7 +86,20 @@ If you bundle custom models as form reference files (e.g., `yolov5.tflite` or `g #### Class labels -Vision ML models can use a `labels.txt` file supplied as a form reference file. The file must be UTF-8 text with one class label per line. The parser supports CRLF, LF, and CR line endings, trims surrounding whitespace, and ignores blank lines. The order of the remaining labels maps to the model's class indexes. +For example, upload these two form reference files: + +* `fulcrum-pylon.tflite` +* `labels.txt` + +For example, the contents of `labels.txt` could be: + +```text +person +vehicle +equipment +``` + +The `labels.txt` file must be UTF-8 text with one class label per line. The parser supports CRLF, LF, and CR line endings, trims surrounding whitespace, and ignores blank lines. The order of the remaining labels maps to the model's class indexes. The inline `config.labels` array takes precedence over `labels.txt`. If no inline labels are provided, the runtime uses `labels.txt` when it is available. A missing, unreadable, or empty file is non-fatal; inference continues without resolved labels. Resolved labels are returned in `result.labels`. @@ -125,6 +140,10 @@ The inline `config.labels` array takes precedence over `labels.txt`. If no inlin ### Example 1: Vision ML ```javascript +// Form reference files uploaded to the form: +// - fulcrum-pylon.tflite +// - labels.txt +// // Perform on-device object detection when a photo is added ON('add-photo', 'photos', (event) => { INFERENCE({ From d0ac6d23a4938d11c47b15904291a7bbf583544c Mon Sep 17 00:00:00 2001 From: Gonzalo Tixilima Date: Thu, 6 Aug 2026 14:09:22 -0500 Subject: [PATCH 09/10] docs: document labels.txt as the supported label source Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../DATA EVENTS/data-events-reference/data-events-inference.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/DATA EVENTS/data-events-reference/data-events-inference.md b/docs/DATA EVENTS/data-events-reference/data-events-inference.md index a33c1a8..ef4eeae 100644 --- a/docs/DATA EVENTS/data-events-reference/data-events-inference.md +++ b/docs/DATA EVENTS/data-events-reference/data-events-inference.md @@ -82,7 +82,6 @@ For Vision ML, upload `labels.txt` as a separate form reference file alongside t * `inputType` string (optional) - The data type of the input layer. Either `'int8'` or `'float'`. * `mean` array (optional) - An array of exactly 3 numbers for normalizing the input data (e.g. `[0.485, 0.456, 0.406]`). * `std` array (optional) - An array of exactly 3 numbers for normalization standard deviations (e.g. `[0.229, 0.224, 0.225]`). - * `labels` array (optional) - Inline class labels. When provided, these take precedence over a `labels.txt` reference file, including when set to an empty array (`[]`). #### Class labels @@ -101,7 +100,7 @@ equipment The `labels.txt` file must be UTF-8 text with one class label per line. The parser supports CRLF, LF, and CR line endings, trims surrounding whitespace, and ignores blank lines. The order of the remaining labels maps to the model's class indexes. -The inline `config.labels` array takes precedence over `labels.txt`. If no inline labels are provided, the runtime uses `labels.txt` when it is available. A missing, unreadable, or empty file is non-fatal; inference continues without resolved labels. Resolved labels are returned in `result.labels`. +The runtime reads labels from `labels.txt` when it is available. A missing, unreadable, or empty file is non-fatal; inference continues without resolved labels. Resolved labels are returned in `result.labels`. --- From a6ef942c9f04fb69d1ba4dac8e7aa8b0eba96b8a Mon Sep 17 00:00:00 2001 From: Gonzalo Tixilima Date: Thu, 6 Aug 2026 14:11:34 -0500 Subject: [PATCH 10/10] docs: simplify labels.txt inference example Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../data-events-reference/data-events-inference.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/DATA EVENTS/data-events-reference/data-events-inference.md b/docs/DATA EVENTS/data-events-reference/data-events-inference.md index ef4eeae..7f1a1a5 100644 --- a/docs/DATA EVENTS/data-events-reference/data-events-inference.md +++ b/docs/DATA EVENTS/data-events-reference/data-events-inference.md @@ -162,11 +162,9 @@ ON('add-photo', 'photos', (event) => { } const detections = result.outputs.detections; - const labels = result.labels || []; - // Process detected objects... - const firstLabel = detections.length > 0 ? labels[detections[0].class] : null; - SETVALUE('class_result', firstLabel || `Detected ${detections.length} object(s)!`); + // Detection class indexes correspond to the entries in labels.txt. + SETVALUE('class_result', `Detected ${detections.length} object(s)!`); }); }); ```