diff --git a/docs/modules/automation-testing/nav.adoc b/docs/modules/automation-testing/nav.adoc index 6c1eecb15..59e80204b 100644 --- a/docs/modules/automation-testing/nav.adoc +++ b/docs/modules/automation-testing/nav.adoc @@ -56,6 +56,5 @@ * Appium AI ** xref:appium-ai/overview-appium-ai.adoc[] -** xref:appium-ai/user-guide-appium-ai.adoc[] ** xref:appium-ai/override-llm.adoc[] ** xref:appium-ai/extend-custom-locator-strategy-for-nll.adoc[] \ No newline at end of file diff --git a/docs/modules/automation-testing/pages/appium-ai/override-llm.adoc b/docs/modules/automation-testing/pages/appium-ai/override-llm.adoc index 5188bfc5d..f48557e9a 100644 --- a/docs/modules/automation-testing/pages/appium-ai/override-llm.adoc +++ b/docs/modules/automation-testing/pages/appium-ai/override-llm.adoc @@ -1,105 +1,40 @@ = Override Appium AI settings per session :description: Use Appium desired capabilities to override the default Appium AI provider and locator settings for a session. -Appium AI uses an LLM provider configured by your Kobiton administrator by default. You can override those settings for a single Appium session by passing `kobiton:*` desired capabilities. +By default, Appium AI uses an LLM provider configured by your Kobiton administrator. You can override those settings for a single Appium session by passing `kobiton:*` desired capabilities. -`kobiton:locatorStrategy` selects how Appium AI resolves elements. It is not an -LLM provider setting, but it is set the same way. See the -xref:appium-ai/user-guide-appium-ai.adoc[Appium AI user guide] for guidance on -choosing a strategy. +See xref:capabilities/available-capabilities.adoc[Available capabilities] for the full reference of the `kobiton:llm*`, `kobiton:visionCoordSpace`, and `kobiton:locatorStrategy` capabilities. `kobiton:locatorStrategy` selects how Appium AI resolves elements; see xref:appium-ai/overview-appium-ai.adoc[Appium AI: Natural language locators] for guidance on choosing a strategy. Use this to: -- Route AI requests through your own OpenAI, Azure OpenAI, or local LLM endpoint to meet compliance or data residency requirements. +- Route AI requests through your own OpenAI, Azure OpenAI, Anthropic Claude, or local LLM endpoint to meet compliance or data residency requirements. - Use a different model without requiring an administrator to update the default configuration. - Run AI features against a local LLM for air-gapped or cost-sensitive environments. == Prerequisites -- An administrator has enabled the Appium AI feature by configuring the `dc.ini` file on each Mac mini in the environment. - +- An administrator has enabled the Appium AI feature in your environment. - You have an API key or reachable local endpoint for the LLM provider you want to use. -== Available capabilities - -Capability names and values are case-insensitive. - -If a capability is omitted, the session uses the default value configured by your Kobiton admin. - -[cols="2,4,2,2", options="header"] -|=== -| Capability | Purpose | Overrides | Default - -| `kobiton:llmBaseUrl` -| LLM provider base URL. -| `LlmClient.BaseUrl` -| `https://api.openai.com/v1` - -| `kobiton:llmApiKey` -| LLM provider API key. Required in either `dc.ini` or capabilities. -| `LlmClient.ApiKey` -| (none) - -| `kobiton:llmApiFormat` -| API protocol format. -| `LlmClient.ApiFormat` -| `OpenAiResponses` - -| `kobiton:llmApiVersion` -| Required only when using Azure OpenAI formats. -| `LlmClient.ApiVersion` -| `2024-02-01` - -| `kobiton:llmTextModel` -| Model used for UI tree-based operations (XPath/CSS generation). For Azure, pass the deployment name. -| `LlmClient.TextModel` -| `gpt-5.2` - -| `kobiton:llmVisionModel` -| Model used for vision-based operations (screenshot analysis). Must support image input. For Azure, pass the deployment name. -| `LlmClient.VisionModel` -| `gpt-5.2` - -| `kobiton:llmTextExtraBody` -| Optional. JSON object passed verbatim into the request body for text model calls. Use to send model-specific parameters such as `temperature`. -| `LlmClient.TextExtraBody` -| (none) - -| `kobiton:llmVisionExtraBody` -| Optional. JSON object passed verbatim into the request body for vision model calls. Use to send model-specific parameters such as `temperature`. -| `LlmClient.VisionExtraBody` -| (none) - -| `kobiton:locatorStrategy` -| Locator strategy used to resolve natural language locators. Pass `visionvirtual` per session for canvas-based apps. Accepts `uitree` and `visionvirtual`. -| `NaturalLanguageLocator.LocatorStrategy` -| `uitree` -|=== - -`kobiton:llmApiFormat` supports the following values: - -- `OpenAiResponses` - -- `OpenAiChatCompletions` - -- `AzureOpenAiV1ChatCompletions` - -- `AzureOpenAiChatCompletions` - == Examples -=== OpenAI (default endpoint, customer API key) +=== OpenAI (default endpoint, your API key) -``` +[source,python] +.python +---- desired_capabilities = { 'kobiton:llmApiKey': '', - 'kobiton:llmTextModel': 'gpt-4o', - 'kobiton:llmVisionModel': 'gpt-4o', + 'kobiton:llmTextModel': 'gpt-5.6-terra', + 'kobiton:llmVisionModel': 'gpt-5.6-terra', } -``` +---- === Azure OpenAI -``` + +[source,python] +.python +---- desired_capabilities = { 'kobiton:llmBaseUrl': 'https://my-resource.openai.azure.com', 'kobiton:llmApiKey': '', @@ -108,13 +43,55 @@ desired_capabilities = { 'kobiton:llmTextModel': 'my-gpt4o-deployment', 'kobiton:llmVisionModel': 'my-gpt4o-deployment', } -``` +---- For Azure, `llmTextModel` and `llmVisionModel` use the Azure deployment name, not the underlying model name. +=== Anthropic Claude + +[source,python] +.python +---- +desired_capabilities = { + 'kobiton:llmBaseUrl': 'https://api.anthropic.com/v1', + 'kobiton:llmApiKey': '', + 'kobiton:llmApiFormat': 'AnthropicMessages', + 'kobiton:llmTextModel': 'claude-opus-4-8', + 'kobiton:llmVisionModel': 'claude-opus-4-8', +} +---- + +=== Kobiton-hosted LLM + +The Kobiton-hosted LLM runs at `https://llm.kobiton.com/v1` and needs no third-party LLM account. It is available on Private and Local devices where the Mac mini host can reach `llm.kobiton.com`. It is unavailable on an air-gapped host or where the domain is blocked. + +The hosted model serves vision only. Set `kobiton:locatorStrategy` to `visionvirtual` for sessions that use it. + +// REVIEWER NOTE: The `` placeholder is unresolved. Admin-side +// configuration falls back to `Kobiton.TenantApiKey` (see admin-docs#217), but +// KOB-54645 states that server-side keys are not sent to session-specified endpoints. +// What value a customer passes for a session-level override to the hosted LLM needs +// confirmation from engineering before publication. + +[source,python] +.python +---- +desired_capabilities = { + 'kobiton:llmBaseUrl': 'https://llm.kobiton.com/v1', + 'kobiton:llmApiKey': '', + 'kobiton:llmApiFormat': 'KobitonChatCompletions', + 'kobiton:llmVisionModel': 'qwen25-vl-7b', + 'kobiton:locatorStrategy': 'visionvirtual', +} +---- + +`qwen25-vl-7b` is the only model this endpoint serves. Any other value in `kobiton:llmVisionModel` returns an error. If your administrator has already set the Kobiton-hosted LLM as the host-wide default, sessions do not need to set these capabilities. + === Local LLM endpoint (Ollama) -``` +[source,python] +.python +---- desired_capabilities = { 'kobiton:llmBaseUrl': 'http://localhost:11434', 'kobiton:llmApiKey': '', @@ -122,11 +99,13 @@ desired_capabilities = { 'kobiton:llmTextModel': '', 'kobiton:llmVisionModel': '', } -``` +---- [NOTE] ==== -Ollama exposes an OpenAI-compatible chat completions API, so `kobiton:llmApiFormat` must be set to `OpenAiChatCompletions`. +Ollama exposes a native API and an OpenAI-compatible chat completions API. Set `kobiton:llmApiFormat` to `OllamaChat` for the native endpoint or `OpenAiChatCompletions` for the compatibility endpoint. + +`kobiton:llmApiKey` is required even for local Ollama endpoints that do not enforce authentication. Pass any value, such as `ollama`. ==== Local endpoints are resolved from the host running the Appium session. For example, `localhost` refers to the Mac mini running the session. @@ -134,6 +113,6 @@ Local endpoints are resolved from the host running the Appium session. For examp == Notes - Capabilities apply only to the session that passed them; other sessions continue to use the defaults. +- A session that sets `kobiton:llmBaseUrl` must also set `kobiton:llmApiKey`. The default server-side key is not sent to a session-specified endpoint. Existing scripts that set `kobiton:llmBaseUrl` without `kobiton:llmApiKey` fail on the first LLM request. - `kobiton:llmApiFormat` must match the provider used by `kobiton:llmBaseUrl`. - Vision-based features require a model that accepts image input such as screenshots. -- Pass `kobiton:locatorStrategy` per session instead of changing the `dc.ini` default. A `dc.ini` default of `visionvirtual` applies to every session on that host, including sessions that test apps exposing a view hierarchy. diff --git a/docs/modules/automation-testing/pages/appium-ai/overview-appium-ai.adoc b/docs/modules/automation-testing/pages/appium-ai/overview-appium-ai.adoc index c090330eb..0545aedd0 100644 --- a/docs/modules/automation-testing/pages/appium-ai/overview-appium-ai.adoc +++ b/docs/modules/automation-testing/pages/appium-ai/overview-appium-ai.adoc @@ -4,40 +4,26 @@ Appium AI enables automation scripts to locate UI elements using natural language instead of traditional selectors such as XPath or accessibility IDs. -This allows test authors to describe elements in plain language while maintaining compatibility with existing Appium workflows. - -This approach improves test resilience in environments where selectors are brittle, unavailable, or frequently changing. +Test authors describe elements in plain language while keeping existing Appium workflows. This improves test resilience in environments where selectors are brittle, unavailable, or frequently changing. == Availability -Appium AI (Natural Language Locators) is currently available in Beta. - -This feature requires access to a compatible LLM provider. Kobiton supports OpenAI, Azure OpenAI, and self-hosted OpenAI-compatible endpoints such as Ollama. You must provide your own API credentials or a reachable local endpoint. - -NOTE: Access to a supported LLM API is required. A ChatGPT subscription alone does not enable this feature. +Appium AI (Natural Language Locators) is in Beta. -This feature is available for private device deployments *only*. Appium AI is not available for Public Cloud devices. +This feature requires access to a compatible LLM provider. Kobiton supports OpenAI, Azure OpenAI, Anthropic Claude, self-hosted OpenAI-compatible endpoints such as Ollama, and the Kobiton-hosted LLM. -Additional configuration may be required in deviceConnect. For setup assistance, contact Kobiton Support. +Most providers require your own API credentials or a reachable local endpoint. The Kobiton-hosted LLM is the exception: it needs no third-party account and is available where the Mac mini host can reach `llm.kobiton.com`. It is unavailable on an air-gapped host or where the domain is blocked. -== How element location works in Appium +NOTE: A ChatGPT subscription alone does not enable this feature. OpenAI, Azure OpenAI, and Anthropic Claude require API access. -UI automation typically depends on selectors such as XPath, accessibility IDs, or view hierarchies to locate elements. +This feature is available on Private and Local devices. Appium AI is not available on Public Cloud devices. -These selectors are often brittle and tightly coupled to the structure of the UI: - -* Small UI changes can break existing tests -* Dynamic interfaces may not expose stable identifiers -* Some applications (such as canvas-based UIs) provide little or no metadata for automation - -This increases maintenance effort and reduces the reliability of automated tests over time. +Your lab administrator configures the LLM provider and model defaults for your environment. Contact them to change the host-wide setup. == Natural language locators Appium AI extends existing Appium workflows by allowing `findElement(...)` to accept natural language descriptions instead of traditional selectors. -Instead of referencing elements by XPath or accessibility ID, you can describe the target element using plain language. - [source,python] .python ---- @@ -45,57 +31,104 @@ element = driver.find_element("natural", "The login button at the bottom") element.click() ---- -IMPORTANT: For Appium java-client 10.x and above, the custom locator strategy needs to be extended to use Appium AI. See xref:automation-testing:appium-ai/extend-custom-locator-strategy-for-nll.adoc[this guide,window=read-later] for instructions. +IMPORTANT: For Appium java-client 10.x and above, extend the custom locator strategy to use Appium AI. See xref:automation-testing:appium-ai/extend-custom-locator-strategy-for-nll.adoc[this guide,window=read-later] for instructions. Appium AI interprets the description, identifies the most relevant UI element, and returns an interactable element for use in the test. Natural language locators identify elements only. Actions such as `click()`, `send++_++keys()`, or assertions must still be performed explicitly in the test script. -This approach works within existing Appium scripts and does not require changes to client libraries or test structure. - === How Appium AI resolves elements -Appium AI resolves elements with one of two locator strategies. You select the -strategy per session with the `kobiton:locatorStrategy` capability. +Appium AI supports two locator strategies: -* *`uitree`:* Reads the view hierarchy or accessibility attributes exposed by the -application and returns a matching element by XPath or CSS selector. This is the -default strategy. +[cols="1,3", options="header"] +|=== +| Strategy | Behavior -* *`visionvirtual`:* Analyzes a screenshot of the current view with a -vision-capable LLM and returns a virtual element representing the matched region. -This enables element location on canvas-based UIs, games, and other -graphics-heavy apps that were previously unreachable with traditional selectors. +| `uitree` +| Reads the view hierarchy or accessibility attributes exposed by the application and returns a matching element by XPath or CSS selector. This is the default strategy. -The strategy is fixed for the session. For instructions on setting it, see the -xref:appium-ai/user-guide-appium-ai.adoc[Appium AI user guide]. +| `visionvirtual` +| Analyzes a screenshot of the current view with a vision-capable LLM and returns a virtual element representing the matched region. Use this for canvas-based apps, games, and other graphics-heavy apps that lack UI metadata. +|=== -== What this feature supports +Appium AI resolves the strategy once when the session starts. You cannot switch strategies during a test run, and a single session cannot use both. To exercise both strategies, run separate sessions. -Appium AI enables natural language-based element location for the following conditions: +== Before you begin -* Using natural language descriptions with `findElement(...)` to locate UI elements -* Native and web applications that expose a view hierarchy, using the `uitree` -strategy -* Canvas-based applications, games, and other graphics-heavy apps that lack UI -metadata, using the `visionvirtual` strategy -* Integration with existing Appium scripts without requiring structural changes +* Your environment is configured to support natural language locators. If unsure, contact your lab administrator or Kobiton Support. +* You are familiar with basic Appium commands such as `findElement(...)`. +* You are using Private and Local devices (natural language locators are not available on Public Cloud devices). -== Use cases +== Choose a locator strategy -Use natural language locators in scenarios where traditional selectors are difficult to maintain or unreliable. +Set the strategy per session with the `kobiton:locatorStrategy` capability: -This approach is especially useful when: +[source,python] +.python +---- +capabilities = { + # ... + 'kobiton:locatorStrategy': 'visionvirtual', +} +---- -* UI elements change frequently, causing selectors to break +A capability passed in the session takes precedence over the host-wide default your administrator set. -* Applications use dynamic layouts or rendering patterns +`visionvirtual` uses the vision model already configured for Appium AI. You do not need to set `kobiton:llmVisionModel` to use it. To route vision requests through a different model or provider, see xref:appium-ai/override-llm.adoc[Override Appium AI settings per session]. -* Readability and maintainability of test scripts are a priority +IMPORTANT: Appium AI does not detect canvas-based apps. To use vision-based resolution, set `kobiton:locatorStrategy` to `visionvirtual`. -* Element identification is possible through visible labels or context +== Use a natural language locator -Natural language locators can help reduce maintenance overhead and improve test clarity in these situations. +Pass a descriptive string into `findElement(...)` using the `natural` locator strategy. + +[source,python] +.python +---- +element = driver.find_element("natural", "The login button at the bottom") +element.click() +---- + +[source,java] +.java +---- +WebElement el = driver.findElement(ByNatural.natural("the Accessibility button")); +el.click(); +---- + +NOTE: The Java example uses the custom `ByNatural` class to extend the default locator strategy. See xref:automation-testing:appium-ai/extend-custom-locator-strategy-for-nll.adoc[this guide,window=read-later] for instructions. + +== Write effective descriptions + +Use clear, specific descriptions to help identify the correct element. + +Good examples: + +---- +driver.findElement(ByNatural.natural("the Accessibility button")); +driver.findElement(ByNatural.natural("the Username field")); +driver.findElement(ByNatural.natural("the City dropdown list")); +---- + +Less effective examples: + +---- +driver.findElement(ByNatural.natural("Click button")); +driver.findElement(ByNatural.natural("Select item")); +---- + +When multiple similar elements are present, include additional context such as position, label, or surrounding UI to improve accuracy. + +== Expected behavior + +Appium AI evaluates the description and returns the most relevant matching element using the locator strategy set for the session. `uitree` returns a selector resolved from the view hierarchy. `visionvirtual` returns a virtual element mapped to coordinates on the screenshot. + +Results may not be deterministic. If multiple elements match the description, refine the description to improve accuracy. + +When the `visionvirtual` strategy cannot find the element on the current screen, it returns a not-found result rather than a best-guess match. Scroll an off-screen element into view, or refine the description, if a visible element is missed. + +Natural language locators work alongside traditional selector strategies and are best used in combination with them, rather than as a complete replacement. == Limitations and considerations @@ -104,31 +137,33 @@ Natural language locators are not intended to replace all selector strategies. U This approach may not be suitable when: * Stable and reliable selectors already exist - * Exact element matching is required (for example, when multiple similar elements are present) - * The environment does not support natural language locators (such as Basic Appium or unsupported configurations) Current limitations: -* Natural language locators identify elements only. They do not perform actions such as clicking or typing +* Natural language locators identify elements only. They do not perform actions such as clicking or typing. +* Script generation and full AI-driven test creation are not supported. +* Automatic fallback from traditional locators to natural language is not supported. + +Element identification quality depends on the locator strategy set for the session. `uitree` depends on the view hierarchy exposed by the application, and `visionvirtual` depends on the visual clarity and layout of the current screen. Clear, specific descriptions improve results for both strategies. -* Script generation and full AI-driven test creation are not supported +== Troubleshooting -* Automatic fallback from traditional locators to natural language is not supported +If an element is not found or the wrong element is returned: -* A session uses a single locator strategy. Switching strategies during a test -run is not supported +* If the app renders its UI on a canvas, set `kobiton:locatorStrategy` to `visionvirtual`. The default `uitree` strategy cannot locate elements in an app that exposes no view hierarchy. +* Use more specific language in the description. +* Include additional context such as position, labels, or surrounding elements. +* Verify that the UI element is visible and accessible in the view hierarchy. +* When using `uitree`, confirm that the application exposes sufficient UI metadata (view hierarchy or accessibility attributes) for element identification. +* When using `visionvirtual`, ensure the target element is clearly visible on the current screen. +* If your script sets `kobiton:llmBaseUrl`, confirm it also sets `kobiton:llmApiKey`. Sessions that override the base URL without providing a key fail on the first LLM request. -* Canvas-based apps are not detected automatically. Set -`kobiton:locatorStrategy` to `visionvirtual` for these apps +If natural language locators are not working as expected, contact your lab administrator to confirm that the LLM provider in use (OpenAI, Azure OpenAI, Anthropic Claude, Ollama, or the Kobiton-hosted LLM) is configured correctly. -Element identification quality depends on the locator strategy set for the -session. `uitree` depends on the view hierarchy exposed by the application, and -`visionvirtual` depends on the visual clarity and layout of the current screen. -Clear, specific descriptions improve results for both strategies. +If issues persist, contact Kobiton Support. == Next steps -To start using natural language locators in your tests, see the -xref:appium-ai/user-guide-appium-ai.adoc[Appium AI user guide] for setup instructions and examples. +To override Appium AI provider or model settings for a specific session, see xref:appium-ai/override-llm.adoc[Override Appium AI settings per session]. diff --git a/docs/modules/automation-testing/pages/appium-ai/user-guide-appium-ai.adoc b/docs/modules/automation-testing/pages/appium-ai/user-guide-appium-ai.adoc deleted file mode 100644 index 43c201d32..000000000 --- a/docs/modules/automation-testing/pages/appium-ai/user-guide-appium-ai.adoc +++ /dev/null @@ -1,150 +0,0 @@ -= Use natural language locators in Appium AI - -== Before you begin - -Ensure the following: - -* Your environment is configured to support natural language locators. If you are unsure, contact your lab administrator or Kobiton Support. -* You are familiar with basic Appium commands such as `findElement(...)` -* You are using a private device deployment (natural language locators are not available for Public Cloud devices) - -== Choose a locator strategy - -Appium AI resolves natural language locators with one of two strategies. Select -the strategy before the session starts. - -[cols="1,3", options="header"] -|=== -| Strategy | Behavior - -| `uitree` -| Reads the view hierarchy, sends the filtered metadata to a text model, and -returns a validated XPath or CSS selector. This is the default. - -| `visionvirtual` -| Captures a screenshot, analyzes it with a vision model, and returns a virtual -element for the matched region. Use this strategy for canvas-based apps, games, -and other apps that expose no usable view hierarchy. -|=== - -Set the strategy with the `kobiton:locatorStrategy` capability: - -[source,python] -.python ----- -capabilities = { - # ... - 'kobiton:locatorStrategy': 'visionvirtual', -} ----- - -Set `visionvirtual` per session with the capability. A lab administrator can also -set a host-wide default with `NaturalLanguageLocator.LocatorStrategy` in `dc.ini`, -but leave that default at `uitree`. - -A `dc.ini` default of `visionvirtual` applies to every session on that host, -including sessions that test apps exposing a view hierarchy. Those sessions lose -the precision of selector-based resolution and capture a screenshot on each -request. Because the strategy is chosen per app, the capability is the correct -place to set it. - -A capability passed in the session takes precedence over the `dc.ini` value. - -`visionvirtual` uses the vision model already configured for Appium AI. You do -not need to set `kobiton:llmVisionModel` to use it. To route vision requests -through a different model or provider, see -xref:appium-ai/override-llm.adoc[Override Appium AI settings per session]. - -IMPORTANT: Appium AI does not detect canvas-based apps. To use vision-based -resolution, set `kobiton:locatorStrategy` to `visionvirtual` explicitly. - -Appium AI resolves the strategy once when the session starts. You cannot switch -strategies during a test run, and a single session cannot use both. To exercise -both strategies, run separate sessions. - -== Use a natural language locator - -To locate an element using natural language, pass a descriptive string into `findElement(...)` using the "natural" locator strategy. - -[source,python] -.python ----- -element = driver.find_element("natural", "The login button at the bottom") -element.click() ----- - -[source,java] -.java ----- -WebElement el = driver.findElement(ByNatural.natural("the Accessibility button")); -el.click(); ----- - -NOTE: The above Java example used the custom `ByNatural` class to extend the default custom locator strategy value. See xref:automation-testing:appium-ai/extend-custom-locator-strategy-for-nll.adoc[this guide,window=read-later] for instructions. - -== Write effective descriptions - -Use clear, specific descriptions to help identify the correct element. - -Good examples: - ----- -java -driver.findElement(ByNatural.natural("the Accessibility button")); -driver.findElement(ByNatural.natural("the Username field")); -driver.findElement(ByNatural.natural("the City dropdown list")); ----- - -Less effective examples: - ----- -driver.findElement(ByNatural.natural("Click button") -driver.findElement("Select item") ----- - -When multiple similar elements are present, include additional context such as position, label, or surrounding UI to improve accuracy. - -== Expected behavior - -Appium AI evaluates the description and returns the most relevant matching -element using the locator strategy set for the session. `uitree` returns a -selector resolved from the view hierarchy. `visionvirtual` returns a virtual -element mapped to coordinates on the screenshot. - -Results are not guaranteed to be deterministic. If multiple elements match the description, you may need to refine the description to improve accuracy. - -Natural language locators work alongside traditional selector strategies and are best used in combination with them, rather than as a complete replacement. - -== Troubleshooting - -If an element is not found or the wrong element is returned: - -* If the app renders its UI on a canvas, set `kobiton:locatorStrategy` to -`visionvirtual`. The default `uitree` strategy cannot locate elements in an app -that exposes no view hierarchy. - -* Use more specific language in the description - -* Include additional context such as position, labels, or surrounding elements - -* Verify that the UI element is visible and accessible in the view hierarchy - -* When using `uitree`, confirm that the application exposes sufficient UI -metadata (view hierarchy or accessibility attributes) for element identification - -* When using `visionvirtual`, ensure the target element is clearly visible on -the current screen - -If natural language locators are not working as expected, contact your lab administrator to verify the following: - -* OpenAI or Azure OpenAI API credentials are configured correctly - -* Natural language locators are properly configured in your environment - -* The deployment uses private devices (natural language locators are not available on Public Cloud devices) - -If issues persist, contact Kobiton Support for assistance. - -== Next steps - -For an overview of how natural language locators work and their limitations, see the xref:appium-ai/overview-appium-ai.adoc[Appium AI] guide. \ No newline at end of file diff --git a/docs/modules/automation-testing/pages/capabilities/available-capabilities.adoc b/docs/modules/automation-testing/pages/capabilities/available-capabilities.adoc index 43e5b13d2..2af82904b 100644 --- a/docs/modules/automation-testing/pages/capabilities/available-capabilities.adoc +++ b/docs/modules/automation-testing/pages/capabilities/available-capabilities.adoc @@ -444,6 +444,173 @@ Enable XPath 2.0 searches on all supported device platforms. Default is `false`. capabilities.setCapability("kobiton:xpath2", true); ---- +[discrete] +=== Appium AI capabilities + +These capabilities override the Appium AI configuration on a per-session basis. For the system-level configuration, see xref:automation-testing:appium-ai/override-llm.adoc[Override Appium AI settings per session]. + +.Appium AI capability index +* <<_llmApiFormat,kobiton:llmApiFormat>> +* <<_llmApiKey,kobiton:llmApiKey>> +* <<_llmApiVersion,kobiton:llmApiVersion>> +* <<_llmBaseUrl,kobiton:llmBaseUrl>> +* <<_llmTextExtraBody,kobiton:llmTextExtraBody>> +* <<_llmTextModel,kobiton:llmTextModel>> +* <<_llmVisionExtraBody,kobiton:llmVisionExtraBody>> +* <<_llmVisionModel,kobiton:llmVisionModel>> +* <<_locatorStrategy,kobiton:locatorStrategy>> +* <<_visionCoordSpace,kobiton:visionCoordSpace>> + +[#_llmApiFormat] +=== `kobiton:llmApiFormat` + +API protocol format for the LLM endpoint. The value must match the provider used by `kobiton:llmBaseUrl`. Accepts `OpenAiResponses`, `OpenAiChatCompletions`, `AnthropicMessages`, `KobitonChatCompletions`, `OllamaChat`, `AzureOpenAiV1ChatCompletions`, or `AzureOpenAiChatCompletions`. Default is `OpenAiResponses`. + +* *Type:* `string` +* *Required capabilities:* None +* *Optional capabilities:* None + +.Example +[source,java] +---- +capabilities.setCapability("kobiton:llmApiFormat", "AnthropicMessages"); +---- + +[#_llmApiKey] +=== `kobiton:llmApiKey` + +LLM provider API key. A session that sets `kobiton:llmBaseUrl` must also set this capability. The default server-side key is not sent to a session-specified endpoint. + +* *Type:* `string` +* *Required capabilities:* None +* *Optional capabilities:* None + +.Example +[source,java] +---- +capabilities.setCapability("kobiton:llmApiKey", ""); +---- + +[#_llmApiVersion] +=== `kobiton:llmApiVersion` + +API version string for Azure OpenAI endpoints. Required when `kobiton:llmApiFormat` is set to an Azure value. Default is `2024-02-01`. + +* *Type:* `string` +* *Required capabilities:* None +* *Optional capabilities:* None + +.Example +[source,java] +---- +capabilities.setCapability("kobiton:llmApiVersion", "2024-02-01"); +---- + +[#_llmBaseUrl] +=== `kobiton:llmBaseUrl` + +LLM provider base URL. Default is `https://api.openai.com/v1`. See xref:automation-testing:appium-ai/override-llm.adoc[Override Appium AI settings per session] for provider-specific endpoint values. + +* *Type:* `string` +* *Required capabilities:* `kobiton:llmApiKey` +* *Optional capabilities:* `kobiton:llmApiFormat`, `kobiton:llmTextModel`, `kobiton:llmVisionModel` + +.Example +[source,java] +---- +capabilities.setCapability("kobiton:llmBaseUrl", "https://api.anthropic.com/v1"); +---- + +[#_llmTextExtraBody] +=== `kobiton:llmTextExtraBody` + +JSON object passed verbatim into the request body for text model calls. Use to send model-specific parameters such as `temperature`. Leave unset unless the model requires it. + +* *Type:* `string` +* *Required capabilities:* None +* *Optional capabilities:* None + +.Example +[source,java] +---- +capabilities.setCapability("kobiton:llmTextExtraBody", "{\"temperature\": 0}"); +---- + +[#_llmTextModel] +=== `kobiton:llmTextModel` + +Model used for UI tree-based operations (XPath/CSS generation). For Azure, pass the deployment name. Default is `gpt-5.6-terra`. + +* *Type:* `string` +* *Required capabilities:* None +* *Optional capabilities:* None + +.Example +[source,java] +---- +capabilities.setCapability("kobiton:llmTextModel", "claude-opus-4-8"); +---- + +[#_llmVisionExtraBody] +=== `kobiton:llmVisionExtraBody` + +JSON object passed verbatim into the request body for vision model calls. Use to send model-specific parameters such as `temperature`. Leave unset unless the model requires it. + +* *Type:* `string` +* *Required capabilities:* None +* *Optional capabilities:* None + +.Example +[source,java] +---- +capabilities.setCapability("kobiton:llmVisionExtraBody", "{\"temperature\": 0}"); +---- + +[#_llmVisionModel] +=== `kobiton:llmVisionModel` + +Model used for vision-based operations (screenshot analysis). Must accept image input. For Azure, pass the deployment name. Default is `gpt-5.6-terra`. + +* *Type:* `string` +* *Required capabilities:* None +* *Optional capabilities:* None + +.Example +[source,java] +---- +capabilities.setCapability("kobiton:llmVisionModel", "claude-opus-4-8"); +---- + +[#_locatorStrategy] +=== `kobiton:locatorStrategy` + +Locator strategy used to resolve natural language locators. Accepts `uitree` and `visionvirtual`. Pass `visionvirtual` for canvas-based apps. Default is `uitree`. See xref:automation-testing:appium-ai/overview-appium-ai.adoc[Appium AI: Natural language locators] for guidance on choosing a strategy. + +* *Type:* `string` +* *Required capabilities:* None +* *Optional capabilities:* None + +.Example +[source,java] +---- +capabilities.setCapability("kobiton:locatorStrategy", "visionvirtual"); +---- + +[#_visionCoordSpace] +=== `kobiton:visionCoordSpace` + +Coordinate convention the vision model returns. Accepts `Pixel` or `Normalized` (0–1000). Default is `Pixel`, which is correct for GPT-5.6, Claude Opus 4.8, and the Kobiton-hosted qwen25-vl-7b. Set `Normalized` only for a model that returns 0–1000 coordinates, such as the Qwen3 family. + +* *Type:* `string` +* *Required capabilities:* None +* *Optional capabilities:* None + +.Example +[source,java] +---- +capabilities.setCapability("kobiton:visionCoordSpace", "Pixel"); +---- + == Appium Capabilities Kobiton supports most standard Appium capabilities. The capabilities listed below are commonly used when running automation sessions on Kobiton devices.