Skip to content

docs(KOB-55116): update Appium AI docs for new providers and consolidate pages #686

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tiffany-kobiton wants to merge 2 commits into
base: main
Choose a base branch
from
Open

docs(KOB-55116): update Appium AI docs for new providers and consolidate pages #686

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f97fda6
docs(KOB-55116): update Appium AI docs for new providers and consolid…
tiffany-kobiton Oct 5, 2026
04273a5
docs(KOB-55116): group Appium AI capabilities into their own sub-section
tiffany-kobiton Oct 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion docs/modules/automation-testing/nav.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,5 @@

* Appium AI
** xref:appium-ai/overview-appium-ai.adoc[]
** xref:appium-ai/user-guide-appium-ai.adoc[]
** xref:appium-ai/override-llm.adoc[]
** xref:appium-ai/extend-custom-locator-strategy-for-nll.adoc[]
149 changes: 64 additions & 85 deletions docs/modules/automation-testing/pages/appium-ai/override-llm.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,105 +1,40 @@
= Override Appium AI settings per session
:description: Use Appium desired capabilities to override the default Appium AI provider and locator settings for a session.

Appium AI uses an LLM provider configured by your Kobiton administrator by default. You can override those settings for a single Appium session by passing `kobiton:*` desired capabilities.
By default, Appium AI uses an LLM provider configured by your Kobiton administrator. You can override those settings for a single Appium session by passing `kobiton:*` desired capabilities.

`kobiton:locatorStrategy` selects how Appium AI resolves elements. It is not an
LLM provider setting, but it is set the same way. See the
xref:appium-ai/user-guide-appium-ai.adoc[Appium AI user guide] for guidance on
choosing a strategy.
See xref:capabilities/available-capabilities.adoc[Available capabilities] for the full reference of the `kobiton:llm*`, `kobiton:visionCoordSpace`, and `kobiton:locatorStrategy` capabilities. `kobiton:locatorStrategy` selects how Appium AI resolves elements; see xref:appium-ai/overview-appium-ai.adoc[Appium AI: Natural language locators] for guidance on choosing a strategy.

Use this to:

- Route AI requests through your own OpenAI, Azure OpenAI, or local LLM endpoint to meet compliance or data residency requirements.
- Route AI requests through your own OpenAI, Azure OpenAI, Anthropic Claude, or local LLM endpoint to meet compliance or data residency requirements.
- Use a different model without requiring an administrator to update the default configuration.
- Run AI features against a local LLM for air-gapped or cost-sensitive environments.

== Prerequisites

- An administrator has enabled the Appium AI feature by configuring the `dc.ini` file on each Mac mini in the environment.

- An administrator has enabled the Appium AI feature in your environment.
- You have an API key or reachable local endpoint for the LLM provider you want to use.

== Available capabilities

Capability names and values are case-insensitive.

If a capability is omitted, the session uses the default value configured by your Kobiton admin.

[cols="2,4,2,2", options="header"]
|===
| Capability | Purpose | Overrides | Default

| `kobiton:llmBaseUrl`
| LLM provider base URL.
| `LlmClient.BaseUrl`
| `https://api.openai.com/v1`

| `kobiton:llmApiKey`
| LLM provider API key. Required in either `dc.ini` or capabilities.
| `LlmClient.ApiKey`
| (none)

| `kobiton:llmApiFormat`
| API protocol format.
| `LlmClient.ApiFormat`
| `OpenAiResponses`

| `kobiton:llmApiVersion`
| Required only when using Azure OpenAI formats.
| `LlmClient.ApiVersion`
| `2024-02-01`

| `kobiton:llmTextModel`
| Model used for UI tree-based operations (XPath/CSS generation). For Azure, pass the deployment name.
| `LlmClient.TextModel`
| `gpt-5.2`

| `kobiton:llmVisionModel`
| Model used for vision-based operations (screenshot analysis). Must support image input. For Azure, pass the deployment name.
| `LlmClient.VisionModel`
| `gpt-5.2`

| `kobiton:llmTextExtraBody`
| Optional. JSON object passed verbatim into the request body for text model calls. Use to send model-specific parameters such as `temperature`.
| `LlmClient.TextExtraBody`
| (none)

| `kobiton:llmVisionExtraBody`
| Optional. JSON object passed verbatim into the request body for vision model calls. Use to send model-specific parameters such as `temperature`.
| `LlmClient.VisionExtraBody`
| (none)

| `kobiton:locatorStrategy`
| Locator strategy used to resolve natural language locators. Pass `visionvirtual` per session for canvas-based apps. Accepts `uitree` and `visionvirtual`.
| `NaturalLanguageLocator.LocatorStrategy`
| `uitree`
|===

`kobiton:llmApiFormat` supports the following values:

- `OpenAiResponses`

- `OpenAiChatCompletions`

- `AzureOpenAiV1ChatCompletions`

- `AzureOpenAiChatCompletions`

== Examples

=== OpenAI (default endpoint, customer API key)
=== OpenAI (default endpoint, your API key)

```
[source,python]
.python
----
desired_capabilities = {
'kobiton:llmApiKey': '<your-openai-key>',
'kobiton:llmTextModel': 'gpt-4o',
'kobiton:llmVisionModel': 'gpt-4o',
'kobiton:llmTextModel': 'gpt-5.6-terra',
'kobiton:llmVisionModel': 'gpt-5.6-terra',
}
```
----

=== Azure OpenAI
```

[source,python]
.python
----
desired_capabilities = {
'kobiton:llmBaseUrl': 'https://my-resource.openai.azure.com',
'kobiton:llmApiKey': '<your-azure-key>',
Expand All @@ -108,32 +43,76 @@ desired_capabilities = {
'kobiton:llmTextModel': 'my-gpt4o-deployment',
'kobiton:llmVisionModel': 'my-gpt4o-deployment',
}
```
----

For Azure, `llmTextModel` and `llmVisionModel` use the Azure deployment name, not the underlying model name.

=== Anthropic Claude

[source,python]
.python
----
desired_capabilities = {
'kobiton:llmBaseUrl': 'https://api.anthropic.com/v1',
'kobiton:llmApiKey': '<your-anthropic-key>',
'kobiton:llmApiFormat': 'AnthropicMessages',
'kobiton:llmTextModel': 'claude-opus-4-8',
'kobiton:llmVisionModel': 'claude-opus-4-8',
}
----

=== Kobiton-hosted LLM

The Kobiton-hosted LLM runs at `https://llm.kobiton.com/v1` and needs no third-party LLM account. It is available on Private and Local devices where the Mac mini host can reach `llm.kobiton.com`. It is unavailable on an air-gapped host or where the domain is blocked.

The hosted model serves vision only. Set `kobiton:locatorStrategy` to `visionvirtual` for sessions that use it.

// REVIEWER NOTE: The `<your-kobiton-llm-key>` placeholder is unresolved. Admin-side
// configuration falls back to `Kobiton.TenantApiKey` (see admin-docs#217), but
// KOB-54645 states that server-side keys are not sent to session-specified endpoints.
// What value a customer passes for a session-level override to the hosted LLM needs
// confirmation from engineering before publication.

[source,python]
.python
----
desired_capabilities = {
'kobiton:llmBaseUrl': 'https://llm.kobiton.com/v1',
'kobiton:llmApiKey': '<your-kobiton-llm-key>',
'kobiton:llmApiFormat': 'KobitonChatCompletions',
'kobiton:llmVisionModel': 'qwen25-vl-7b',
'kobiton:locatorStrategy': 'visionvirtual',
}
----

`qwen25-vl-7b` is the only model this endpoint serves. Any other value in `kobiton:llmVisionModel` returns an error. If your administrator has already set the Kobiton-hosted LLM as the host-wide default, sessions do not need to set these capabilities.

=== Local LLM endpoint (Ollama)

```
[source,python]
.python
----
desired_capabilities = {
'kobiton:llmBaseUrl': 'http://localhost:11434',
'kobiton:llmApiKey': '<token-or-placeholder>',
'kobiton:llmApiFormat': 'OpenAiChatCompletions',
'kobiton:llmTextModel': '<local-model-name>',
'kobiton:llmVisionModel': '<vision-capable-local-model>',
}
```
----

[NOTE]
====
Ollama exposes an OpenAI-compatible chat completions API, so `kobiton:llmApiFormat` must be set to `OpenAiChatCompletions`.
Ollama exposes a native API and an OpenAI-compatible chat completions API. Set `kobiton:llmApiFormat` to `OllamaChat` for the native endpoint or `OpenAiChatCompletions` for the compatibility endpoint.

`kobiton:llmApiKey` is required even for local Ollama endpoints that do not enforce authentication. Pass any value, such as `ollama`.
====

Local endpoints are resolved from the host running the Appium session. For example, `localhost` refers to the Mac mini running the session.

== Notes

- Capabilities apply only to the session that passed them; other sessions continue to use the defaults.
- A session that sets `kobiton:llmBaseUrl` must also set `kobiton:llmApiKey`. The default server-side key is not sent to a session-specified endpoint. Existing scripts that set `kobiton:llmBaseUrl` without `kobiton:llmApiKey` fail on the first LLM request.
- `kobiton:llmApiFormat` must match the provider used by `kobiton:llmBaseUrl`.
- Vision-based features require a model that accepts image input such as screenshots.
- Pass `kobiton:locatorStrategy` per session instead of changing the `dc.ini` default. A `dc.ini` default of `visionvirtual` applies to every session on that host, including sessions that test apps exposing a view hierarchy.
Loading
Loading