Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions docs/README.skills.md

Large diffs are not rendered by default.

76 changes: 42 additions & 34 deletions skills/arize-ai-provider-integration/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ description: "INVOKE THIS SKILL when creating, reading, updating, or deleting Ar

# Arize AI Integration Skill

> **`SPACE`** — Most `--space` flags and the `ARIZE_SPACE` env var accept a space **name** (e.g., `my-workspace`) or a base64 space **ID** (e.g., `U3BhY2U6...`). Find yours with `ax spaces list`.
> **Note:** `ai-integrations create` does **not** accept `--space` — AI integrations are account-scoped. Use `--space` only with `list`, `get`, `update`, and `delete`.

## Concepts

- **AI Integration** = stored LLM provider credentials registered in Arize; used by evaluators to call a judge model and by other Arize features that need to invoke an LLM on your behalf
Expand All @@ -19,9 +22,10 @@ Proceed directly with the task — run the `ax` command you need. Do NOT check v

If an `ax` command fails, troubleshoot based on the error:
- `command not found` or version error → see references/ax-setup.md
- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong: check `.env` for `ARIZE_API_KEY` and use it to create/update the profile via references/ax-profiles.md. If `.env` has no key either, ask the user for their Arize API key (https://app.arize.com/admin > API Keys)
- Space ID unknown → check `.env` for `ARIZE_SPACE_ID`, or run `ax spaces list -o json`, or ask the user
- LLM provider call fails (missing OPENAI_API_KEY / ANTHROPIC_API_KEY) → check `.env`, load if present, otherwise ask the user
- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys
- Space unknown → run `ax spaces list` to pick by name, or ask the user
- LLM provider call fails (missing OPENAI_API_KEY / ANTHROPIC_API_KEY) → run `ax ai-integrations list --space SPACE` to check for platform-managed credentials. If none exist, ask the user to provide the key or create an integration via the **arize-ai-provider-integration** skill
- **Security:** Never read `.env` files or search the filesystem for credentials. Use `ax profiles` for Arize credentials and `ax ai-integrations` for LLM provider keys. If credentials are not available through these channels, ask the user.

---

Expand All @@ -30,32 +34,32 @@ If an `ax` command fails, troubleshoot based on the error:
List all integrations accessible in a space:

```bash
ax ai-integrations list --space-id SPACE_ID
ax ai-integrations list --space SPACE
```

Filter by name (case-insensitive substring match):

```bash
ax ai-integrations list --space-id SPACE_ID --name "openai"
ax ai-integrations list --space SPACE --name "openai"
```

Paginate large result sets:

```bash
# Get first page
ax ai-integrations list --space-id SPACE_ID --limit 20 -o json
ax ai-integrations list --space SPACE --limit 20 -o json

# Get next page using cursor from previous response
ax ai-integrations list --space-id SPACE_ID --limit 20 --cursor CURSOR_TOKEN -o json
ax ai-integrations list --space SPACE --limit 20 --cursor CURSOR_TOKEN -o json
```

**Key flags:**

| Flag | Description |
|------|-------------|
| `--space-id` | Space to list integrations in |
| `--space` | Space name or ID to filter integrations |
| `--name` | Case-insensitive substring filter on integration name |
| `--limit` | Max results (1–100, default 50) |
| `--limit` | Max results (1–100, default 15) |
| `--cursor` | Pagination token from a previous response |
| `-o, --output` | Output format: `table` (default) or `json` |

Expand All @@ -77,8 +81,9 @@ ax ai-integrations list --space-id SPACE_ID --limit 20 --cursor CURSOR_TOKEN -o
## Get a Specific Integration

```bash
ax ai-integrations get INT_ID
ax ai-integrations get INT_ID -o json
ax ai-integrations get NAME_OR_ID
ax ai-integrations get NAME_OR_ID -o json
ax ai-integrations get NAME_OR_ID --space SPACE # required when using name instead of ID
```

Use this to inspect an integration's full configuration or to confirm its ID after creation.
Expand All @@ -90,7 +95,7 @@ Use this to inspect an integration's full configuration or to confirm its ID aft
Before creating, always list integrations first — the user may already have a suitable one:

```bash
ax ai-integrations list --space-id SPACE_ID
ax ai-integrations list --space SPACE
```

If no suitable integration exists, create one. The required flags depend on the provider.
Expand Down Expand Up @@ -125,25 +130,24 @@ ax ai-integrations create \

### AWS Bedrock

AWS Bedrock uses IAM role-based auth instead of an API key. Provide the ARN of the role Arize should assume:
AWS Bedrock uses IAM role-based auth. Provide the ARN of the role Arize should assume via `--provider-metadata`:

```bash
ax ai-integrations create \
--name "My Bedrock Integration" \
--provider awsBedrock \
--role-arn "arn:aws:iam::123456789012:role/ArizeBedrockRole"
--provider-metadata '{"role_arn": "arn:aws:iam::123456789012:role/ArizeBedrockRole"}'
```

### Vertex AI

Vertex AI uses GCP service account credentials. Provide the GCP project and region:
Vertex AI uses GCP service account credentials. Provide the GCP project and region via `--provider-metadata`:

```bash
ax ai-integrations create \
--name "My Vertex AI Integration" \
--provider vertexAI \
--project-id "my-gcp-project" \
--location "us-central1"
--provider-metadata '{"project_id": "my-gcp-project", "location": "us-central1"}'
```

### Gemini
Expand Down Expand Up @@ -182,8 +186,8 @@ ax ai-integrations create \
| `openAI` | `--api-key <key>` |
| `anthropic` | `--api-key <key>` |
| `azureOpenAI` | `--api-key <key>`, `--base-url <azure-endpoint>` |
| `awsBedrock` | `--role-arn <arn>` |
| `vertexAI` | `--project-id <gcp-project>`, `--location <region>` |
| `awsBedrock` | `--provider-metadata '{"role_arn": "<arn>"}'` |
| `vertexAI` | `--provider-metadata '{"project_id": "<gcp-project>", "location": "<region>"}'` |
| `gemini` | `--api-key <key>` |
| `nvidiaNim` | `--api-key <key>`, `--base-url <nim-endpoint>` |
| `custom` | `--base-url <endpoint>` |
Expand All @@ -192,18 +196,21 @@ ax ai-integrations create \

| Flag | Description |
|------|-------------|
| `--model-names` | Comma-separated list of allowed model names; omit to allow all models |
| `--enable-default-models` / `--no-default-models` | Enable or disable the provider's default model list |
| `--function-calling` / `--no-function-calling` | Enable or disable tool/function calling support |
| `--model-name` | Allowed model name (repeat for multiple, e.g. `--model-name gpt-4o --model-name gpt-4o-mini`); omit to allow all models |
| `--enable-default-models` | Enable the provider's default model list |
| `--function-calling-enabled` | Enable tool/function calling support |
| `--auth-type` | Authentication type: `default`, `proxy_with_headers`, or `bearer_token` |
| `--headers` | Custom headers as JSON object or file path (for proxy auth) |
| `--provider-metadata` | Provider-specific metadata as JSON object or file path |

### After creation

Capture the returned integration ID (e.g., `TGxtSW50ZWdyYXRpb246MTI6YUJjRA==`) — it is needed for evaluator creation and other downstream commands. If you missed it, retrieve it:

```bash
ax ai-integrations list --space-id SPACE_ID -o json
# or, if you know the ID:
ax ai-integrations get INT_ID
ax ai-integrations list --space SPACE -o json
# or by name/ID directly:
ax ai-integrations get NAME_OR_ID
```

---
Expand All @@ -214,19 +221,19 @@ ax ai-integrations get INT_ID

```bash
# Rename
ax ai-integrations update INT_ID --name "New Name"
ax ai-integrations update NAME_OR_ID --name "New Name"

# Rotate the API key
ax ai-integrations update INT_ID --api-key $OPENAI_API_KEY
ax ai-integrations update NAME_OR_ID --api-key $OPENAI_API_KEY

# Change the model list
ax ai-integrations update INT_ID --model-names "gpt-4o,gpt-4o-mini"
# Change the model list (replaces all existing model names)
ax ai-integrations update NAME_OR_ID --model-name gpt-4o --model-name gpt-4o-mini

# Update base URL (for Azure, custom, or NIM)
ax ai-integrations update INT_ID --base-url "https://new-endpoint.example.com/v1"
ax ai-integrations update NAME_OR_ID --base-url "https://new-endpoint.example.com/v1"
```

Any flag accepted by `create` can be passed to `update`.
Add `--space SPACE` when using a name instead of ID. Any flag accepted by `create` can be passed to `update`.

---

Expand All @@ -235,7 +242,8 @@ Any flag accepted by `create` can be passed to `update`.
**Warning:** Deletion is permanent. Evaluators that reference this integration will no longer be able to run.

```bash
ax ai-integrations delete INT_ID --force
ax ai-integrations delete NAME_OR_ID --force
ax ai-integrations delete NAME_OR_ID --space SPACE --force # required when using name instead of ID
```

Omit `--force` to get a confirmation prompt instead of deleting immediately.
Expand All @@ -249,8 +257,8 @@ Omit `--force` to get a confirmation prompt instead of deleting immediately.
| `ax: command not found` | See references/ax-setup.md |
| `401 Unauthorized` | API key may not have access to this space. Verify key and space ID at https://app.arize.com/admin > API Keys |
| `No profile found` | Run `ax profiles show --expand`; set `ARIZE_API_KEY` env var or write `~/.arize/config.toml` |
| `Integration not found` | Verify with `ax ai-integrations list --space-id SPACE_ID` |
| `has_api_key: false` after create | Credentials were not saved — re-run `update` with the correct `--api-key` or `--role-arn` |
| `Integration not found` | Verify with `ax ai-integrations list --space SPACE` |
| `has_api_key: false` after create | Credentials were not saved — re-run `update` with the correct `--api-key` or `--provider-metadata` |
| Evaluator runs fail with LLM errors | Check integration credentials with `ax ai-integrations get INT_ID`; rotate the API key if needed |
| `provider` mismatch | Cannot change provider after creation — delete and recreate with the correct provider |

Expand Down
16 changes: 8 additions & 8 deletions skills/arize-ai-provider-integration/references/ax-profiles.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ ax profiles create work --api-key $ARIZE_API_KEY --region us-east-1b

To use a named profile with any `ax` command, add `-p NAME`:
```bash
ax spans export PROJECT_ID -p work
ax spans export PROJECT -p work
```

## 4. Getting the API key
Expand All @@ -81,19 +81,19 @@ ax profiles show

Confirm the API key and region are correct, then retry the original command.

## Space ID
## Space

There is no profile flag for space ID. Save it as an environment variable:
There is no profile flag for space. Save it as an environment variable — accepts a space **name** (e.g., `my-workspace`) or a base64 space **ID** (e.g., `U3BhY2U6...`). Find yours with `ax spaces list -o json`.

**macOS/Linux** — add to `~/.zshrc` or `~/.bashrc`:
```bash
export ARIZE_SPACE_ID="U3BhY2U6..."
export ARIZE_SPACE="my-workspace" # name or base64 ID
```
Then `source ~/.zshrc` (or restart terminal).

**Windows (PowerShell):**
```powershell
[System.Environment]::SetEnvironmentVariable('ARIZE_SPACE_ID', 'U3BhY2U6...', 'User')
[System.Environment]::SetEnvironmentVariable('ARIZE_SPACE', 'my-workspace', 'User')
```
Restart terminal for it to take effect.

Expand All @@ -103,13 +103,13 @@ At the **end of the session**, if the user manually provided any credentials dur

**Skip this entirely if:**
- The API key was already loaded from an existing profile or `ARIZE_API_KEY` env var
- The space ID was already set via `ARIZE_SPACE_ID` env var
- The user only used base64 project IDs (no space ID was needed)
- The space was already set via `ARIZE_SPACE` env var
- The user only used base64 project IDs (no space was needed)

**How to offer:** Use **AskQuestion**: *"Would you like to save your Arize credentials so you don't have to enter them next time?"* with options `"Yes, save them"` / `"No thanks"`.

**If the user says yes:**

1. **API key** — Run `ax profiles show` to check the current state. Then run `ax profiles create --api-key $ARIZE_API_KEY` or `ax profiles update --api-key $ARIZE_API_KEY` (the key must already be exported as an env var — never pass a raw key value).

2. **Space ID** — See the Space ID section above to persist it as an environment variable.
2. **Space** — See the Space section above to persist it as an environment variable.
4 changes: 2 additions & 2 deletions skills/arize-ai-provider-integration/references/ax-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Consult this only when an `ax` command fails. Do NOT run these checks proactivel

## Check version first

If `ax` is installed (not `command not found`), always run `ax --version` before investigating further. The version must be `0.8.0` or higher — many errors are caused by an outdated install. If the version is too old, see **Version too old** below.
If `ax` is installed (not `command not found`), always run `ax --version` before investigating further. The version must be `0.14.0` or higher — many errors are caused by an outdated install. If the version is too old, see **Version too old** below.

## `ax: command not found`

Expand All @@ -19,7 +19,7 @@ If `ax` is installed (not `command not found`), always run `ax --version` before
3. Install: `pip install arize-ax-cli`
4. Add to PATH: `$env:PATH = "$env:APPDATA\Python\Scripts;$env:PATH"`

## Version too old (below 0.8.0)
## Version too old (below 0.14.0)

Upgrade: `uv tool install --force --reinstall arize-ax-cli`, `pipx upgrade arize-ax-cli`, or `pip install --upgrade arize-ax-cli`

Expand Down
Loading
Loading