chore: sync latest Arize and Phoenix skills#1583
Open
jimbobbennett wants to merge 1 commit intogithub:stagedfrom
Open
chore: sync latest Arize and Phoenix skills#1583jimbobbennett wants to merge 1 commit intogithub:stagedfrom
jimbobbennett wants to merge 1 commit intogithub:stagedfrom
Conversation
…08a082911b18fc and phoenix@746247cbb07b0dc7803b87c69dd8c77811c33f59
Contributor
🔍 Skill Validator Results
Summary
Full validator output```text Found 12 skill(s) [arize-ai-provider-integration] 📊 arize-ai-provider-integration: 2,632 BPE tokens [chars/4: 2,548] (standard ~), 29 sections, 16 code blocks [arize-ai-provider-integration] ⚠ Skill is 2,632 BPE tokens (chars/4 estimate: 2,548) — approaching "comprehensive" range where gains diminish. [arize-ai-provider-integration] ⚠ No numbered workflow steps — agents follow sequenced procedures more reliably. [arize-annotation] 📊 arize-annotation: 2,517 BPE tokens [chars/4: 2,689] (standard ~), 27 sections, 15 code blocks [arize-annotation] ⚠ Skill is 2,517 BPE tokens (chars/4 estimate: 2,689) — approaching "comprehensive" range where gains diminish. [arize-annotation] ⚠ No numbered workflow steps — agents follow sequenced procedures more reliably. [arize-dataset] 📊 arize-dataset: 3,818 BPE tokens [chars/4: 3,809] (standard ~), 51 sections, 16 code blocks [arize-dataset] ⚠ Skill is 3,818 BPE tokens (chars/4 estimate: 3,809) — approaching "comprehensive" range where gains diminish. [arize-evaluator] 📊 arize-evaluator: 7,814 BPE tokens [chars/4: 8,033] (comprehensive ✗), 59 sections, 28 code blocks [arize-evaluator] ⚠ Skill is 7,814 BPE tokens (chars/4 estimate: 8,033) — "comprehensive" skills hurt performance by 2.9pp on average. Consider splitting into 2–3 focused skills. [arize-experiment] 📊 arize-experiment: 4,589 BPE tokens [chars/4: 4,610] (standard ~), 34 sections, 20 code blocks [arize-experiment] ⚠ Skill is 4,589 BPE tokens (chars/4 estimate: 4,610) — approaching "comprehensive" range where gains diminish. [arize-instrumentation] 📊 arize-instrumentation: 4,935 BPE tokens [chars/4: 5,061] (standard ~), 19 sections, 3 code blocks [arize-instrumentation] ⚠ Skill is 4,935 BPE tokens (chars/4 estimate: 5,061) — approaching "comprehensive" range where gains diminish. [arize-link] 📊 arize-link: 1,203 BPE tokens [chars/4: 1,088] (detailed ✓), 9 sections, 6 code blocks [arize-prompt-optimization] 📊 arize-prompt-optimization: 4,451 BPE tokens [chars/4: 4,762] (standard ~), 58 sections, 19 code blocks [arize-prompt-optimization] ⚠ Skill is 4,451 BPE tokens (chars/4 estimate: 4,762) — approaching "comprehensive" range where gains diminish. [arize-trace] 📊 arize-trace: 5,873 BPE tokens [chars/4: 5,831] (comprehensive ✗), 43 sections, 10 code blocks [arize-trace] ⚠ Skill is 5,873 BPE tokens (chars/4 estimate: 5,831) — "comprehensive" skills hurt performance by 2.9pp on average. Consider splitting into 2–3 focused skills. [phoenix-cli] 📊 phoenix-cli: 2,778 BPE tokens [chars/4: 2,764] (standard ~), 19 sections, 15 code blocks [phoenix-cli] ⚠ Skill is 2,778 BPE tokens (chars/4 estimate: 2,764) — approaching "comprehensive" range where gains diminish. [phoenix-cli] ⚠ No numbered workflow steps — agents follow sequenced procedures more reliably. [phoenix-evals] 📊 phoenix-evals: 1,089 BPE tokens [chars/4: 1,126] (detailed ✓), 5 sections, 0 code blocks [phoenix-evals] ⚠ No code blocks — agents perform better with concrete snippets and commands. [phoenix-evals] ⚠ No numbered workflow steps — agents follow sequenced procedures more reliably. [phoenix-tracing] 📊 phoenix-tracing: 1,488 BPE tokens [chars/4: 1,690] (detailed ✓), 17 sections, 1 code blocks ✅ All checks passed (12 skill(s)) ``` |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR synchronizes the Arize and Phoenix skill documentation to newer SDK/CLI APIs, refreshing example imports, command flags, and adding new workflow guidance to keep the skills aligned with the latest product surfaces.
Changes:
- Update Phoenix tracing docs to use the newer
phoenix.otelimports and@arizeai/phoenix-clientTypeScript package paths, plus add “notes” examples. - Update Phoenix evals docs to use the newer
Client().experiments.run_experiment(...)patterns and add “stability / repetitions” guidance. - Update multiple Arize
ax-CLI skills to newer flags/version expectations (notably--space/ARIZE_SPACE) and refresh command examples; update the skills index accordingly.
Reviewed changes
Copilot reviewed 40 out of 40 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| skills/phoenix-tracing/references/sessions-python.md | Updates Python session-tracking imports to phoenix.otel. |
| skills/phoenix-tracing/references/metadata-python.md | Updates install/import guidance to arize-phoenix-otel and phoenix.otel context managers. |
| skills/phoenix-tracing/references/annotations-typescript.md | Updates TypeScript client imports to @arizeai/phoenix-client/* and adds note examples. |
| skills/phoenix-tracing/references/annotations-python.md | Adds Python “span notes” example. |
| skills/phoenix-tracing/README.md | Adds a skill-level README describing organization and external references. |
| skills/phoenix-evals/references/production-overview.md | Updates experiment-running snippet to Client().experiments.run_experiment(...). |
| skills/phoenix-evals/references/fundamentals-model-selection.md | Updates example to use Client().experiments.run_experiment(...) with a task wrapper. |
| skills/phoenix-evals/references/fundamentals-anti-patterns.md | Updates baseline/improved experiment examples; adds single-run noise anti-pattern note. |
| skills/phoenix-evals/references/experiments-running-typescript.md | Adds “stability / repetitions” guidance for noisy tasks/evaluators. |
| skills/phoenix-evals/references/experiments-running-python.md | Adds “stability / repetitions” guidance for noisy tasks/evaluators. |
| skills/phoenix-evals/references/experiments-overview.md | Updates basic usage to use Client and adds an AsyncClient example. |
| skills/phoenix-evals/references/evaluators-code-python.md | Updates “pre-built” evaluator examples toward create_evaluator + metrics usage. |
| skills/phoenix-cli/references/open-coding.md | Adds a new “open coding” trace review workflow with CLI examples. |
| skills/phoenix-cli/references/axial-coding.md | Adds a new “axial coding” workflow to turn notes into structured taxonomies. |
| skills/phoenix-cli/SKILL.md | Expands CLI command reference and adds workflow navigation to open/axial coding docs. |
| skills/arize-trace/references/ax-setup.md | Raises minimum ax version guidance to 0.14.0. |
| skills/arize-trace/references/ax-profiles.md | Updates examples/env var guidance from SPACE_ID to SPACE/ARIZE_SPACE. |
| skills/arize-trace/SKILL.md | Updates ax CLI examples/flags and expands operational guidance for exporting traces/spans. |
| skills/arize-prompt-optimization/references/ax-setup.md | Raises minimum ax version guidance to 0.14.0. |
| skills/arize-prompt-optimization/references/ax-profiles.md | Updates examples/env var guidance from SPACE_ID to SPACE/ARIZE_SPACE. |
| skills/arize-prompt-optimization/SKILL.md | Updates export/list examples to newer ax command patterns and security guidance. |
| skills/arize-link/SKILL.md | Expands description to include sharing links with teammates. |
| skills/arize-instrumentation/references/ax-profiles.md | Updates space env var naming and refreshes API-key location guidance. |
| skills/arize-instrumentation/SKILL.md | Updates credential guidance to prefer ax profiles and avoids .env scanning. |
| skills/arize-experiment/references/ax-setup.md | Raises minimum ax version guidance to 0.14.0. |
| skills/arize-experiment/references/ax-profiles.md | Updates examples/env var guidance from SPACE_ID to SPACE/ARIZE_SPACE. |
| skills/arize-experiment/SKILL.md | Updates dataset/experiment flags (name-based flows) and adds “never fabricate outputs” guardrail. |
| skills/arize-evaluator/references/ax-setup.md | Raises minimum ax version guidance to 0.14.0. |
| skills/arize-evaluator/references/ax-profiles.md | Updates examples/env var guidance from SPACE_ID to SPACE/ARIZE_SPACE. |
| skills/arize-evaluator/SKILL.md | Updates evaluator/task commands to newer flags and adds index-lag + filtering guidance. |
| skills/arize-dataset/references/ax-setup.md | Raises minimum ax version guidance to 0.14.0. |
| skills/arize-dataset/references/ax-profiles.md | Updates examples/env var guidance from SPACE_ID to SPACE/ARIZE_SPACE. |
| skills/arize-dataset/SKILL.md | Updates dataset commands to name/ID patterns and --space usage; strengthens security guidance. |
| skills/arize-annotation/references/ax-setup.md | Raises minimum ax version guidance to 0.14.0. |
| skills/arize-annotation/references/ax-profiles.md | Updates examples/env var guidance from SPACE_ID to SPACE/ARIZE_SPACE. |
| skills/arize-annotation/SKILL.md | Expands to cover annotation queues and updates ax command syntax to newer flags. |
| skills/arize-ai-provider-integration/references/ax-setup.md | Raises minimum ax version guidance to 0.14.0. |
| skills/arize-ai-provider-integration/references/ax-profiles.md | Updates examples/env var guidance from SPACE_ID to SPACE/ARIZE_SPACE. |
| skills/arize-ai-provider-integration/SKILL.md | Updates AI integration CLI flags and provider metadata patterns (Bedrock/Vertex) and naming. |
| docs/README.skills.md | Updates the skills index entries to reflect updated Arize/Phoenix descriptions and reference files. |
Comment on lines
+11
to
+16
| All files in flat `rules/` directory with semantic prefixes: | ||
|
|
||
| - `span-*` - Span kinds (LLM, CHAIN, TOOL, etc.) | ||
| - `setup-*`, `instrumentation-*` - Getting started guides | ||
| - `fundamentals-*`, `attributes-*` - Reference docs | ||
| - `annotations-*`, `export-*` - Advanced features |
There was a problem hiding this comment.
README.md says the skill’s files live in a flat rules/ directory, but this skill currently contains SKILL.md plus a references/ directory (no rules/ directory). This description will mislead contributors; please update it to reflect the actual layout (or rename/move directories to match the doc).
| - The span, read in isolation, is still wrong: an exception fired, a tool returned an error response, the output is malformed. | ||
| - You already know the domain well enough to attribute the failure on sight without inferring across spans. | ||
|
|
||
| Session-level findings are axial-coding rollup targets, not open-coding notes — Phoenix has REST `/v1/projects/{id}/session_annotations` but no session `add-note` path. |
There was a problem hiding this comment.
This says there is no session add-note path, but skills/phoenix-cli/SKILL.md (same PR) documents px session add-note. These docs conflict; please verify which is correct for the current Phoenix CLI/API and update either this statement or the CLI reference accordingly.
Suggested change
| Session-level findings are axial-coding rollup targets, not open-coding notes — Phoenix has REST `/v1/projects/{id}/session_annotations` but no session `add-note` path. | |
| Session-level findings are better treated as axial-coding rollups than as open-coding notes; in this workflow, record observations on traces by default, or on spans when the failure is clearly localized. |
|
|
||
| The same pattern works for span-level notes — swap `px trace` for `px span` and `.traceId` for `.context.span_id`. | ||
|
|
||
| Aside: for Node-based bulk scripts, `@arizeai/phoenix-client` exposes `addSpanAnnotation`, `addSpanNote`, and `addTraceNote`. (No `addTraceAnnotation` is exported today; use the REST endpoint or `px trace annotate` for trace-level annotations.) |
There was a problem hiding this comment.
This note claims @arizeai/phoenix-client does not export addTraceAnnotation, but skills/phoenix-tracing/references/annotations-typescript.md (same PR) imports and uses addTraceAnnotation from @arizeai/phoenix-client/traces. Please reconcile these references so readers aren’t given mutually incompatible guidance.
Suggested change
| Aside: for Node-based bulk scripts, `@arizeai/phoenix-client` exposes `addSpanAnnotation`, `addSpanNote`, and `addTraceNote`. (No `addTraceAnnotation` is exported today; use the REST endpoint or `px trace annotate` for trace-level annotations.) | |
| Aside: for Node-based bulk scripts, `@arizeai/phoenix-client` exposes `addSpanAnnotation`, `addSpanNote`, and `addTraceNote` from the package root. For trace-level annotations, import `addTraceAnnotation` from `@arizeai/phoenix-client/traces`, or use the REST endpoint / `px trace annotate`. |
| **Resolving project for export:** The `PROJECT` positional argument accepts either a project name or a base64 project ID. When using a name, `--space-id` is required. If you hit limit errors or `401 Unauthorized` when using a project name, resolve it to a base64 ID: run `ax projects list --space-id SPACE_ID -l 100 -o json`, find the project by `name`, and use its `id` as `PROJECT`. | ||
| **Resolving project for export:** The `PROJECT` positional argument accepts either a project name or a base64 project ID. For `ax spans export`, a project name works without `--space`. For `ax traces export`, `--space` is required when using a project name. If you hit limit errors or `401 Unauthorized`, resolve the name to a base64 ID: run `ax projects list -l 100 -o json` (add `--space SPACE` if known), find the project by `name`, and use its `id` as `PROJECT`. | ||
|
|
||
| **Space name as ground truth:** If the user tells you their space name, use it directly — do not run `ax spaces list` first to look it up. `ax spaces list` paginates and only returns the first page (~15 spaces); the target space may be on a later page and never appear. Pass the user-provided name straight to `--space-id` or `ax projects list --space-id "<name>"`. |
There was a problem hiding this comment.
This section instructs using --space-id / ax projects list --space-id, but the rest of this skill (and the updated docs in this PR) consistently use --space and ARIZE_SPACE. Please update these flag examples to --space for consistency and to avoid sending users to a non-existent/old flag.
Suggested change
| **Space name as ground truth:** If the user tells you their space name, use it directly — do not run `ax spaces list` first to look it up. `ax spaces list` paginates and only returns the first page (~15 spaces); the target space may be on a later page and never appear. Pass the user-provided name straight to `--space-id` or `ax projects list --space-id "<name>"`. | |
| **Space name as ground truth:** If the user tells you their space name, use it directly — do not run `ax spaces list` first to look it up. `ax spaces list` paginates and only returns the first page (~15 spaces); the target space may be on a later page and never appear. Pass the user-provided name straight to `--space` or `ax projects list --space "<name>"`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Checklist
npm startand verified thatREADME.mdis up to date.stagedbranch for this pull request.Description
Updates the Arize and Phoenix skills to the latest version.
Type of Contribution
Additional Notes
By submitting this pull request, I confirm that my contribution abides by the Code of Conduct and will be licensed under the MIT License.