Feature: CLI context orchestrator

[PredictabilityAtScale]

Context

Implements the tooling workflow from plans/feature-cli-context-orchestrator.md, allowing curated CLI utilities to feed IssueTriage assessments.

Goals

• Tool registration via settings plus curated defaults shipped with the extension.
• Panel UI and command palette entry to run tools, view outputs, and rerun on demand.
• Structured output capture with normalization and storage for downstream consumers.
• Telemetry instrumentation (success/failure/duration) respecting opt-out preferences.

Acceptance Criteria

• Workspace can register CLI tools through issuetriage.cliTools with command, timeout, output type, and optional parameters.
• Panel surfaces registered tools, runs them in a controlled environment, and displays the latest results with rerun support.
• Assessment flow can request the most recent tool output by logical name when composing prompts.
• Execution enforces runtime/output limits, sanitizes environment, and returns actionable errors without crashing.
• Tool result updates emit events for other components to subscribe to.

Dependencies

• Existing SettingsService and StateService for persistence.
• Telemetry and notification mechanisms shared with other features.

Notes

• Use child_process.spawn with shell off by default; document opt-in shell behavior and security warnings.
• Store artifacts in managed temp directories with TTL handling.

[PredictabilityAtScale]

IssueTriage Assessment

Model: openai/gpt-5-mini
Issue: PredictabilityAtScale/IssueTriage #8
View issue

Dimension	Score
Composite	60.0
Requirements	65.0
Complexity	55.0
Security	45.0
Business Impact	75.0

Summary: The issue defines a useful feature with clear high-level goals and acceptance criteria for registering/running CLI tools and surfacing outputs. However, many low-level specifications needed for safe, testable, and automatable implementation are missing (schemas, event/telemetry shapes, sanitization rules, TTL/storage details, UI API signatures). Without those, an autonomous agent cannot confidently implement the feature end-to-end or provide safe defaults. Supplying concrete interfaces, validation rules, and test fixtures will materially increase automation readiness.

Recommendations:

• Add a machine-readable JSON Schema for issuetriage.cliTools (fields: id/logicalName, command, args array, timeoutMs, outputType enum [json,text,lines], cwd optional, envWhitelist array, shell boolean default false, description). Include examples of curated defaults shipped with the extension.
• Specify the runtime API and storage contracts: exact StateService/SettingsService method signatures, event names and payload shapes (e.g. ToolRunStarted/Completed/Failed with toolId, timestamp, durationMs, exitCode, stdout/stderr snippets, artifactPath), and a function for assessment flow to request latest output (e.g. getLatestToolOutput(toolId): {meta, parsedOutput, artifactPath}).
• Define structured output normalization and limits: canonical output envelope (generatedAt, toolId, exitCode, durationMs, stdout, stderr, parsedJson optional), max capture size (bytes), parsing rules for JSON/text, TTL policy and temp-dir path policy, and precise error codes/messages to return to callers when limits or parsing fail.
• Document security and runtime safeguards: mandate child_process.spawn with shell=false by default, list which environment variables are allowed (or allow explicit envWhitelist per tool), require opt-in flag to enable shell mode with an explicit user-visible warning, and describe sandboxing / timeout / kill semantics and how to surface actionable errors without crashing the host.
• Add test fixtures and acceptance tests: provide mock CLI tools (deterministic scripts) and unit/integration tests that validate registration, run/kill behavior, output normalization, TTL cleanup, event emission, telemetry emission (success/failure/duration) and settings opt-out. Include one end-to-end UI/command-palette acceptance test that reruns a tool and retrieves an output for the assessment flow.

Last updated: 10/19/2025, 6:53:26 PM

[PredictabilityAtScale]

IssueTriage Assessment

Model: openai/gpt-5-mini
Issue: PredictabilityAtScale/IssueTriage #8
View issue

Dimension	Score
Composite	56.0
Requirements	65.0
Complexity	55.0
Security	45.0
Business Impact	75.0

Summary: The issue clearly defines the feature intent, goals, and acceptance criteria for registering and running curated CLI tools and surfacing their outputs, making a v1 implementation feasible. However, important integration contracts and runtime policies (SettingsService/StateService APIs, telemetry opt-out and event shapes, artifact storage/TTL) are unspecified and block safe, reproducible implementation. With answers to a few focused questions an autonomous agent can deliver a working v1. Security-sensitive defaults (shell, env whitelists, timeouts) are noted but need explicit opt-in/opt-out hooks.

Pre-implementation questions:

• Confirm whether existing SettingsService and StateService method signatures are available; if not, authorize the agent to introduce minimal APIs and state keys for registering tools and storing run artifacts.
• Specify which telemetry API and opt-out preference key to use (or provide minimal event names/payload shapes) so the agent can emit success/failure/duration telemetry conforming to project standards.
• Provide the artifact storage root path policy and default TTL (or confirm the agent may choose a managed temp dir with a sensible default TTL) and indicate whether an existing event bus/event names must be used for 'tool result updated' notifications.

Last updated: 11/3/2025, 1:17:59 PM

[PredictabilityAtScale]

IssueTriage Assessment

Model: openai/gpt-5-mini
Issue: PredictabilityAtScale/IssueTriage #8
View issue

Dimension	Score
Composite	56.0
Requirements	65.0
Complexity	55.0
Security	45.0
Business Impact	75.0

Summary: The issue clearly defines the feature intent, goals, and acceptance criteria for registering and running curated CLI tools, making a v1 implementation feasible. However, critical integration contracts and runtime policies (settings/state APIs, telemetry/event contracts, and artifact storage/TTL) are unspecified and block a safe, reproducible implementation. With answers to a few focused questions an autonomous agent can deliver a working v1. Security-sensitive defaults around child process execution further require explicit opt-in/opt-out guidance.

Pre-implementation questions:

• Can you confirm whether the existing SettingsService and StateService APIs/signatures are available for use, or explicitly allow the agent to introduce minimal settings/state keys and methods (e.g., for storing issuetriage.cliTools and tool run artifacts)?
• Which telemetry API and opt-out preference key should be used (or do you permit the agent to define minimal telemetry events and an opt-out key)? Also confirm whether specific event names/payload shapes are required for 'tool result updated' notifications or if the agent may define them.
• Please specify the artifact storage root path policy and default TTL for stored tool artifacts (or confirm the agent may use a managed temp directory with a sensible default TTL and configurable setting).

Last updated: 11/3/2025, 4:04:13 PM

[PredictabilityAtScale]

IssueTriage Assessment

Model: openai/gpt-5-mini
Issue: PredictabilityAtScale/IssueTriage #8
View issue

Dimension	Score
Composite	56.0
Requirements	65.0
Complexity	55.0
Security	45.0
Business Impact	75.0

Summary: The issue clearly defines the intent, goals, and acceptance criteria for registering and running curated CLI tools, making a v1 implementation feasible for an autonomous agent. However, critical integration contracts and opt-in security decisions are unspecified (SettingsService/StateService APIs, telemetry/event shapes, and artifact storage/TTL), which block a safe, reproducible implementation. Answering a few focused questions will unblock a working, secure v1.

Pre-implementation questions:

• Do we have existing SettingsService and StateService method signatures that must be used, or may the agent introduce minimal settings/state keys and APIs for issuetriage.cliTools and tool run artifacts?
• Which telemetry API and opt-out preference key should be used (or may the agent define minimal telemetry event names/payloads) so success/failure/duration can be emitted consistently?
• What artifact storage policy should be used for tool outputs (root path policy and default TTL), or may the agent use a managed temp directory with a sensible default TTL and a configurable setting?

Last updated: 11/13/2025, 1:50:32 PM