[fix] Align azd ai agent optimize/eval with the v2 optimization API

[Zyysurely]

Align azd ai agent optimize/eval with the v2 optimization API

This change adapts the agent optimize and eval flows to the new agent_optimization_jobs v2 service contract, modernizes the local eval.yaml schema, and tightens a few UX rules around required inputs.

Why

The optimization service moved to a new v2 contract: different endpoint paths, snake_case payloads, a reshaped job-status response, and a new dataset/evaluator structure. The CLI request/response models, the local config schema, and several command flows needed to be updated to match. Along the way we also cleaned up a few stale defaults and prompts.

API contract (snake_case + new shapes)

• Casing: all optimize request/response fields are now snake_case.
• Endpoints: moved from /optimize to /agent_optimization_jobs. Candidate endpoints (config / files / promote) are now nested under the job ID (/agent_optimization_jobs/{jobId}/candidates/{candidateId}/...).
• Datasets: inline datasets are folded into train_dataset with a type discriminator (inline | reference); validation_dataset follows the same shape.
• Evaluators: now carry an optional version (previously just a name string).
• optimization_config keys: baselineModel → model, the model search list model → model_search_space, systemPrompt → system_prompt.
• Job status: reshaped to id / inputs / result. inputs reuses the request type (it's the echoed request); result.baseline / result.best are candidate IDs resolved against result.candidates; progress is candidates_completed; created_at / updated_at are unix timestamps.
• Removed pass_rate and rationale from candidate results (no longer returned).
• Added Foundry-Features: AgentsOptimization=V2Preview header on every optimization request via a per-call pipeline policy.
• OptimizeResponse accepts the job id returned by POST so polling gets a valid operation ID.

Local eval.yaml schema

• Renamed keys for clarity, with backward-compatible reads of the old keys:
  • dataset_reference → dataset
  • validation_reference → validation_dataset
• Dropped the explicit type field on local datasets — local vs. remote is inferred from whether name (remote) or local_uri (local) is set. Legacy dataset_file still works and takes precedence.
• Renamed max_iterations → max_candidates (direct rename, no compat shim).

Behavior / UX

• optimization_model is now required — it no longer falls back to the eval model. Validation fails fast with guidance to pass --optimize-model or set it in config.
• Removed the default evaluator (builtin.task_adherence). An evaluator must come from config or --evaluator; optimize's --evaluator flag now appends to (and dedupes against) config evaluators instead of replacing them.
• Per-agent job ID is persisted in the azd environment (AGENT_{KEY}_OPTIMIZATION_JOB_ID) so apply / deploy / post-deploy promote target the correct optimization job; falls back to the global last-job key.
• Results table adds a Strategy column (the candidate's mutation keys) placed last so it can grow freely without breaking alignment.
• eval generate skips the max-samples prompt when an existing dataset is supplied via --dataset (nothing to generate).
• Candidate config parsing accepts snake_case (system_prompt, agent_name) with camelCase fallbacks.

Tests

• Updated and extended unit tests across optimize_api, opt_eval, eval_api, and cmd (new contract round-trips, dataset/validation rename + backward compat, required optimization_model, evaluator merge/dedupe, per-agent job ID, candidate endpoint URLs, Strategy column, etc.).
• Full extension test suite passes; go build ./... is clean.

[Copilot]

Pull request overview

This pull request updates the Azure AI Agents extension’s optimize/eval command flows and API client/models to match the v2 agent_optimization_jobs service contract (new paths, snake_case payloads, reshaped job status/result), while also modernizing the local eval.yaml/optimize.yaml dataset schema and tightening validation/UX rules (required optimization model + evaluator selection, per-agent job ID persistence).

Changes:

• Updated optimization service request/response models + client endpoints (including candidate endpoints nested under job ID) and added the required Foundry-Features: AgentsOptimization=V2Preview header on requests.
• Updated local YAML config schema and helpers to use dataset / validation_dataset with backward-compatible reads for legacy keys, plus new helper methods for local/remote dataset resolution.
• Updated command flows and output formatting (job persistence per-agent, results table strategy column, prompt behavior tweaks) and expanded unit test coverage accordingly.

Reviewed changes

Copilot reviewed 31 out of 31 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
cli/azd/extensions/azure.ai.agents/internal/pkg/agents/optimize_api/poller_test.go	Updates polling tests to match v2 job status fields (id, candidates_completed).
cli/azd/extensions/azure.ai.agents/internal/pkg/agents/optimize_api/models.go	Refactors optimize API models to v2 shapes (snake_case, datasets/evaluators, result structure).
cli/azd/extensions/azure.ai.agents/internal/pkg/agents/optimize_api/models_test.go	Updates model round-trip tests for the new v2 wire format and helper methods.
cli/azd/extensions/azure.ai.agents/internal/pkg/agents/optimize_api/client.go	Updates optimize endpoints to /agent_optimization_jobs, nests candidate calls, adds Foundry-Features policy.
cli/azd/extensions/azure.ai.agents/internal/pkg/agents/optimize_api/client_test.go	Updates client tests for new endpoints/header and adds candidate endpoint tests.
cli/azd/extensions/azure.ai.agents/internal/pkg/agents/opt_eval/yaml.go	Modernizes shared eval/optimize YAML config schema + adds dataset normalization and helpers.
cli/azd/extensions/azure.ai.agents/internal/pkg/agents/opt_eval/yaml_test.go	Adds/updates tests for dataset inference, normalization, and max_candidates.
cli/azd/extensions/azure.ai.agents/internal/pkg/agents/eval_api/eval_config.go	Normalizes dataset legacy key on load and updates dataset validation logic.
cli/azd/extensions/azure.ai.agents/internal/pkg/agents/eval_api/eval_config_test.go	Updates eval config tests to the new dataset schema + precedence rules.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize.go	Updates optimize flags/overrides, job persistence, progress display, and results table formatting.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_test.go	Updates optimize command tests for evaluator requirements, dataset schema, and result formatting.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_status.go	Updates status output to use v2 fields + timestamp formatting.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_prompts.go	Updates dataset-required messaging and model search space key naming.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_prompts_test.go	Updates tests for model_search_space key behavior.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_list.go	Updates list output to use v2 job fields + formatted timestamps.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_helpers.go	Adds per-agent optimization job ID persistence/loading + updates postdeploy promotion to job-nested API.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_helpers_test.go	Updates helper tests for new endpoints and adds tests for per-agent job key naming.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_deploy.go	Updates deploy flow to resolve job ID and call job-nested candidate endpoints + promote.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_config.go	Updates optimize config schema/validation, request building, evaluator refs, dataset handling, and compat for validation dataset.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_config_test.go	Updates optimize config tests for v2 request shapes and new validation rules.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_apply.go	Updates apply flow to use job-nested candidate endpoints and accept snake_case candidate config keys.
cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_apply_test.go	Adds tests for snake_case system_prompt extraction precedence.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_update.go	Updates update flow to use new dataset field names.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_test.go	Updates eval command tests for new dataset schema.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_run.go	Updates eval run dataset source selection to use dataset helper methods and new messaging.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_helpers.go	Updates dataset-required prompt error messaging to match new schema wording.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_generate.go	Updates eval generate flow to write dataset via dataset.local_uri and use dataset helper methods for printing/links.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_generate_test.go	Updates eval generate tests for new dataset schema.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_generate_prompts.go	Skips max-samples prompt when using an existing dataset; updates dataset label logic for new schema.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_generate_jobs.go	Updates dataset reference assignment to the new dataset field.
cli/azd/extensions/azure.ai.agents/internal/cmd/eval_api_version_test.go	Updates API version tests for new dataset field naming.