Improve evaluation sample test recordings and LLM validation

[aprilk-ms]

Changes (4 files)

1. Replace unsafe inline data in sample (sample_evaluations_builtin_with_inline_data.py)

• Replaced violent/provocative test data with benign content (health tips, writing skills)
• The original data triggered LLM content safety filters during validation, causing persistent test failures

2. Improve LLM validation prompt (test_samples_evaluations.py)

• Rewrote evaluations_instructions prompt to reduce false positives:
  • Clarify that JSON metric counters ("failed": 0, "error": null) are normal output
  • Explicitly state absence of crash means success
  • List specific failure indicators (Python tracebacks, FAILED_EXECUTION)
• Added _annotate_eval_metric_fields() preprocessor function scoped to evaluation tests only
• Added 2 missing samples to skip list per PR Howie/recording 7 #45412 review feedback:
  • sample_evaluations_builtin_with_dataset_id.py (requires Blob Storage prerequisite)
  • sample_continuous_evaluation_rule.py (requires manual RBAC assignment)

3. Add generic validation text preprocessing support (sample_executor.py)

• Added validation_text_preprocessor callback parameter (domain-agnostic) to allow test-specific text transformations before LLM validation

4. Re-recorded all 24 evaluation sample tests (assets.json)

• All recordings updated with new prompt and safe inline data
• Tag: python/ai/azure-ai-projects_e4ec8a475a
• All 24 tests pass in playback (~96s)

Addresses

• Review feedback from PR Howie/recording 7 #45412

[Copilot]

Pull request overview

This PR improves the reliability of evaluation sample test recordings and reduces false-positive LLM validation failures. It replaces unsafe/provocative test data in a sample with benign content, rewrites the LLM validation instructions to better handle evaluation-specific output patterns, and introduces a validation_text_preprocessor callback parameter to allow domain-specific text transformations before LLM validation.

Changes:

• Rewrote the evaluations_instructions prompt and added _annotate_eval_metric_fields() to reduce LLM false positives caused by JSON metric counters like "failed": 0 being mistaken for actual errors
• Added validation_text_preprocessor: Optional[Callable[[str], str]] parameter to BaseSampleExecutor, SyncSampleExecutor, and AsyncSampleExecutor
• Replaced violent/provocative inline test data with benign health and writing content; added 2 samples to the skip list; updated asset tag for re-recorded tests

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
test_samples_evaluations.py	Rewrote LLM validation prompt, added _annotate_eval_metric_fields preprocessor, added 2 entries to skip list, passed preprocessor to all 3 test executors
sample_executor.py	Added validation_text_preprocessor optional parameter to all executor classes; applied it in _build_validation_text; minor refactoring of _capture_print
sample_evaluations_builtin_with_inline_data.py	Replaced violent inline test data with safe health/writing content
assets.json	Updated recording tag to python/ai/azure-ai-projects_e4ec8a475a for re-recorded tests

---

You can also share your feedback on Copilot code review. Take the survey.