Add account-independent AgentCore evaluation benchmark

[longwind48]

TLDR

Add an opt-in, account-independent answer-quality benchmark using real Bedrock inference and AgentCore Evaluations.

• Run 12 versioned synthetic CloudOps cases through Strands and the shared production prompt.
• Score Helpfulness, Faithfulness and ground-truth Correctness; keep a deliberately wrong-answer sanity case separate.
• Capture approved content locally. Preserve production metadata-only telemetry. Require explicit profile, Region, model and paid-operation opt-in.
• Commit sanitized responses, spans, evaluator metadata, usage, mappings and reproducible commands.

User Flow

Before

1. Deploy and invoke the sample.
2. Inspect metadata-only traces. No repeatable answer-quality benchmark or measured baseline is available.

After

1. Configure your own AWS profile, supported Region and model.
2. Follow evaluations/README.md: uv sync --directory evaluations --locked, then runner.py invoke, score and report.
3. Inspect per-case evidence and separate metric summaries. Re-score saved traces without invoking the agent again.

Relevant issues

Refs #24. The owner approved the fixture-backed baseline and test boundaries. New reference-answer wording remains for maintainer review.

Pre-Submission checklist

• One specific problem; no evaluation service/dashboard or production content export.
• Relevant local tests pass and evaluation boundary regressions are covered.
• Required remote CI checks pass — previous tip passed; checks for the updated evidence commit are pending.
• Documentation covers setup, costs, permissions, retention and limitations.
• Evidence identifies revisions, Region, model and hashes without account credentials.

Proof of implementation

Before (fd57ec7)

No evaluation runner/dataset or measured README quality metrics. Production exports metadata only.

After (cbc3c6b)

The baseline was rerun in the owner's aiops_demo account after verifying STS identity against the intended demo account. Capture and scoring used clean commit bae50fe; cbc3c6b replaces the evidence and documentation. The earlier prod-profile run used the wrong account and is superseded, not relabeled. No account IDs or credentials are included in the public artifact.

Real AWS run on 2026-09-18 in us-east-1, model us.anthropic.claude-sonnet-4-5-20250929-v1:0, Strands 1.20.0, temperature 0. All 12 cases were invoked again in the demo account and all 39 judge calls completed there. Earlier clean-checkout installation and replay verification were performed in the other account. The shared production prompt remains unchanged.

Metric	Returned scale	Completed / expected	Mean	Labels
Helpfulness	0–1	12 / 12	0.9292	Above And Beyond: 7; Very Helpful: 5
Faithfulness	0–1	12 / 12	0.9583	Completely Yes: 10; Generally Yes: 2
Correctness	0–1	12 / 12	0.9167	Correct: 11; Incorrect: 1

Zero execution failures/skips per metric. Wrong-answer sanity case: zero on all metrics, excluded above. No pass threshold or combined score. GetEvaluator's Helpfulness metadata advertises 0–6, while Evaluate returned normalized 0–1 values; both retained without conversion.

Evidence: evaluations/evidence/baseline-2026-09-18.json and .md. Observed shortcomings include unsupported claims after empty results and a missing inventory scope caveat. The demo baseline replaces the previous account’s evidence. Built-in judges remain nondeterministic.

Console-visible batch verification (d5ee83d)

The owner requested console visibility. evaluations/batch.py now publishes only the approved synthetic traces into an isolated seven-day-retention log group, waits for Logs Insights visibility, and creates separate baseline and sanity jobs in aiops_demo/us-east-1. Production telemetry remains metadata-only.

Verified in the console and through GetBatchEvaluation:

• cloudops_baseline_20260918_verified-e861b35767: COMPLETED, 12/12 sessions, 36 evaluator results, no failures/ignored sessions. Service summary means: Helpfulness 0.96, Faithfulness 0.96, Correctness 0.92 (API-rounded).
• cloudops_sanity_20260918_verified-aabfbcd479: COMPLETED, 1/1 session, three results, all scores zero.
• Full end-to-end command replay also succeeded: jobs cloudops_baseline_20260918_063523-5623e46c00 and cloudops_sanity_20260918_063523-d9f9bd86d8, with all 39 matching result events downloaded.

Four earlier setup jobs failed because batch queries ran before Logs Insights could see uploaded spans. They remain visible for diagnosis. GetLogEvents visibility was insufficient; the runner now waits for completed Logs Insights counts before job creation. No scores from those failed jobs are used.

Batch evidence: evaluations/evidence/console-batch-2026-09-18.json, containing job metadata, references and all 39 actual per-session output events. Account IDs in ARNs are replaced with ACCOUNT_ID. Screenshot is cropped only to omit account/user/browser chrome; failed setup jobs remain visible.

The original on-demand workflow still creates no console job. The optional batch path now creates jobs, CloudWatch result events and evaluation metrics; it does not create an online evaluation configuration or alter production Runtime telemetry.

Local verification

• Evaluation boundary suite: 10 passed; mypy clean for runner.py and batch.py.
• Agent suite: 107 passed, 7 live-configuration skips, 14 live tests deselected; includes metadata-only OTLP export regression.
• CDK: 15 passed; TypeScript build passed.
• Frontend: 41 passed; production build passed with existing bundle-size warning.
• Lambda suites: 20 passed.
• Inventory: 6 deployed-source tests and 2 standalone tests passed.
• Standalone EOL scraper: 70 passed; existing inventory utcnow deprecation warnings.
• Verified 39 reference/result mappings, source hashes, report regeneration and sensitive-marker exclusion.

Type

• New feature
• Documentation
• Test

Caveats

• Fixture-backed answer quality only. No deployed Gateway/IAM, live inventory, current AWS pricing, Runtime health or persistent Memory validation.
• Live execution now covered two accounts in one Region; the published baseline is the intended demo account. Fixed fixtures and explicit configuration avoid account-specific expected facts, but model access and SCP/Region support remain prerequisites.
• Batch input log groups have seven-day retention; job records and service-managed output logs require explicit retention/cleanup. No shared log groups are deleted by the runner.
• New reference wording awaits maintainer review. Synthetic prices/lifecycle dates are not vendor guidance.
• No deployment, destructive outage testing, Docker/MCP rebuild or live Gateway security tests performed.
• Content allowlists and secret-pattern rejection are defense in depth, not a general customer-data anonymizer. Only committed synthetic fixtures are approved.

QA runbook

Use evaluations/README.md for exact locked install, offline test, invoke, score, replay and report commands. All paid commands require explicit opt-in. Expect 12 baseline results per metric plus one separate sanity result; missing content/references/results must fail. On-demand creates no cloud resources. Batch creates isolated fixture input logs and job records; follow the documented exact-target cleanup. Local raw batch evidence contains account ARNs and stays gitignored.

Final Attestation

• Evidence provenance, later corrections, skipped checks and visibility limitations are disclosed. Standards and spec were reviewed separately by one reviewer; parallel subagents were unavailable.

[longwind48]

Corrected the AWS account in cbc3c6b. Verified aiops_demo identity, reinvoked all 12 cases, and completed all 39 real AgentCore judge calls there. Replaced the baseline artifacts and README results: Helpfulness 0.9292, Faithfulness 0.9583, Correctness 0.9167; all three wrong-answer sanity scores remain 0. The prior prod-profile run was in the wrong account. Console visibility is unchanged: these are direct Evaluate calls with local capture, not batch jobs or CloudWatch-exported sessions.

[longwind48]

Console-visible batch workflow is implemented and verified in aiops_demo/us-east-1 (d5ee83d). Look for cloudops_baseline_20260918_verified and cloudops_sanity_20260918_verified: both show Success. The PR body now includes the cropped console screenshot and links to all 39 actual per-session result events. Baseline: 12/12 complete; sanity: 1/1 complete and all scores zero. Four failed ingestion-timing setup jobs remain visible; corrected readiness polling and a subsequent full command replay succeeded.