Skip to content

fix: activate Reporting compliance track with dedicated scenarios#1782

Merged
bokelley merged 4 commits intomainfrom
bokelley/fix-reporting-eval
Apr 1, 2026
Merged

fix: activate Reporting compliance track with dedicated scenarios#1782
bokelley merged 4 commits intomainfrom
bokelley/fix-reporting-eval

Conversation

@bokelley
Copy link
Copy Markdown
Contributor

@bokelley bokelley commented Mar 31, 2026

Summary

  • Activate the Reporting compliance track, which was permanently stuck at EXPECTED status because it had no test scenarios wired up
  • Bump @adcp/client to ^4.19.0 for the dedicated reporting_flow and deterministic_delivery scenarios
  • Fix buildScenarioList to include track-specific scenarios not in DEFAULT_SCENARIOS (without this fix, the new scenarios were silently filtered out)
  • Remove dead 'expected' track status from types, rendering, and logic

Context

ONX seller agent (Greg Morey) reported that Reporting track stayed at EXPECTED despite correct capabilities declaration and passing all other tracks. Root cause: TRACK_SCENARIOS['reporting'] was an empty array with a hard-coded special case returning 'expected'.

Test plan

  • Unit test: reporting track maps to reporting_flow + deterministic_delivery
  • Unit test: buildScenarioList(['reporting']) includes both scenarios
  • Unit test: buildScenarioList() includes all track scenarios
  • All 1014 server unit tests pass
  • Full test suite passes (pre-commit hook)
  • TypeScript compiles cleanly
  • CI passes

🤖 Generated with Claude Code

bokelley and others added 2 commits April 1, 2026 06:26
The Reporting track was permanently stuck at EXPECTED status because
TRACK_SCENARIOS['reporting'] was an empty array with a hard-coded
special case. Map it to full_sales_flow (which already tests
get_media_buy_delivery when the agent exposes that tool) and remove
the dead 'expected' status from types and rendering.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
bokelley and others added 2 commits April 1, 2026 15:59
Bump @adcp/client to ^4.19.0 and map the reporting compliance track to
['reporting_flow', 'deterministic_delivery'] instead of ['full_sales_flow'].
This gives precise evaluation of reporting capabilities rather than
piggybacking on the broader sales flow.

Closes adcontextprotocol/adcp-client#412

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
buildScenarioList filtered against DEFAULT_SCENARIOS, which silently
dropped reporting_flow and deterministic_delivery since they aren't in
the client's default list. Now appends track-specific scenarios after
the default ordering. Added integration test to catch this class of bug.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@bokelley bokelley changed the title fix: activate Reporting compliance track fix: activate Reporting compliance track with dedicated scenarios Apr 1, 2026
@bokelley bokelley merged commit 6db2868 into main Apr 1, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant