Skip to content

feat(evaluation): add DeepSeek multi-agent incremental ablation - #122

Merged
Atlas2005 merged 1 commit into
mainfrom
pr122-deepseek-multi-agent-incremental-ablation
Jul 12, 2026
Merged

feat(evaluation): add DeepSeek multi-agent incremental ablation#122
Atlas2005 merged 1 commit into
mainfrom
pr122-deepseek-multi-agent-incremental-ablation

Conversation

@Atlas2005

Copy link
Copy Markdown
Owner

Summary

  • add a PIT-safe incremental ablation comparing:
    • non-LLM baseline
    • single-agent advisory
    • independent multi-agent advisory
    • research-committee multi-agent advisory
  • reuse the existing DeepSeek advisory and research committee infrastructure
  • add offline replay support using deterministic cached advisory outputs
  • record decision changes, fallback/invalid-output ratios, token usage, estimated API cost, and net performance after API cost
  • keep the runner offline by default with zero live requests unless explicitly enabled

Validation

  • 63 focused tests passed
  • 1607 full-suite tests passed
  • PR121 full-A artifact replay completed successfully
  • 5 common OOS folds
  • 5 PIT-safe decision packets
  • 4 expected ablation arms
  • 0 physical model calls in offline replay
  • committee provenance recorded
  • offline replay validation passed

Offline replay result

The deterministic synthetic advisory replay produced:

  • non-LLM compounded return: 16.81%
  • single-agent compounded return: 20.36%
  • multi-agent compounded return: 21.65%
  • benchmark compounded return: 31.86%

The final recommendation remains no_ai_arm because no AI arm met the evaluation thresholds and the replay used synthetic advisory evidence. No profitability claim is made from this offline fixture.

Boundaries

  • no broker interaction
  • no live DeepSeek calls by default
  • no test-period outcome leakage
  • no generated artifacts committed

@Atlas2005
Atlas2005 merged commit d47fe94 into main Jul 12, 2026
1 check passed
@Atlas2005
Atlas2005 deleted the pr122-deepseek-multi-agent-incremental-ablation branch July 12, 2026 06:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant