Skip to content

Make 6-epoch harness evolution results real & reproducible #19

Description

@OnePunchMonk

Summary

The current 6-epoch results in results/harness_evolution_6epochs_results.json and benchmark numbers were generated as simulations, not from actual script execution. This needs to be real and reproducible.

Related

Task

  1. Verify scripts/verify_tools.py loads the tool registry (with graceful fallback if APIs unavailable)
  2. Execute scripts/harness_evolution_6_epochs.py end-to-end with the fallback paths
  3. Regenerate all output files:
    • results/harness_evolution_6epochs_results.json
    • results/benchmark_report.json
    • .harness/v*.json evolved configs
  4. Compare new numbers against committed versions - if they differ materially, update:
    • README.md results table (line ~24)
    • HARNESS_EVOLUTION_RESULTS.md metrics (all sections)
    • Mermaid diagram in README

Files to Check/Modify

  • scripts/verify_tools.py — Run to verify tool registry
  • scripts/harness_evolution_6_epochs.py — Execute this
  • results/harness_evolution_6epochs_results.json — Output file
  • results/benchmark_report.json — Output file
  • .harness/*.json — Evolved configs

Acceptance Criteria

  • verify_tools.py runs without errors
  • Evolution scripts complete and produce JSON output
  • Output files match committed versions OR all references updated to match new numbers
  • No stale simulated numbers left alongside real pipeline
  • Test the results by running the scripts locally

Notes

This is the critical blocker. The headline claim (+37.4% Sharpe) must be reproducible, not simulated.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions