Skip to content

[codex] collect server metrics during benchmarks - #17

Draft
jamesdborin wants to merge 4 commits into
mainfrom
codex/server-metrics-scraping
Draft

[codex] collect server metrics during benchmarks#17
jamesdborin wants to merge 4 commits into
mainfrom
codex/server-metrics-scraping

Conversation

@jamesdborin

Copy link
Copy Markdown
Contributor

Summary

  • add JSONL dataset support and request model injection for CLI benchmarks
  • ignore experiments/ so local benchmark artifacts do not get staged accidentally
  • add server-side Prometheus metrics scraping during benchmark runs

Metrics details

  • adds --metrics-output-dir, --metrics-endpoint, --metrics-interval-ms, --metrics-timeout-ms, and --metrics-fail-on-error
  • captures a baseline scrape before workers start and writes raw scrapes, parsed samples, metadata, and a derived summary
  • exposes metrics scrape metadata through BenchmarkReport, Python bindings, and CSV output
  • updates the Prime harness to copy remote metrics artifacts back into the local run directory

Validation

  • cargo test --manifest-path rust/Cargo.toml
  • python3 -m py_compile src/batchbench/harness.py
  • local smoke test with mock /v1/chat/completions and /metrics endpoint

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant