Skip to content

feat(adapters): add Harbor CLI integration - #19

Open
FreakyAdy wants to merge 8 commits into
masterfrom
feat/harbor-adapter-20261019
Open

FreakyAdy wants to merge 8 commits into
masterfrom
feat/harbor-adapter-20261019

Conversation

@FreakyAdy

Copy link
Copy Markdown
Owner

Summary

Adds a first-party adapter for running local Harbor task directories from EvalGuard without importing Harbor's Python dependency tree.

  • Maps stable EvalGuard task IDs to local Harbor task directories and validates task.toml before execution.
  • Invokes harbor run with an explicit agent, model, jobs directory, unique job name, and one attempt using argument-list subprocess execution (no shell).
  • Reads per-trial verifier_result.rewards and derives pass/fail from a configurable reward key and threshold; a zero exit alone is not treated as success.
  • Reports timeout, startup, CLI-exit, and missing-reward errors; retains result files for audit snapshots.
  • Adds mocked CLI tests for success, reward threshold, CLI error, missing reward, timeout, and invalid task mapping, plus adapter documentation.

Validation

Tests are mocked and do not require Harbor credentials, Docker, or installation of Harbor in EvalGuard's environment. GitHub Actions will validate lint, typing, unit tests, and coverage.

Scope

This contributes the Harbor half of #4. NeMo Gym is intentionally left for a separate adapter because its Python requirement and server protocols need a separate compatibility boundary.

@FreakyAdy
FreakyAdy marked this pull request as ready for review October 19, 2026 13:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant