Add Harbor Framework Support

[hrdkbhatnagar]

Adds Harbor framework support to PostTrainBench, enabling anyone to run our benchmark on cloud GPUs (Modal, Daytona) without needing access to our internal HTCondor cluster.

At the moment:

• Generate Harbor-compatible task directories from PostTrainBench benchmarks
• Almost full parity with original pipeline

Tested:

• Generated task for gsm8k + qwen3-1.7b
• Ran 1-hour test with Claude Code on Sonnet 4 on Modal
• Verified end-to-end pipeline (including eval + contam judge)
• Confirmed accuracy metrics extracted correctly

Usage

  cd src/harbor_adapter
  uv sync

  # Generate a task
  python run_adapter.py --benchmark gsm8k --model qwen3-1.7b --output ./tasks

  # Run with Harbor
  harbor run \
      --path ./tasks/posttrainbench-gsm8k-qwen3-1.7b \
      --agent claude-code \
      --model anthropic/claude-sonnet-4 \
      --env modal

See src/harbor_adapter/README.md for detailed parity tracking. Key points:

• Agent timeout, GPU access, evaluation: Full parity
• Contamination judge: Parity
• Agent duration: Tracked by Harbor in result.json
• timer.sh: Minor difference (created at task generation vs job start)

Note: Right now I have skipped the installation of flash-attn in the container as we need to have a CUDA runtime for it. In modal the GPU is attached to the sandbox after the container is built, so installation doesn't occur.

Note: I have added a uv environment for us to use in PTB. This is used for using modal and harbor, and is useful in general for reproducibility

Todos:

• directly before agent is run, install flash_attn and build timer.sh
• huggingface cache in a modal storage
• before running evaluation, uninstall and reinstall major dependencies (like transformers, inspect-ai, ...). Make sure NOT to use the cache. ((Alternatively we can look into docker in docker))

[hrdkbhatnagar]

pin versions like the current images

[hrdkbhatnagar]

things that are remaining to get full parity with the original PTB implementation:

1. Separate verifier container (eval integrity)
   In our setup, we evaluate the agent's post-trained model in a different container than the one the agent trained in. This prevents reward hacking, the agent could modify eval files in its workspace to inflate its score. In Harbor, the verifier runs in the same sandbox as the agent, so it uses whatever files the agent may have tampered with. We need a way to run the verifier in an isolated environment.

2. Pre-agent shell command inside the container
   We have a timer script that the agent calls to check remaining time (out of 10 hours). It needs to know when the agent actually started. In our original setup, the host orchestrator writes the start timestamp before launching the agent container. In Harbor, I see lifecycle hooks on the Trial object (TrialEvent.AGENT_START etc.), but those run on the orchestrator side, not inside the sandbox. we need a way to execute a shell command inside the container right before the agent starts , like a pre-agent hook.

3. Downloading additional directories after a run
   After the agent finishes, we'd like to download its full workspace (/home/agent/workspace/), including the code it wrote and the fine-tuned model weights. Currently Harbor only downloads /logs/agent and /logs/verifier.

[hrdkbhatnagar]

after discussing with Alex from Harbor/tbench:

1. we could put the verifier in the tests/ directory which only gets uploaded after the agent runs

2. this is not yet supported be we should look into this, potentially make a PR to harbor

3. Artifact collection is supported now in harbor, so we should use that

[rank-and-file]

Added modal storage for hf-cache for harbor in the branch add_harbor_support_with_hfcache. This is necessary to have the pre-cached models and datasets. This is necessary for full parity.

Although there are some other changes as well, so you can probably clone the repo with this branch in another directory and ask your agent:

Consider PostTrainBench_harbor_modal and port all changes which are related to the modal storage integration for the huggingface cache to PostTrainBench_harbor.

[rank-and-file]

Also there is an upcoming change to the judge which will need to be integrated. Will post here.

[rank-and-file]

We need to hardcode baseline values to a json, instead of fetching them from the POST_TRAIN_BENCH_RESULTS_DIR.

This is needed for harbor integration (harbor should output the baseline value, in case the judge flags the run).

[hrdkbhatnagar]

Merged main into Harbor branch, @rank-and-file maybe we should push the new judge to main soon so we can pull it here. The new judge would require some major changes for Harbor