Repository navigation
fix(harness): command arms send the task instruction as written, no dokimi wrapper - #401
Conversation
…okimi wrapper
The 'Solve this coding task...' line and '# Task: <name>' header that
_task_prompt put around {prompt} are text a user's agent never sees.
Pass the bare instruction, as the real harnesses do.
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues block the change to pass task instructions directly to command and Harbor-style harnesses.
No open issues 🎉
What changed in this PR
0 new findings; nothing to fix.
This PR removes dokimi’s prompt wrapper from command and Harbor-style harnesses so agents receive only the task instruction. Docker-Claude arms are unchanged.
Changes:
- Makes
_task_promptreturntask.instructiondirectly. - Adds a regression test verifying the exact instruction reaches command arguments.
- Documents the prompt change and its effect on comparisons with earlier runs.
| File | Description |
|---|---|
dokimi/tests/test_task_harnesses.py |
Tests that command arguments contain the unwrapped instruction. |
dokimi/dokimi/harnesses/command.py |
Removes the introductory wrapper and task heading. |
dokimi/changelog.d/20261104_120000_napetrov_raw-task-prompt.md |
Records the behavior change and historical prompt-comparability caveat. |
Review effort: Balanced
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 3 selected item(s). CI at `f36cef8`: 3 still running, 11 passing, 2 skipped
Review details
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #401 +/- ##
========================================
Coverage 94.75% 94.76%
========================================
Files 352 352
Lines 46507 46407 -100
Branches 7970 7949 -21
========================================
- Hits 44070 43976 -94
+ Misses 1544 1541 -3
+ Partials 893 890 -3
🚀 New features to boost your workflow:
|
test_task_harnesses.py went past its 1000-line cap; the check is a test of command.py, so it gets its own file. Refs intel#401 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…raw-prompt Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
No change. This review has 0 findings, so there is no claim about the code to answer. The round only merges main (8d54c93) into the branch. No change. This report has 0 findings across 3 files, so there is no claim to answer. No change. This is the Codecov report: modified lines are fully covered. The "behind main" note is informational, not a code claim. |
antonwolfy
left a comment
There was a problem hiding this comment.
LGTM. _task_prompt returns the bare instruction, it has one caller (build_command), and the new test fails against the old wrapper. Two nits inline.
What I ran, on a Linux review host at c4e06c9:
make ci-localwith Docker: 35 passed, 0 failed, 7 skipped (jobs that do not run on a PR, or need tools absent here). Thechangelogstep passes.test_command_prompt.py,test_task_harnesses.pyandtest_payload_audit.py: 69 passed.- A mutation run of the new test against 9 variants of
_task_promptand theprompt_*context.
Checked, no issues
- Wrapper gone: a grep for "Solve this coding task", "Work only inside" and "# Task:" finds only the two changelog entries and the new docstring.
- Other arms: docker-claude builds its prompt in
docker_solver.pyfrom the notice plustask.instruction, so it is unaffected. - Commits: both non-merge commits carry
Signed-off-by.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Thread: PRRT_kwDOVBg7g860FaDg: already fixed in dba386c Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g860FaDn: already fixed in cdcb9c8 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
LGTM
I checked f36cef8. A command arm now gets instruction.md byte for byte, the same text a docker-claude arm gets without its skill notice, and the tests pin it.
Merge: safe now: it merges cleanly into current main, 0 commits behind, and no other open PR touches its files.
What I ran, at f36cef8:
make ci-local --offline: 19 passed, 0 failed, 13 skipped (jobs that need the network or Docker, or do not run on a PR).tests/test_command_prompt.pyon the merge-base 272d0b2: both tests fail against the old wrapper.- Mutations of
_task_promptand the prompt context:.strip(),splitlines()[0],prompt_escaped = promptandprompt = ""each fail the multi-line test. - The Docker smoke (
docker-oracleanddocker-probeon the pinned catalog task): 2/2 pass, every isolation probe closed.
Checked, no issues
- Callers:
_task_prompthas one caller,build_command; Harbor-backed arms pass{task_path}and readinstruction.mdthemselves, and the repo profiles (repos/profiles/*.yaml) build their prompt elsewhere, so neither changes. - Earlier nits: the changelog is scoped to command arms and says
{prompt_escaped}doubles braces (dba386c); the test covers all three placeholders on a multi-line instruction with a trailing newline (cdcb9c8). - Changelog:
Changedis the right category for a prompt change that moves scores, and the entry says old command-arm runs are not comparable.
Summary
{prompt},{prompt_quoted}and{prompt_escaped}in a command arm's template now expand to the bareinstruction.md. The wrapper is text a user's agent never sees, so it can move a command arm's score for no reason the task measures.Evidence
test_command_prompt_is_the_raw_instructionassertsargv == ["echo", "Fix the off-by-one in sum()."].tests/test_task_harnesses.pyandtests/test_payload_audit.py: 69 passed. ci-local static and typecheck: 0 failed.Merge Danger
Door: two-way. The prompt text is a template input; reverting restores the wrapper.
Command-arm runs measured before this change are not comparable in their prompt text (noted in the changelog fragment).
Blast Radius: command and Harbor-style arms only (
_task_prompthas one caller). docker-claude arms are not affected.