Skip to content

fix(harness): command arms send the task instruction as written, no dokimi wrapper - #401

Merged
napetrov merged 9 commits into
intel:mainfrom
napetrov:feat/cmd-raw-prompt
Nov 5, 2026
Merged

napetrov merged 9 commits into
intel:mainfrom
napetrov:feat/cmd-raw-prompt

Conversation

@napetrov

@napetrov napetrov commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Summary

 def _task_prompt(task: TaskSpec) -> str:
-    return (
-        "Solve this coding task. Work only inside the task environment. "
-        "When finished, leave the files ready for the verifier.\n\n"
-        f"# Task: {task.name}\n\n{task.instruction}"
-    )
+    return task.instruction

{prompt}, {prompt_quoted} and {prompt_escaped} in a command arm's template now expand to the bare instruction.md. The wrapper is text a user's agent never sees, so it can move a command arm's score for no reason the task measures.

Evidence

  • Before: a command arm's argv carries "Solve this coding task…" and "# Task: " around the instruction.
  • After: test_command_prompt_is_the_raw_instruction asserts argv == ["echo", "Fix the off-by-one in sum()."]. tests/test_task_harnesses.py and tests/test_payload_audit.py: 69 passed. ci-local static and typecheck: 0 failed.

Merge Danger

Door: two-way. The prompt text is a template input; reverting restores the wrapper.

Command-arm runs measured before this change are not comparable in their prompt text (noted in the changelog fragment).

Blast Radius: command and Harbor-style arms only (_task_prompt has one caller). docker-claude arms are not affected.

…okimi wrapper

The 'Solve this coding task...' line and '# Task: <name>' header that
_task_prompt put around {prompt} are text a user's agent never sees.
Pass the bare instruction, as the real harnesses do.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov
napetrov requested a review from a team as a code owner November 4, 2026 21:58
Copilot AI balanced review requested due to automatic review settings November 4, 2026 21:58

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block the change to pass task instructions directly to command and Harbor-style harnesses.

No open issues 🎉

What changed in this PR

0 new findings; nothing to fix.

This PR removes dokimi’s prompt wrapper from command and Harbor-style harnesses so agents receive only the task instruction. Docker-Claude arms are unchanged.

Changes:

  • Makes _task_prompt return task.instruction directly.
  • Adds a regression test verifying the exact instruction reaches command arguments.
  • Documents the prompt change and its effect on comparisons with earlier runs.
File Description
dokimi/​tests/​test_task_harnesses.py Tests that command arguments contain the unwrapped instruction.
dokimi/​dokimi/​harnesses/​command.py Removes the introductory wrapper and task heading.
dokimi/​changelog.d/​20261104_120000_napetrov_raw-task-prompt.md Records the behavior change and historical prompt-comparability caveat.

Review effort: Balanced

@exai-bot

exai-bot commented Nov 4, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 3 selected item(s).

CI at `f36cef8`: 3 still running, 11 passing, 2 skipped
Check Status
CLI interface diff still running
CodeQL (actions) still running
CodeQL (python) still running
Review details
  • Commit reviewed: f36cef8 against merge base 272d0b2
  • Coverage: 3 files in 1 groups
  • Engine: OpenCodeReview v1.13.5-exai.3.d7112371, us.openai.gpt-6.1-sol
  • Usage: 41k tokens (26k from cache), 5s
  • Tool calls: 2 (2 file_read)
  • small change set: 3 files

@codecov-commenter

codecov-commenter commented Nov 4, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.76%. Comparing base (8d54c93) to head (f36cef8).
⚠️ Report is 2 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@           Coverage Diff            @@
##             main     #401    +/-   ##
========================================
  Coverage   94.75%   94.76%            
========================================
  Files         352      352            
  Lines       46507    46407   -100     
  Branches     7970     7949    -21     
========================================
- Hits        44070    43976    -94     
+ Misses       1544     1541     -3     
+ Partials      893      890     -3     
Files with missing lines Coverage Δ
dokimi/dokimi/harnesses/command.py 97.10% <100.00%> (ø)

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

test_task_harnesses.py went past its 1000-line cap; the check is a test of command.py, so it
gets its own file. Refs intel#401

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…raw-prompt

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

napetrov commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

No change. This review has 0 findings, so there is no claim about the code to answer. The round only merges main (8d54c93) into the branch.

No change. This report has 0 findings across 3 files, so there is no claim to answer.

No change. This is the Codecov report: modified lines are fully covered. The "behind main" note is informational, not a code claim.

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. _task_prompt returns the bare instruction, it has one caller (build_command), and the new test fails against the old wrapper. Two nits inline.

What I ran, on a Linux review host at c4e06c9:

  • make ci-local with Docker: 35 passed, 0 failed, 7 skipped (jobs that do not run on a PR, or need tools absent here). The changelog step passes.
  • test_command_prompt.py, test_task_harnesses.py and test_payload_audit.py: 69 passed.
  • A mutation run of the new test against 9 variants of _task_prompt and the prompt_* context.

Checked, no issues

  • Wrapper gone: a grep for "Solve this coding task", "Work only inside" and "# Task:" finds only the two changelog entries and the new docstring.
  • Other arms: docker-claude builds its prompt in docker_solver.py from the notice plus task.instruction, so it is unaffected.
  • Commits: both non-merge commits carry Signed-off-by.

Comment thread dokimi/tests/test_command_prompt.py Resolved
antonwolfy and others added 6 commits November 5, 2026 13:25
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Thread: PRRT_kwDOVBg7g860FaDg: already fixed in dba386c
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g860FaDn: already fixed in cdcb9c8
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…dcb9c8

Review: PRR_kwDOVBg7g88AAAABWNlvRA: already fixed in dba386c and cdcb9c8
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

napetrov commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Both nits are fixed in dba386c and cdcb9c8 above; the rest is not a claim.

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I checked f36cef8. A command arm now gets instruction.md byte for byte, the same text a docker-claude arm gets without its skill notice, and the tests pin it.

Merge: safe now: it merges cleanly into current main, 0 commits behind, and no other open PR touches its files.

What I ran, at f36cef8:

  • make ci-local --offline: 19 passed, 0 failed, 13 skipped (jobs that need the network or Docker, or do not run on a PR).
  • tests/test_command_prompt.py on the merge-base 272d0b2: both tests fail against the old wrapper.
  • Mutations of _task_prompt and the prompt context: .strip(), splitlines()[0], prompt_escaped = prompt and prompt = "" each fail the multi-line test.
  • The Docker smoke (docker-oracle and docker-probe on the pinned catalog task): 2/2 pass, every isolation probe closed.

Checked, no issues

  • Callers: _task_prompt has one caller, build_command; Harbor-backed arms pass {task_path} and read instruction.md themselves, and the repo profiles (repos/profiles/*.yaml) build their prompt elsewhere, so neither changes.
  • Earlier nits: the changelog is scoped to command arms and says {prompt_escaped} doubles braces (dba386c); the test covers all three placeholders on a multi-line instruction with a trailing newline (cdcb9c8).
  • Changelog: Changed is the right category for a prompt change that moves scores, and the entry says old command-arm runs are not comparable.

@napetrov
napetrov merged commit 94ad4db into intel:main Nov 5, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants