feat(repo): repo tasks authoring builds hidden-test tasks from merged PRs by antonwolfy · Pull Request #425 · intel/dokimi

antonwolfy · 2026-11-06T18:54:21Z

Summary

Settles #294: repo tasks replay builds dokimi:review_replay tasks, which measure guidance for reviewing, but most guidance is read while writing code, and authoring tasks were written by hand. This adds repo tasks authoring, which builds one authoring task per merged pull request. It takes the same kind of input and leaves out the same files as review replay, and follows the decisions taken in the issue's comments (merged PRs only, one shared task writer, eval check decides, leaked-test lines dropped and counted, all-pass gate plus a passing share, flaky detection, public_at).

On main today no command writes such a task. The command needs a way to run the repository's tests on files held out of the workspace, which the manifest could not express until now. Built-in graders cannot start a process (spawns only in dokimi.process), so the framework runs the test command itself, and a stdlib grader reads its report.

Real output, on dokimi's own history. Five merged PRs, with titles and descriptions from their commit messages:

PR 392: skipped: its diff outside the tests changes 444 lines, more than --max-lines 400
PR 393: written authoring_pr393: 19 hidden test file(s), 276 line(s) outside the tests; the prompt dropped 0 line(s) naming a hidden test, 0 quoted block(s), 0 link(s)
PR 395: written authoring_pr395: 1 hidden test file(s), 308 line(s) outside the tests; the prompt dropped 0 line(s) naming a hidden test, 0 quoted block(s), 0 link(s)
PR 405: written authoring_pr405: 4 hidden test file(s), 155 line(s) outside the tests; the prompt dropped 0 line(s) naming a hidden test, 0 quoted block(s), 0 link(s)
PR 415: skipped: it adds or changes no test file
3 task(s) written, 0 kept, 2 pull request(s) skipped

eval check on each written task: OK, 1/1 ok. Both runs of each step gave the same result:

Task	base (null), both runs	oracle, both runs
`authoring_pr405`	220 of 229 passed, 9 failed, `pass: false` (`pass_share` 0.96)	229 of 229, `pass: true`
`authoring_pr395`	0 of 22 passed, `pass: false`	22 of 22, `pass: true`
`authoring_pr393`	1107 of 1118 passed, 11 failed, `pass: false` (`pass_share` 0.99)	1118 of 1118, `pass: true`

How these were run: the clone has no refs/pull/*/head, so each PR's squash commit stood in for its headRefOid. A merge commit with that commit's tree and parents (^1, the commit) stood in for its mergeCommit, so base was the commit before the merge. createdAt came from the commit's author date. Docker was not available, so the self-check ran through the test suite's FakeRunner, which runs each container's command on the host with the mounts mapped. The manifest got test: "python3 -m pytest -q -p no:cacheprovider --junitxml={out} {tests}" through --manifest-file.

Changes

repo tasks authoring <source> --prs FILE --out DIR [--manifest] [--manifest-file] [--max-lines N] [--prompt-file] [--force] [--cache-dir] (commands/repo_run.py, cli_routes.py, CLI snapshot, flag allowlist rows for --prs and --max-lines).
One task writer (repos/replay/authoring.py) takes a Change (base, target, title, description, its source's fields), so a later failure-ledger source (Configuration Effectiveness (agents.md#213): receipt, reach matrix, rule ledger, arms #342) can use the same writer. authoring_prs.py makes a Change from each merged PR.
- base is the merge base of headRefOid with mergeCommit^1.
- hidden/ gets the test files the PR adds or changes. hidden_tests is the subset a runner collects by name.
- reference/ and reference_deleted hold the rest of the diff (the oracle). oracle_lines is its size.
- Guidance and eval_dir files are listed in left_out_files. patch_files and public_at (the PR's createdAt) are set too, so repo compare flags these tasks by the replay rule (feat(repo): repo compare flags replay tasks a guidance revision may have seen #392) with no change to compare_leak.py.
- family is fix when the title starts with fix, else feature.
What review replay and authoring share (merged/head checks, the fast-forward skip, the merge base, the left-out rule) moved from replay/build.py into replay/merged.py. Replay's behaviour is unchanged.
Prompt (authoring_prompt.py): the title, then the description, then a fixed instruction that --prompt-file replaces. Three things are dropped from the description and counted in prompt_dropped:
- fenced blocks that are a diff or quote the change's lines;
- links to its commits, diff or PR page, and prefixes of its commit SHAs;
- lines naming a hidden test (its path, its file name, or a test* function or class the PR adds). A PR with more than 5 such lines is skipped.
The build's own filter: the PR has a hidden test, changes something outside its tests, and has at most --max-lines (default 400) lines there. A PR that fails it is listed with the reason (contract §4.7 has the table).
Manifest: optional entry_points.test, which must name {tests} and {out} (a JUnit XML report). {tests} is refused in other entry points. The key is required when a task uses dokimi:hidden_tests (manifest.py, repo_manifest.v1.json widened in place).
Built-in grader dokimi:hidden_tests (graders/hidden_tests.py, stdlib, Python 3.8):
- Before it runs, grading.run_entry copies the hidden files into the workspace, with their file mode. A file or link there is replaced; a linked directory on the way is refused. It then runs the test command in a grading-like container and records its exit.
- pass is true when the command exited 0 and every case ran and passed. Metrics: n_tests, n_passed, n_failed, n_errors, n_skipped, pass_share, tests_rc, tests_timed_out.
- A report that is missing, unreadable (including one the grader cannot open, an OSError), or declares a DOCTYPE fails the run.
- Its oracle copies reference/, reusing dokimi:typecheck's, and deletes reference_deleted through no link.
eval check runs null and oracle twice each for dokimi:hidden_tests tasks. A task whose two runs of a step differ is flaky and exits 1. Other tasks run as before.
History gains statuses, n_lines_changed, blob and commits. read_threads / load_pull_requests take the key that marks a PR object. PullRequest gains body and created.
Contract §2 (entry_points.test, {tests}), §4.3 (flaky), §4.5 (the grader), new §4.7 (every flag, field, skip reason and exit code), §9 (example); architecture.md.
Changelog fragment (Added).

Risk

Review replay's output is unchanged; its helpers moved to merged.py. The existing replay tests pass with their assertions unchanged; one gained a case.
repo_manifest.v1 gains an optional test key under entry_points. A manifest valid before stays valid. The placeholder {tests}, which no manifest could use before, is now refused outside entry_points.test.
Workspace prep, review graders, typecheck tasks and every other task's eval check behave as before. Only dokimi:hidden_tests tasks get the second runs and the flaky verdict.
The test command runs the subject's code with the run dir writable, as a repository's own grade does. A run that rewrites its JUnit report is not detected (said in §4.5).
Adding a grader module changes graders_digest(), so a pinned batch whose eval dir uses a built-in grader refuses to resume on this version, as with any grader change.

Testing

On a Linux review host, at d449291 (on main 25f776e):

make ci-local: 30 passed, 0 failed, 7 skipped. The skipped steps: external-catalog-smoke (needs docker), test-latest, docs deploy and pip-audit report (job conditions false on a PR), cli-surface packaging (an expression the local runner does not evaluate), conda-forge-deps (no micromamba), semgrep (offline). 16421 tests passed, 11 skipped. Changed-line coverage 100% (547 lines).
New tests/test_repo_authoring.py (53 tests, no network, fixture git repository built per test):
- a PR with a test becomes a task, and eval check (FakeRunner) passes it: base fails, oracle passes, twice each, pass_share 0.0 / 1.0;
- a flipping test makes the task flaky, 0/1 ok; a second oracle run that cannot run reads not graded and makes it flaky;
- prompt dropping and counting, and the links and hex words it keeps;
- more than 5 leaked lines are skipped;
- skips for no test file, only tests, fast-forward, missing mergeCommit or head, an empty diff, guidance only, a symbolic link, a path git gives that is not UTF-8, a head with no shared history;
- a deleted file in reference_deleted, and a quote of a removed line dropped;
- a remote source fetches the PR's head by SHA;
- --max-lines at and over the limit;
- guidance and eval_dir are left out;
- public_at with and without an offset;
- kept without --force, rewritten with it;
- bad input exits 1 with ✗: missing file, empty file, wrong shape, "number": null;
- the grader's pass and share rules; a missing, DOCTYPE, oversized or unopenable report; a missing or malformed run record; duplicate or uncovered hidden paths;
- the oracle's deletes, refusal through a link, and refusal of a bad task;
- staging through links, a bad task, a workspace that is no directory, a directory at a hidden path (recorded, the tests not run, stale outputs cleared), a hidden file's mode (0o755 and 0o640) kept over the workspace's, and no test entry point;
- the manifest's test rules, and repo validate on a missing hidden file.
tests/test_repo_replay.py: a reviewed revision with no history of the base branch is skipped with the reason.
Mutants: 31, each a fresh copy of the commit with one change to the code, each failing the test that pins it. Examples: the hidden-file copy dropped, the second eval check run dropped, the flaky comparison disabled, a non-zero test exit counted as a pass, skipped cases counted in pass_share, the DOCTYPE refusal removed, --max-lines > made >=, the fast-forward skip removed, leaked lines counted but kept. One mutant first survived (leaked lines kept in the prompt); the test was tightened and it fails now. Review round: 30 more, each failing: the grader before the unreadable-report fix, and 29 breaks of the code the new tests cover (each skip and refusal disabled, the report size limit > made >=, removed lines left out of the quote check, the remote fetch dropped, the ungraded second run's reading dropped). Second review round: 2 more, each failing the mode test: the staging before the mode fix (both files staged 0o664), and every staged file forced to 0o755.
Not run: eval check with docker (docker could not be used on this host), so no real image was built or run. The self-check above used the FakeRunner. No real gh api graphql output either: the PR file was written by hand from the commits.

Revert

Revert the commits. Nothing persisted depends on it except tasks it wrote (authoring_pr<N>/ directories) and a manifest that uses entry_points.test, which the reverted schema would refuse.

Checklist

Tests added/updated and pytest passes
ruff, mypy, and schema checks pass
Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
Self-reviewed per "Before opening a PR" in AGENTS.md
No ad-hoc run artifacts committed
New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

Closes #294

… PRs Settles intel#294: review replay measures guidance for reviewing, while most guidance is read while writing code. `repo tasks authoring` writes one `dokimi:hidden_tests` task per merged pull request of a `gh api graphql` file, with the same input and the same left-out rules as replay. - base: the merge base of headRefOid with mergeCommit^1 (shared with replay in repos/replay/merged.py, with the guidance and eval_dir rule); a fast-forward merge is skipped. - prompt: the PR's title and description without fenced blocks that quote the change, links to its commits or diff, and lines naming a hidden test (counted in prompt_dropped; more than 5 skip the PR), then a fixed instruction that --prompt-file replaces. - hidden/: the test files the PR adds or changes; reference/ and reference_deleted: the rest of its diff, the oracle. - filter: a test file, a change outside the tests, and at most --max-lines (default 400) lines there; public_at is the PR's createdAt, so repo compare flags these tasks as it does replay tasks. Grading: the manifest gets an optional entry_points.test, the repository's test command, with {tests} and a JUnit report at {out}. For dokimi:hidden_tests the framework copies the hidden files into the workspace, runs that command in the grading container and records its exit; the stdlib grader reads the report: pass when every hidden test ran and passed, pass_share beside it. eval check runs null and oracle twice each for these tasks and reports differing runs as flaky (exit 1). Contract §2, §4.3, §4.5 and a new §4.7 document it. Closes intel#294 Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

codecov-commenter · 2026-11-06T19:08:11Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.07%. Comparing base (8a6207b) to head (4353b20).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #425      +/-   ##
==========================================
+ Coverage   94.99%   95.07%   +0.07%     
==========================================
  Files         366      371       +5     
  Lines       48636    49150     +514     
  Branches     8373     8452      +79     
==========================================
+ Hits        46204    46729     +525     
+ Misses       1539     1535       -4     
+ Partials      893      886       -7

Files with missing lines	Coverage Δ
dokimi/dokimi/cli_routes.py	`100.00% <ø> (ø)`
dokimi/dokimi/commands/repo_run.py	`95.72% <100.00%> (+0.51%)`	⬆️
dokimi/dokimi/repos/graders/__init__.py	`100.00% <100.00%> (ø)`
dokimi/dokimi/repos/graders/hidden_tests.py	`100.00% <100.00%> (ø)`
dokimi/dokimi/repos/graders/main.py	`82.85% <100.00%> (+0.24%)`	⬆️
dokimi/dokimi/repos/grading.py	`93.92% <100.00%> (+2.87%)`	⬆️
dokimi/dokimi/repos/manifest.py	`94.76% <100.00%> (+0.27%)`	⬆️
dokimi/dokimi/repos/replay/__init__.py	`100.00% <ø> (ø)`
dokimi/dokimi/repos/replay/authoring.py	`100.00% <100.00%> (ø)`
dokimi/dokimi/repos/replay/authoring_prompt.py	`100.00% <100.00%> (ø)`
... and 6 more

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open issues

The contract says a report that is missing or unreadable should fail the run (pass=false) rather… · New
{tests} expansion doesn’t match the contract when {tests} appears inside a longer token.… · New
Parsing of ISO 8601 timestamps with a trailing Z is not supported by datetime.fromisoformat on… · New

What changed in this PR

Adds infrastructure to generate “authoring” tasks (one per merged PR) and a new built-in grader that runs hidden tests during grading, enabling task creation from real PR history without manual authoring.

Changes:

Add repo tasks authoring to build dokimi:hidden_tests tasks from merged PRs (prompt sanitization + diff splitting into hidden tests vs oracle).
Introduce dokimi:hidden_tests grader and framework support to stage hidden files + run entry_points.test producing a JUnit report.
Extend manifest schema/validation and eval check to support hidden-test tasks (including flaky detection via double-runs).

File	Description
dokimi/tests/test_repo_authoring.py	Adds comprehensive tests for authoring task generation, prompt redaction, manifest rules, and hidden-test grading/selfcheck behavior.
dokimi/tests/data/cli_flags_allowlist.txt	Allowlists new CLI flags `--prs` and `--max-lines` for the new command.
dokimi/dokimi/schemas/repo_manifest.v1.json	Extends schema to allow `entry_points.test` for hidden test execution.
dokimi/dokimi/repos/selfcheck.py	Updates `eval check` logic to double-run null/oracle for hidden-test tasks and mark flaky.
dokimi/dokimi/repos/replay/threads.py	Generalizes PR JSON ingestion to support both replay threads and authoring PR files (body/createdAt).
dokimi/dokimi/repos/replay/merged.py	Centralizes merged-PR common logic (merge/head checks, merge base, left-out rules).
dokimi/dokimi/repos/replay/history.py	Adds git helpers for authoring pipeline (name-status, numstat, blobs, commit lists).
dokimi/dokimi/repos/replay/build.py	Refactors replay builder to use shared merged-PR helpers.
dokimi/dokimi/repos/replay/authoring_prs.py	Builds `Change` objects from merged PRs, including `public_at` handling.
dokimi/dokimi/repos/replay/authoring_prompt.py	Implements prompt redaction (quoted blocks, links/SHAs, test-name leaks) and counters.
dokimi/dokimi/repos/replay/authoring.py	Implements authoring task writer: hidden tests/files, oracle reference, filters, and task.json fields.
dokimi/dokimi/repos/replay/__init__.py	Updates module docs to cover both replay and authoring paths.
dokimi/dokimi/repos/manifest.py	Adds `{tests}` placeholder handling and validation; requires `entry_points.test` for hidden-test tasks; validates hidden files exist.
dokimi/dokimi/repos/grading.py	Adds framework support to stage hidden files and run `entry_points.test` before grading hidden-test tasks.
dokimi/dokimi/repos/graders/main.py	Registers the new `hidden_tests` grader/oracle with the built-in grader runner.
dokimi/dokimi/repos/graders/hidden_tests.py	New grader that interprets JUnit XML + run record and implements reference-copy + deletes oracle.
dokimi/dokimi/repos/graders/__init__.py	Exposes hidden-tests grader in built-in name lists and task-problem validation.
dokimi/dokimi/commands/repo_run.py	Adds CLI command implementation and argparse wiring for `repo tasks authoring`.
dokimi/dokimi/cli_routes.py	Exposes new free CLI route `repo tasks authoring`.
dokimi/docs/repo-evaluation-contract.md	Documents manifest `entry_points.test`, hidden-tests grader semantics, flaky detection, and authoring task generation contract.
dokimi/docs/reference/cli-interface.json	Updates CLI surface snapshot to include `repo tasks authoring` args.
dokimi/docs/architecture.md	Documents new authoring task pipeline and hidden-test execution flow.
dokimi/changelog.d/20261106_120000_antonwolfy_authoring-tasks.md	Changelog entry for the new command and grader.

_{Review effort: Lite}

Settles the review point that dokimi:hidden_tests turned a JUnit report it could not stat or read (an OSError, such as a permission the subject's code left on hidden-tests.xml) into a grader error with pass: null. Contract section 4.5 says an unreadable report is a failed run with no test case. The grader now catches OSError around the stat and the read and returns pass false, n_tests 0, with details.report saying why. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

Settles the codecov report on the changed lines no test reached. Each gets a test of what the code does there: - repo tasks authoring skips a PR whose diff is empty, only guidance, a symbolic link, a path git gives that is not UTF-8, or a head with no shared history; a deleted file lands in reference_deleted and a quote of a removed line is dropped; a remote source fetches the PR's head. - the prompt drops a diff block and change links, and keeps other links and hex words that are no commit prefix. - dokimi:hidden_tests: a missing or non-object run record, a report over the size limit, duplicate or uncovered hidden paths, and the oracle on a bad task. - staging: a bad task, a workspace that is no directory, a directory at the hidden path (recorded, the tests not run, stale outputs cleared), and a manifest with no test entry point. - eval check: a second oracle run that cannot run reads "not graded". - review replay: a reviewed revision with no history of the base branch. - History.n_lines_changed of no path is 0, not the whole diff. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

1 open issue

_stage_hidden() uses shutil.copyfile(), which does not preserve file mode bits. If a hidden… · New

3 resolved since last review

{tests} expansion doesn’t match the contract when {tests} appears inside a longer token.…
The contract says a report that is missing or unreadable should fail the run (pass=false) rather…
Parsing of ISO 8601 timestamps with a trailing Z is not supported by datetime.fromisoformat on…

_{Review effort: Lite}

Review point: _stage_hidden copied each hidden file with shutil.copyfile, which writes a new file with the default mode, so a hidden helper that authoring wrote 0o755 (a 100755 blob, as the contract's authoring section says it keeps the mode) reached the workspace as 0o664 and the test command could not run it as written. It now copies with shutil.copy, which also copies the mode bits, as the oracle's copytree of reference/ already does. The contract and the changelog fragment say the mode is kept, and a test stages a 0o755 and a 0o640 file over a 0o777 one and checks both modes. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

antonwolfy · 2026-11-06T20:48:07Z

@exai-bot /review

github-actions · 2026-11-06T20:57:11Z

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

+ added command `dokimi repo tasks authoring`
    + argument `source`
    + option `--cache-dir`
    + option `--force`
    + option `--help/-h`
    + option `--manifest`
    + option `--manifest-file`
    + option `--max-lines`
    + option `--out`
    + option `--prompt-file`
    + option `--prs`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

antonwolfy · 2026-11-06T21:30:43Z

Review rounds done at d449291: CI is green, and codecov reports no changed line uncovered.

Round	Commits	Concerns
1	`91018e90`, `867b78b4`	fixed: dokimi/dokimi/repos/graders/hidden_tests.py:90 (copilot-pull-request-reviewer), codecov; declined: dokimi/dokimi/repos/grading.py:208 (copilot-pull-request-reviewer), dokimi/dokimi/repos/replay/authoring_prs.py:62 (copilot-pull-request-reviewer)
2	`d4492919`	fixed: dokimi/dokimi/repos/grading.py:245 (copilot-pull-request-reviewer)

Declined, with the reason in each thread:

dokimi/dokimi/repos/grading.py:208 (copilot-pull-request-reviewer): This is what the contract specifies.
dokimi/dokimi/repos/replay/authoring_prs.py:62 (copilot-pull-request-reviewer): authoring_prs.py runs in dokimi itself, not on the image's Python 3.8 (only graders/ does), and dokimi requires Python 3.11 or newer (pyproject.toml: requires-python = ">=3.11"; the CI test matrix is 3.11 to 3.14).

No review of this head yet from Copilot.

The host-side copy followed a link under hidden/, so any file the dokimi user can read reached the workspace, and a dangling one aborted eval check with no check.json. Such a file is now a grader error, a copy failure too, and repo validate reports a hidden file that is a link. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

…path The lookbehind refused a name after '/', so `tests/test_a.py` for the hidden pkg/tests/test_a.py kept its line, left test_lines low and could miss the more-than-5 skip. A file name after '/' now matches. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

A run that broke the test command (no report, a crashing conftest) graded pass_share null, so repo compare left it out of the mean while the untouched base counted 0; contract 4.2 says excluding such a run favours arms that break workspaces. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

The contract invites reusing the replay file; without body and createdAt it wrote title-only prompts with no public_at and said nothing about the description. A missing body is now a skip with the reason; an empty body is still a PR with no description. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

Main already has a 4.7 (graph tasks from a code index), so after the merge the --prs help and every 4.7 this PR adds would point at the graph section. The authoring section and its references become 4.8. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

…ards Each test fails when its guard alone is removed: a test file the PR deletes is not hidden, authoring writes a 100755 hidden file 0755, and a first oracle grade that cannot run is a broken task, not a crash. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

Keeps both built-in graders (graph and hidden_tests) in NAMES and _MODULES, the table rows of both, and the contract sections as 4.7 graph tasks then 4.8 authoring. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

A test that skips when the change is absent shows nothing, so pass needs every case passed and pass_share counts skips; a task whose oracle run skips a case is broken in eval check. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

xaleryb · 2026-11-07T05:30:17Z

Pushed to this branch, on top of d449291:

The six fixes from the review: staging refuses a hidden file that is a symbolic link; the prompt drops a line naming a hidden test by a shorter path; a hidden-tests run in which no case ran has pass_share 0; authoring skips a PR whose input object has no body; the authoring section is §4.8, after main's graph tasks; tests pin the deleted-test, file-mode and first-oracle-error guards.
A merge of main (8a6207b). The four conflicts came from feat(eval): eval tasks generate --from graph: enumeration tasks from a SCIP index #399's graph grader and keep both sides: graph and hidden_tests are both in NAMES and in main.py's _MODULES, the contract table has both rows, §4.7 is graph tasks and §4.8 authoring, and architecture.md names repo tasks authoring among the commands that write into the tasks directory.
fix(repo): a skipped hidden test case fails the hidden-tests grade. Before it, a run with 1 passed and 1 skipped case (pytest.importorskip of the module the PR adds, with the module absent) graded pass: true, pass_share: 1.0. §4.5's prose said "every hidden test ran and passed", while its table row and the module docstring accepted skips. Now pass needs every case in the report to have passed, pass_share is passed over all cases, and not_passed lists skipped cases too. A task whose oracle run skips a case (a platform skipif) therefore fails the oracle step of eval check and is broken, so a task that passes eval check expects no skips. §4.5's prose and table, the docstring and the changelog fragment now say the same thing. New eval check tests cover both cases, the importorskip task (now ok, because its null run fails) and the platform-skip task (now broken); both failed before the change.

What was run on the merged head: the full suite (16719 passed, 32 skipped; test_process.py::test_sighup_kills_the_live_groups_too timed out once under -n 8, the PR does not touch that file, and on a rerun of the file all 72 tests passed), ruff, mypy, bandit -r . -ll, check_comment_blocks --base origin/main, check_growth, schema_compat --base origin/main, the CLI snapshot dump (identical to docs/reference/cli-interface.json) and the benchmark gate with --strict. All were clean. From our side nothing is left open.

xaleryb

Approved at 4353b20: CI green on the merge with main. Our points are fixed with tests: symlinked hidden files are refused, the prompt drops shorter-path test names, pass_share is 0 when no case ran, a PR without a body is skipped, and a skipped hidden case fails the grade so eval check marks an oracle that skips as broken.

napetrov

Three points about where this fits with #41. None of them is about the grader:

Placement. #94 D11 (#94 (comment)) puts the repository task builders under eval tasks generate --from replay|graph|history, and --from graph merged in #399. Could this be --from prs with the same --out and --force flags, rather than a new repo tasks authoring leaf?
Section number. #399 took §4.7 ("Graph tasks from a code index"), so this section becomes §4.8 on rebase.
Follow-up. I'll stack --from history on top of Change and dokimi:hidden_tests. It mines bug-fix commits and needs no gh file. In a free trial, 55 of 57 mined dokimi commits were valid fail-to-pass, and 7 of 15 on oneDAL. A direct commit has no PR, so could Change.number be int | None, with the prompt's link-dropping keyed on the SHA when it is None?

xaleryb · 2026-11-07T06:32:27Z

@napetrov on the three points:

Section number: done in ebe0cbd. After the merge with main the contract has §4.7 graph tasks and §4.8 authoring.
Placement (eval tasks generate --from prs vs repo tasks authoring): this needs a decision between you and @antonwolfy before the merge, since it moves the command, its flags and the CLI snapshot.
Change.number as int | None with SHA-keyed link dropping: it depends on the placement decision, because a move to --from prs reshapes the same code. Best done after that is settled.

Keeps main's repo tasks authoring (intel#425) and novelty and provider changes (intel#418, intel#428). The M3 moves stay: repo tasks replay is a tombstone for eval tasks generate --from replay, and repo tasks authoring is kept. cli-interface.json regenerated with scripts/cli_surface.py dump under the venv. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

antonwolfy requested a review from a team as a code owner November 6, 2026 18:54

Copilot AI balanced review requested due to automatic review settings November 6, 2026 18:54

antonwolfy self-assigned this Nov 6, 2026

Copilot AI reviewed Nov 6, 2026

View reviewed changes

Comment thread dokimi/dokimi/repos/graders/hidden_tests.py Resolved

Comment thread dokimi/dokimi/repos/grading.py Resolved

Comment thread dokimi/dokimi/repos/replay/authoring_prs.py Resolved

antonwolfy added 2 commits November 6, 2026 19:28

Copilot stopped reviewing on behalf of antonwolfy due to an error November 6, 2026 19:50
Workflow completed but conclusion is not available. This may indicate rate limiting or API issues.

antonwolfy requested a balanced review from Copilot November 6, 2026 19:56

Copilot started reviewing on behalf of antonwolfy November 6, 2026 20:14

View session

Copilot AI reviewed Nov 6, 2026

View reviewed changes

Comment thread dokimi/dokimi/repos/grading.py Resolved

github-actions Bot added the cli-interface-change Changes dokimi's user-facing CLI surface label Nov 6, 2026

xaleryb added 8 commits November 6, 2026 21:15

Merge origin/main into feat/authoring-replay

bbc399c

Keeps both built-in graders (graph and hidden_tests) in NAMES and _MODULES, the table rows of both, and the contract sections as 4.7 graph tasks then 4.8 authoring. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

xaleryb approved these changes Nov 7, 2026

View reviewed changes

napetrov reviewed Nov 7, 2026

View reviewed changes

napetrov merged commit e4e2f5f into intel:main Nov 7, 2026
31 checks passed

napetrov mentioned this pull request Nov 7, 2026

fix(cli): input refusals go through InputError (#94 A2b, part of #275) #451

Open

antonwolfy deleted the feat/authoring-replay branch November 7, 2026 11:12

Navigation Menu

Repository navigation

feat(repo): repo tasks authoring builds hidden-test tasks from merged PRs - #425

feat(repo): repo tasks authoring builds hidden-test tasks from merged PRs#425
napetrov merged 12 commits into
intel:mainfrom
antonwolfy:feat/authoring-replay

antonwolfy commented Nov 6, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Nov 6, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

antonwolfy commented Nov 6, 2026

Uh oh!

github-actions Bot commented Nov 6, 2026

Uh oh!

antonwolfy commented Nov 6, 2026

Uh oh!

xaleryb commented Nov 7, 2026

Uh oh!

xaleryb left a comment

Uh oh!

napetrov left a comment

Uh oh!

xaleryb commented Nov 7, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

antonwolfy commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Risk

Testing

Revert

Checklist

Uh oh!

codecov-commenter commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

antonwolfy commented Nov 6, 2026

Uh oh!

github-actions Bot commented Nov 6, 2026

CLI interface change detected

Uh oh!

antonwolfy commented Nov 6, 2026

Uh oh!

xaleryb commented Nov 7, 2026

Uh oh!

xaleryb left a comment

Choose a reason for hiding this comment

Uh oh!

napetrov left a comment

Choose a reason for hiding this comment

Uh oh!

xaleryb commented Nov 7, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

antonwolfy commented Nov 6, 2026 •

edited

Loading

codecov-commenter commented Nov 6, 2026 •

edited

Loading