feat(runs): run envelope with run_id on run artifacts (#277 B1) by napetrov · Pull Request #398 · intel/dokimi

napetrov · 2026-11-04T20:16:19Z

Summary

Run artifacts get a run_id right after schema_version. Input kinds get none: question sets
(their question_set_hash digests the file's bytes, so a per-save id would change it on every
re-save) and the authored or generated kinds.

save_artifact(kind, data, path, run_id=None)
  payload = {schema_version}
  if kind in RUN_KINDS                # questions, dokimi_config, cli_routes excluded
    payload.run_id = run_id or current_run_id()   # one id per dokimi process
  payload += data (keys not already set)
  validate, then write atomically

 runs list --json   row: schema_version, kind, subject, model, generated_at, path
+                        run_id (null for artifacts written before the envelope)
 runs show          + run_id row
 plan_digest        ignores run_id, like generated_at
 schemas            run_id declared, optional, on every run kind
+                   the closed run schemas name it explicitly (questions stays open)

Not in this PR (#277 B1b / #278 / follow-ups): eval/* writers that bypass
save_artifact, grouping rows by run_id, the common cost block, and
child-process propagation (repo conformance children get their own id).

Evidence

Before: save_artifact("questions", ...) wrote a run_id; re-saving a question set changed
its bytes and so its question_set_hash. After: tests/test_run_envelope.py pins it:
test_re_saving_a_question_set_keeps_its_hash, test_a_question_set_is_an_input_and_carries_no_run_id,
and the RUN_KINDS membership test. Red first: 7 failed before the fix (new and reverted tests on the B1 head), green after.
A question set written with a run_id by an earlier build still loads (schema is open).
repo_ledger keeps its run_id. Nothing digests its bytes: file_sha256 callers hash only the
manifest spec and docs, subjects/scorecard.py digests only question files and subject/arm trees,
and the ledger's output records its path, not a hash.
Full suite (.venv, pytest -q): 15723 passed, 12 skipped, 0 failed. ruff check: passed.
mypy dokimi: no issues in 358 files.
eval headroom: a copy whose only difference is the top-level run_id now counts once
(test_runs_that_differ_only_in_run_id_count_once).
Fresh-context self-review of the delta (sonnet, diff and description only): VERDICT pass, 0 blocking.

Merge Danger

Door: two-way. Additive optional key; reverting drops the key from new files and nothing else depends on it.

Held item resolved: the owner's call (2026-11-04) keeps question sets out of the envelope. Their
question_set_hash is unchanged by a re-save, so no existing baseline hash moves. repo_ledger is the
one hand-written kind left with a run_id; its file changes on a re-save, and no digest reads it.

Blast Radius: artifact files and runs output. A dokimi build older than this PR that closes
a run schema without run_id will reject artifacts this one writes. Old artifacts without run_id keep
loading and listing here, and question sets written with one still load.

Part of #94, Part of #277

🤖 Generated with Claude Code

save_artifact stamps every run kind with run_id, right after schema_version: the run_id argument if given, else one id per dokimi process (current_run_id). A run_id inside the data is not read, so a payload copied from another artifact is written under this run. dokimi_config (authored) and cli_routes (generated) get no envelope. Input kinds (harness_profile, repo_manifest) get none. - Schemas: run_id is declared, optional, on every run kind. The closed run schemas need it to admit the key; the closed-world fixture check needs it on the open ones too. The four named closed schemas (harness_profile, repo_manifest, repo_readiness, tool_preference_groups) name it explicitly. - stamp() also sets run_id for run kinds, so a printed --json document and the saved file carry the same envelope. - Readers: runs list --json and runs show report run_id; artifacts written before the envelope load and list with run_id null. - plan_digest leaves run_id out, like generated_at. Not in this change: grouping by run_id (intel#278), the common cost block (per-kind spend shapes differ today), child-process propagation (repo conformance gives each child its own id), and routing eval/* writers that bypass save_artifact (B1b). Part of intel#94, intel#277 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

Copilot

🟢 Approval recommended

There are no active review issues blocking approval.

No open issues 🎉

What changed in this PR

0 new findings; nothing to fix.

This PR adds the first run-envelope support for #277, giving dokimi artifacts a shared per-process run_id and preparing them for future run grouping.

Changes:

Stamp run artifacts with a generated or explicitly supplied run_id.
Declare optional envelope fields in schemas and expose recorded IDs through runs list and runs show, while supporting older artifacts.
Keep plan digests stable and update tests, fixture comparisons, and documentation.

File	Description
`dokimi/tests/test_task_format_corpus.py`	Exclude process IDs from corpus comparisons.
`dokimi/tests/test_runs_show.py`	Expect recorded IDs in show output.
`dokimi/tests/test_runs_scan.py`	Check IDs according to artifact kind.
`dokimi/tests/test_runs_cli.py`	Expect IDs in list JSON.
`dokimi/tests/test_run_envelope.py`	Cover stamping, validation, scanning, and compatibility.
`dokimi/tests/test_repo_environment.py`	Exclude process IDs from fixture comparisons.
`dokimi/tests/test_questions_artifact.py`	Account for IDs during question round trips.
`dokimi/tests/test_artifacts.py`	Check envelope key ordering.
`dokimi/dokimi/schemas/tool_preference.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/tool_preference_groups.v1.json`	Reserve the envelope field in the closed schema.
`dokimi/dokimi/schemas/task_runs.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/task_inspect.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/subject_scorecard.v2.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/skill_evidence.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/skill_discovery.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/result.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/repo_readiness.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/repo_manifest.v1.json`	Reserve the envelope field for inputs.
`dokimi/dokimi/schemas/repo_ledger.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/repo_hits_judged.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/repo_environment.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/repo_budget.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/redteam.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/reach_matrix.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/questions.v2.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/plugin_delta.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/plan.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/novelty.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/need_graph.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/matrix_rollup.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/headroom.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/harness_profile.v1.json`	Reserve the envelope field for inputs.
`dokimi/dokimi/schemas/guidance_reach.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/explore_runs.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/eval.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/eval_coverage.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/effective_instructions.v1.json`	Distinguish envelope IDs from receipt IDs.
`dokimi/dokimi/schemas/catalog_eval.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/arms.v1.json`	Admit optional run IDs.
`dokimi/dokimi/schemas/answers.v1.json`	Admit optional run IDs.
`dokimi/dokimi/runs/scan.py`	Read and serialize optional run IDs.
`dokimi/dokimi/runner/plan.py`	Exclude run IDs from plan digests.
`dokimi/dokimi/commands/runs.py`	Display recorded IDs or a missing-ID label.
`dokimi/dokimi/artifacts.py`	Generate process IDs and stamp artifact envelopes.
`dokimi/docs/runs.md`	Explain envelope behavior and current limitations.
`dokimi/changelog.d/20261104_120000_napetrov_run_envelope_b1.md`	Announce run-envelope support.

_{Review effort: Balanced}

exai-bot · 2026-11-04T20:20:13Z

✅ OpenCodeReview: Review complete: 0 finding(s) across 48 selected item(s).

CI at `e82dd60`: 1 still running, 18 passing, 2 skipped

Check	Status
CLI interface diff	still running

Review details

Commit reviewed: e82dd60 against merge base 272d0b2
Coverage: 48 files in 9 groups
Engine: OpenCodeReview v1.13.5-exai.3.d7112371, us.openai.gpt-6.1-sol
Usage: 462k tokens (286k from cache), 30s
Tool calls: 24 (17 file_read, 5 file_read_diff, 2 code_search)
Run envelope implementation and documentation: 4 files
Run discovery, CLI output, and schemas: 8 files
Headroom command and output schema: 3 files
Dataset export and tests: 2 files
Task planning and format compatibility: 4 files
Repository artifact schemas and environment tests: 7 files
Evaluation and result artifact schemas: 9 files
Guidance, reach, and skill artifact schemas: 8 files
Harness and tool preference schemas: 3 files

codecov-commenter · 2026-11-04T20:30:26Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.75%. Comparing base (8d54c93) to head (e82dd60).
⚠️ Report is 2 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #398      +/-   ##
==========================================
- Coverage   94.75%   94.75%   -0.01%     
==========================================
  Files         352      352              
  Lines       46507    46427      -80     
  Branches     7970     7953      -17     
==========================================
- Hits        44070    43994      -76     
+ Misses       1544     1542       -2     
+ Partials      893      891       -2

Files with missing lines	Coverage Δ
dokimi/dokimi/artifacts.py	`96.07% <100.00%> (+0.72%)`	⬆️
dokimi/dokimi/commands/headroom.py	`98.98% <100.00%> (+0.01%)`	⬆️
dokimi/dokimi/commands/runs.py	`100.00% <ø> (ø)`
dokimi/dokimi/datasets_export.py	`85.05% <ø> (ø)`
dokimi/dokimi/runner/plan.py	`100.00% <100.00%> (ø)`
dokimi/dokimi/runs/scan.py	`97.71% <100.00%> (+0.02%)`	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The headroom dedupe keys a file by the digest of its parsed content. A copy of an arms file whose only difference is its run_id counted as a second run. The top-level run_id is left out of that digest, as plan_digest leaves it out. A file from another process still differs in generated_at, so it counts as a run. Docs and changelog: re-saving a questions or repo_ledger file adds a new run_id. Part of intel#94, Part of intel#277 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

scorecard's question_set_hash digests the raw bytes of each question file, so a run_id stamped on every save would change the hash on every re-save and break baseline matching. Question sets feed later runs; the envelope is for run artifacts. questions leaves RUN_KINDS and the questions schema drops its run_id property. Old files that carry one still load (the schema is open). repo_ledger stays a run kind: no code digests its bytes (file_sha256 callers hash only the spec and docs; the ledger writes no digest and its output records the path only). Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

napetrov · 2026-11-05T04:17:30Z

No change needed: this review reports 0 findings and nothing to fix, so there is no claim to act on.

No change needed: this summary reports 0 findings, so there is no claim to act on.

No change needed: coverage is complete for the modified lines. The Codecov app install is an org setting, not a code change for this PR.

antonwolfy

LGTM. The envelope does what the description says: run_id leads the payload after schema_version for the 28 run kinds, stays off questions, dokimi_config and cli_routes, and both plan_digest and the headroom dedupe digest ignore it. One nit inline.

What I ran, on a Linux review host at d959e56:

make ci-local: 29 passed, 1 failed, 7 skipped. 15,724 tests passed. The one failure is test_repo_environment.py::test_unbalanced_quote_in_hook_completes_quickly (2.08 s against a 1 s limit), a wall-clock assertion on a loaded machine; it passes on its own (2 passed in 1.1 s) and the PR does not touch the code it times. The changed-line coverage, schema_compat, changelog fragment, mypy, docs build and wheel smoke steps passed.
A mutation run on the new code: 23 mutants, 3 of which pass the full suite (one is the nit below; the other two change only the "- (none recorded)" placeholder text and the empty-string run_id read in runs/scan.py).
Schema probe: all 28 run-kind schemas admit a run_id and reject an empty one and one of 129 characters.

Checked, no issues

stamp() copy: it now returns a copy instead of mutating. All callers (commands/repo.py, commands/tasks.py) use the return value.
Digests: I found no digest of a saved run artifact's bytes; the ones in scorecard.py, eval_coverage.py and bundle.py cover question files, rows or node lists.
Tests: the new tests write only under tmp_path, and the file fails on the base modules (RUN_KINDS does not exist there).

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

vlad-perevezentsev

Review at 63062d8. The envelope is right: every run kind gets run_id right after schema_version, question sets stay out of it, and plan_digest and the headroom dedupe ignore it. What is left is small and in the line comments: dataset export drops the id, plus two nits.

What I ran, at 63062d8:

make ci-local --offline: 0 failed, 19 passed, 13 skipped (network and Docker steps).
A repro for the export comment: an answers artifact saved through save_artifact, then flatten("answers", ...): the rows have generated_at and no run_id.

Checked, no issues

Writers: catalog eval saves plan and result in one process, so both share an id, and plan_digest is the same before and after the save. repo_ledger and need_graph are never saved through save_artifact (the ledger is hand-written, save_graph writes plain JSON), so stamping them changes nothing today.
Readers: runs list --json and runs show read run_id and give null for an old file or a non-string value. The scan's header sniff still works with the extra key.
Schemas: the change only widens them: run_id is optional everywhere, and the four closed schemas #277 names admit it.
Shape of the id: the regex test from 63062d8 settles the open nit on test_run_envelope.py:94.

vlad-perevezentsev · 2026-11-05T14:05:20Z

@napetrov, it looks like this PR and #387 do the same thing: both add the B1 envelope of #277. They change the same 35 files, among them artifacts.py, runs/scan.py, the run-kind schemas and a new tests/test_run_envelope.py, so only one of them can merge. Would you be OK with closing this one in favour of #387?

Where they differ:

Inputs. feat(runs): run envelope with run_id on run artifacts (#277 B1) #398 leaves out questions. feat(artifacts): run envelope (run_id, dokimi_version) on run artifacts #387 leaves out plan, repo_ledger and need_graph, and stamps questions, so a re-saved set changes its question_set_hash. I am fine with your call on question sets.
plan. feat(runs): run envelope with run_id on run artifacts (#277 B1) #398 stamps it and leaves run_id out of plan_digest; feat(artifacts): run envelope (run_id, dokimi_version) on run artifacts #387 treats it as an input.
Only in feat(artifacts): run envelope (run_id, dokimi_version) on run artifacts #387: dokimi_version; DOKIMI_RUN_ID (one id for several commands, checked in cli.main, exit 2); the run_id pattern in the schemas; the reason runs show gives for a missing id.
Only in feat(runs): run envelope with run_id on run artifacts (#277 B1) #398: run_id left out of the headroom digest; run_id right after schema_version in stamp(); the closed input schemas widened.

Some of this PR is better, and I would be glad to bring it over to #387: question sets as inputs, plan stamped with run_id left out of plan_digest, run_id left out of the headroom digest, and the envelope right after schema_version in stamp().

Thread: PRRT_kwDOVBg7g860GCVn Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

Thread: PRRT_kwDOVBg7g860GCVy Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

Thread: PRRT_kwDOVBg7g860GCV6 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

Thread: PRRT_kwDOVBg7g860EoCM: already fixed in 63062d8 Thread: PRR_kwDOVBg7g88AAAABWNL8SQ: already fixed in 63062d8 Thread: PRR_kwDOVBg7g88AAAABWN6aDA: fixed in 63ff86e, b8087f7, 683af78 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

…lope-b1-277 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

napetrov · 2026-11-05T15:49:14Z

Already fixed in 5274d8d: the nit is the shape test in 63062d8.

Fixed in 5274d8d: the export and both nits, each under its thread.

Waiting on a decision before I change this: close #398 in favour of #387, or keep both? My recommendation: close #398 and take its three extra pieces into #387 (question sets as inputs, plan run_id out of plan_digest, run_id out of the headroom digest). #387 carries more of the envelope and only one can land. This stays open until you decide.

vlad-perevezentsev · 2026-11-05T16:01:30Z

Thanks, @napetrov. All three are in #387 now, one commit each:

question sets as inputs (7848996);
a plan with the envelope, run_id and dokimi_version left out of plan_digest (b862875);
both envelope keys left out of the headroom digest (99164c6).

stamp() in save_artifact's order (1227d94) and the envelope in the export's META_KEYS (27e5326) are there too. The one piece I did not take is widening the closed input schemas (harness_profile, repo_manifest, tool_preference_groups): save_artifact never writes them, so no envelope reaches them.

So I think this PR can be closed in favour of #387. Grouping runs list by run_id (#278) comes next from my side, on top of it.

napetrov · 2026-11-05T22:32:47Z

Agreed, closing this in favour of #387. Thanks for the comparison.

Please do bring over the pieces you listed: question sets as inputs (so a re-saved set keeps its question_set_hash), plan stamped with run_id left out of plan_digest, run_id left out of the headroom digest, and the envelope right after schema_version in stamp(). The export fix here (run_id in META_KEYS, 63ff86e) is worth checking against #387 too.

napetrov requested a review from a team as a code owner November 4, 2026 20:16

Copilot AI balanced review requested due to automatic review settings November 4, 2026 20:16

Copilot started reviewing on behalf of napetrov November 4, 2026 20:16

View session

Copilot AI reviewed Nov 4, 2026

View reviewed changes

napetrov added 2 commits November 4, 2026 12:56

antonwolfy reviewed Nov 5, 2026

View reviewed changes

Comment thread dokimi/tests/test_run_envelope.py Resolved

test(runs): pin the shape of a minted run_id

63062d8

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

vlad-perevezentsev reviewed Nov 5, 2026

View reviewed changes

Comment thread dokimi/dokimi/artifacts.py Resolved

Comment thread dokimi/tests/test_task_format_corpus.py Resolved

Comment thread dokimi/dokimi/schemas/plan.v1.json Resolved

vlad-perevezentsev mentioned this pull request Nov 5, 2026

feat(artifacts): run envelope (run_id, dokimi_version) on run artifacts #387

Merged

6 tasks done

napetrov added 5 commits November 5, 2026 07:40

Address review: dataset export keeps run_id on every row

63ff86e

Thread: PRRT_kwDOVBg7g860GCVn Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

Address review: drop the dead run_id pop in the corpus test

b8087f7

Thread: PRRT_kwDOVBg7g860GCVy Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

Address review: say run_id is left out of plan_digest in plan.v1

683af78

Thread: PRRT_kwDOVBg7g860GCV6 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

Merge branch 'main' of https://github.com/intel/dokimi into feat/enve…

e82dd60

…lope-b1-277 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

vlad-perevezentsev mentioned this pull request Nov 5, 2026

feat(task-formats): judge custom scorers, read hf_dataset and code promptfoo configs, name Harbor suites #402

Merged

napetrov closed this Nov 5, 2026

Navigation Menu

Repository navigation

feat(runs): run envelope with run_id on run artifacts (#277 B1) - #398

feat(runs): run envelope with run_id on run artifacts (#277 B1)#398
napetrov wants to merge 9 commits into
intel:mainfrom
napetrov:feat/envelope-b1-277

napetrov commented Nov 4, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

exai-bot commented Nov 4, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Nov 4, 2026 •

edited

Loading

Uh oh!

napetrov commented Nov 5, 2026

Uh oh!

antonwolfy left a comment

Uh oh!

Uh oh!

vlad-perevezentsev left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vlad-perevezentsev commented Nov 5, 2026 •

edited

Loading

Uh oh!

napetrov commented Nov 5, 2026

Uh oh!

vlad-perevezentsev commented Nov 5, 2026

Uh oh!

napetrov commented Nov 5, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

napetrov commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Evidence

Merge Danger

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

exai-bot commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

napetrov commented Nov 5, 2026

Uh oh!

antonwolfy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vlad-perevezentsev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vlad-perevezentsev commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

napetrov commented Nov 5, 2026

Uh oh!

vlad-perevezentsev commented Nov 5, 2026

Uh oh!

napetrov commented Nov 5, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

napetrov commented Nov 4, 2026 •

edited

Loading

exai-bot commented Nov 4, 2026 •

edited

Loading

codecov-commenter commented Nov 4, 2026 •

edited

Loading

vlad-perevezentsev commented Nov 5, 2026 •

edited

Loading