[napetrov]

Summary

eval score panel
  _save_incremental(results)              (eval/panel.py, once per question)
    before: open(tmp) + json.dump + os.replace   -> no schema_version, no run_id
    after:  save_artifact("eval", output, path)  -> eval.v1 stamp + run_id + dokimi_version

Part of #94; continues #277 (B1). The other run-artifact writers already go through save_artifact.
A scan of the package for direct json.dump / write_text of registered run kinds found no other
bypass: the remaining direct writes are reports, logs, caches, exports, or unregistered payloads
(no schema_version), which runs list does not read. #278 (group runs list by run_id) is already
in #427 and is not repeated here.

Changes:

• dokimi/eval/panel.py: _save_incremental saves through save_artifact; json/os imports dropped.
• dokimi/tests/test_run_envelope.py: the panel output is an eval.v1 artifact with the envelope and
  is listed by scan (the code runs list uses); a rejected save (question_id None) writes nothing
  and keeps panel.json.rejected with its payload; a missing output directory is created.
• dokimi/changelog.d/: Fixed fragment.

No CLI surface change (no flag, route or exit-code change); no cli-interface-change label.

Evidence

• Before: on the base panel.py, pytest tests/test_run_envelope.py -k panel -> 2 failed
  (the rejected-save and envelope tests; the missing-directory test passes on base, as expected).
• After: pytest tests/test_run_envelope.py tests/test_baseline_compare.py tests/test_panel.py
  -> 209 passed (the panel path, including the existing full evaluate_answers test with real
  verdict rows, runs through the new save). Wider run before the last two tests were added:
  tests/test_writers_stop_on_rejected_save.py tests/test_artifacts.py tests/test_runs_*.py tests/test_eval_judge.py in the same batch passed.
• Full pytest tests: 16746 passed, 31 skipped, 27 failed. 26 of the failures also fail on the
  unmodified base (test_judge_sampling, test_llm_delegates, test_model_defaults,
  test_openai_compatible_route, test_repo_typecheck_grader): environment (locked litellm pricing,
  provider routes, typecheck grader), not this change. test_process.py::test_sighup_kills_the_live_groups_too
  failed once in the full run and passes alone (72 passed): flaky.
• ruff check on the changed files and mypy on eval/panel.py and artifacts.py: clean.
• NaN floats in a panel row serialize and validate as before (probed).

Merge Danger

Door: two-way. The change is a writer swap; reverting restores the old unstamped file.

Blast Radius: eval score panel output only. The file gains schema_version, run_id and
dokimi_version at the top and is now schema-checked on every incremental save: a record the eval.v1
schema rejects stops the panel run at that question (the paid results so far stay on disk, the bad payload
goes to panel.json.rejected). Readers that read evaluations / panel_config are unaffected; baseline save
copies the file as is.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

The release note and test reference use the invalid command eval panel score instead of the registered eval score panel and need correction.

1 open issue

• Correct the eval score panel command order · New

What changed in this PR

1 new finding (nit).

This PR routes panel-judge output through the shared artifact writer, continuing #277’s run-envelope work and making panel results discoverable by runs list.

Changes:

• Replace manual incremental JSON writes with save_artifact, adding schema validation and run metadata.
• Add tests for the envelope, scan discovery, rejected payload preservation, and output-directory creation.
• Document the change in a changelog fragment.

File	Description
dokimi/​tests/​test_run_envelope.py	Tests panel artifact metadata, discovery, rejection, and directory creation.
dokimi/​dokimi/​eval/​panel.py	Saves incremental panel results through the shared artifact writer.
dokimi/​changelog.d/​20261106_220000_napetrov_eval-panel-envelope.md	Announces the panel output’s run envelope and validation.

_{Review effort: Balanced}

[napetrov]

Fixed in 0e68afd: the changelog fragment and the test docstring now say eval score panel, the registered command (evaluate.py:578).

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.31%. Comparing base (1b205bd) to head (0120a22).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #450      +/-   ##
==========================================
- Coverage   95.31%   95.31%   -0.01%     
==========================================
  Files         378      378              
  Lines       50337    50333       -4     
  Branches     8679     8679              
==========================================
- Hits        47981    47975       -6     
- Misses       1481     1482       +1     
- Partials      875      876       +1     

Files with missing lines	Coverage Δ
dokimi/dokimi/eval/panel.py	97.71% <100.00%> (-0.05%)	⬇️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[napetrov]

Already fixed in 0e68afd, as in the thread above.

No change: this is the Codecov report, and it finds all modified lines covered (patch 100%). Nothing to act on here.

[antonwolfy]

LGTM. Routing the panel output through save_artifact is the right fix: the file now carries eval.v1, run_id and dokimi_version, keeps the atomic write and the parent-directory creation, and a rejected save keeps its payload in panel.json.rejected. No findings.

What I ran, on Linux at e6a6510:

• make ci-local: lint, mypy (378 files), deptry, schema_compat, changelog fragment, comment blocks and docs build pass. Two timeout errors in the full pytest run (test_import_architecture.py, test_audit_test_quality.py) pass when run alone (225 passed). The gitleaks install failed on a download error, so I did not scan the commits.
• Mutation run on the three new tests and test_panel.py: 12 mutants (wrong kind, dropped call, plain json.dump with and without mkdir, swallowed validation error, renamed panel_config, and others). 9 killed. The three survivors are run_metadata, judge_model and library_name; test_baseline_compare.py covers the first two, and nothing pins library_name.

Checked, no issues

• Readers: baseline save copies the file as is, and _recorded_models reads judge_model and panel_config only. The eval.v1 schema has no additionalProperties: false, so the extra keys are harmless.
• Other writers: no other json.dump or os.replace of a registered kind is left under eval/.
• Rejected save: the ArtifactValidationError is not caught in cmd_eval_score_panel, but records() already refuses a missing or non-string/integer question_id before any judge call, so the first save cannot be rejected through the CLI.