[napetrov]

Summary

When novelty routing --model --max-usd hit the cap, it exited with spend cap reached ...; nothing was written, and every paid call was lost. On real skill-repo runs this lost more than $9, one run at 444 of 456 calls.

	Before	After
Cap hit, single or --all-vs-all	exit 1, no file	exit 1; --out written with status: "partial" and partial: {reason: "spend_cap", calls_done, calls_planned}
Rates in a partial file	n/a	only finished calls are scored; a skill never reached has own_rate: null and its queries go to n_unscored
Console	cap message only	summary and cost lines; model line reads partial: N of M calls; warning when fewer than 5 calls answered
--out already holds a full result	n/a	partial goes to <stem>.partial.json next to it; the exit message names the path

A finished run is unchanged: JSON, stdout, exit code and files written are identical on base and head for 7 scenarios.

Evidence

• New tests in tests/test_novelty_routing_spend_cap_partial.py fail on base and pass on head; make ci-local passes
• Two fresh-context review passes: pass 1 found 2 blockers (a capped run wrote status: "ok"; the model line had no partial tag), both fixed and confirmed by probes in pass 2

Merge danger

• Not covered: analyze, atoms, probe, shorten still write nothing at the cap (each writes once after several stages, so it is a larger change per command)
• Known nits: two dated --out names share one .partial.json; when a call has no price, its paid reply is not kept; when the cap and failed calls both happen, the exit message names only the cap

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Capped runs can overwrite finished results in a fallback report file, while partial all-vs-all runs omit required summaries, partial-run labels, and low-sample warnings.

2 open issues

• Avoid overwriting finished results at fallback output paths · New
• Show partial all-vs-all summaries when budget limits a run · New

What changed in this PR

2 new findings.

This PR preserves partial novelty-routing measurements when --max-usd stops a run, rather than discarding completed work.

Changes:

• Retains completed model calls, marks capped results as partial, and excludes unfinished work from scoring and repeat-noise calculations.
• Adds partial-result saving and reporting while retaining a nonzero exit.
• Adds regression tests and documents capped-run behavior.

File	Description
dokimi/​tests/​test_novelty_routing_spend_cap_partial.py	Tests partial results, scoring, output, and file preservation.
dokimi/​scripts/​growth_baseline.json	Lowers the command module’s size baseline.
dokimi/​dokimi/​novelty/​routing.py	Retains completed rows when the spend cap interrupts routing.
dokimi/​dokimi/​novelty/​client.py	Removes the unconditional “nothing was written” claim.
dokimi/​dokimi/​novelty/​all_vs_all.py	Aggregates partial results and marks unreached calls unscored.
dokimi/​dokimi/​commands/​novelty.py	Saves partial results before exiting routing commands.
dokimi/​dokimi/​commands/​novelty_routing_report.py	Adds partial-result persistence, reporting, and exit helpers.
dokimi/​docs/​skill-novelty.md	Documents partial records, scoring, and reruns.
dokimi/​docs/​skill-novelty-walkthrough.md	Explains routing’s exception to cap-triggered data loss.
dokimi/​changelog.d/​20261104_130000_napetrov_fix-cap-partial-save.md	Announces partial-result preservation.

_{Review effort: Balanced}

[exai-bot]

OpenCodeReview: Commit 8560de6 was not reviewed. This PR has reached its review limit (6 reviews).

CI at `8560de6`: 7 still running, 7 passing, 2 skipped

Check	Status
CLI interface diff	still running
CodeQL (actions)	still running
CodeQL (python)	still running
Package install smoke test (py3.11)	still running
Tests (py3.11)	still running
Tests (py3.14)	still running
Windows smoke test (py3.13)	still running

To review this commit: re-request review from exai-bot in the Reviewers list.

<details><summary>Last review (d9a7538): no findings</summary>

Open threads from the bot: 0

</details>

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.00%. Comparing base (5deeb85) to head (39823f2).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #406      +/-   ##
==========================================
+ Coverage   94.99%   95.00%   +0.01%     
==========================================
  Files         363      364       +1     
  Lines       48348    48437      +89     
  Branches     8318     8332      +14     
==========================================
+ Hits        45928    46019      +91     
+ Misses       1531     1530       -1     
+ Partials      889      888       -1     

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/novelty.py	98.70% <100.00%> (+0.40%)	⬆️
dokimi/dokimi/commands/novelty_routing_report.py	98.51% <100.00%> (+0.32%)	⬆️
dokimi/dokimi/novelty/all_vs_all.py	99.31% <100.00%> (+0.07%)	⬆️
dokimi/dokimi/novelty/client.py	98.27% <100.00%> (+0.07%)	⬆️
dokimi/dokimi/novelty/noise.py	100.00% <100.00%> (ø)
dokimi/dokimi/novelty/routing.py	96.56% <100.00%> (+0.07%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[napetrov]

Both findings are answered below (#1, #2), fixed in 64b71de.

"Avoid overwriting finished results at fallback output paths": fixed in 64b71de by Nikolay Petrov. A capped run writes to the first free <stem>.partial<N>.json. Pinned by test_a_partial_run_skips_a_sidecar_that_holds_a_finished_result.

"Show partial all-vs-all summaries when budget limits a run": fixed in 64b71de by Nikolay Petrov. A capped all-vs-all run prints the summary, partial: N of M calls and the fewer-than-five warning. Pinned by test_an_all_vs_all_run_stopped_by_the_cap_scores_the_queries_it_finished.

Its findings are the threads PRRT_kwDOVBg7g86z4ihc and PRRT_kwDOVBg7g86z4ihf, fixed in 64b71de.

Nothing to change: this summary reports 0 findings on 86903d8, so there is no claim to act on.

Covered in b5fdca5: tests for the unreadable sidecar and two more lines. Test-only; the code was right.

[antonwolfy]

The cap handling holds up: a capped routing run writes a partial result, keeps finished files safe, and still exits non-zero. One risk is left in the tests for the capped --all-vs-all --repeats N path, and one doc page is now stale.

What I ran, on a Linux review host at b5fdca5:

• make ci-local: 29 passed, 1 failed, 7 skipped; 15,724 tests. The failure is test_unbalanced_quote_in_hook_completes_quickly (a 1 s timing bound, 2.09 s measured) plus two 60 s timeouts in test_audit_test_quality.py. Both files pass on a rerun (680 passed), and the PR does not touch them. The changed-line coverage step passes.
• Reproductions for the code findings, with the LLM transport mocked.
• A mutation run over the new code: 68 mutants, 19 of which pass the new tests plus tests/test_novelty*.py.

Other finding, outside the diff:

1. Nit: docs/reviewing.md:79-80 still says --max-usd "writes no artifact when the cap trips" for novelty steps. That is no longer true for novelty routing --model, which now saves a partial file or a .partial.json sidecar. Suggestion: add "except novelty routing, which saves the calls it made as a partial result", as the walkthrough page now does.

Checked, no issues

• Sidecar naming: a finished --out sends the partial result to <stem>.partial.json, then .partial2.json, and the exit message names the path.
• Documented keys: partial, status, n_calls, n_unscored, own_rate: null and the noise exclusion match the output of a mocked capped run.
• Cap on the first call: zero rows, calls_done 0, no crash, both modes.
• Failed calls with the cap: counted in n_failed, and the refused calls land in n_unscored.
• Repo rules: changelog fragment, growth baseline, docs nav and the offline tests pass; the new tests write only under tmp_path.
• Commits: all four carry Signed-off-by.

[antonwolfy]

I pushed a commit per nit, up to 664b082, which fix the nit from my review:

• 664b082: dokimi/docs/reviewing.md:79: the --max-usd sentence now says novelty routing --model is the exception and saves the calls it made as a partial result.

make ci-local at 664b082: 30 passed, 0 failed, 7 skipped.

Not approving yet. Still open: the risk from my review.

[napetrov]

Fixed in 664b082 by Anton Volkov: reviewing.md:79 names the routing exception.

Open risk (the repeats test gap) is fixed in 2fe172c: see the thread.

[xaleryb]

d9a7538: merged main after #396. The growth baseline records novelty.py at its 921 merged lines, and skill-novelty.md and the changelog fragment name the moved skill novelty analyze/atoms/probe and skill shorten commands.

[antonwolfy]

Review at d9a7538. Both points from my last review are fixed: the reviewing.md sentence names the routing exception, and the new repeat-run test pins calls_done, calls_planned, both n_unscored values and the noise block. The problems are at the edges of the unpriced-call path and in two measurements the tests do not pin. Three things are worth fixing before merge, plus one nit inline.

What I ran, on a Linux review host at d9a7538:

• make ci-local: 30 passed, 0 failed, 7 skipped. Changed-line coverage passes.
• Reproductions for the code findings, with the LLM transport mocked.
• A mutation run over the new code: 67 mutants applied, 22 of which pass the new test file. Two of the survivors are inline below.

Checked, no issues

• Uncapped output: a finished single-artifact run writes the same routing.json on the PR's base and on head, apart from the artifact path.
• Schema: routing files are written with write_text, carry no schema_version and are not validated against novelty.v1.json, so status and partial need no schema change.
• Sidecars: a finished --out sends the partial result to <stem>.partial.json, then .partial2.json; the exit message names the path.
• Commits: all non-merge commits carry Signed-off-by.
• Docs: the keys and console lines documented in skill-novelty.md match a mocked capped run, single and --all-vs-all --repeats 2.

[antonwolfy]

I pushed a commit per nit, up to 8560de6, which fix the nit from my review:

• 8560de6: dokimi/docs/skill-novelty.md:762: the resume advice now says DOKIMI_RESPONSE_CACHE must also have been set on the capped run.

make ci-local at 8560de6: 29 passed, 1 failed (host timing flake, green on rerun), 7 skipped.

GitHub CI has not been waited for at this head: the Actions queue held the runs for over an hour, so the fix is not checked there yet. Locally the fix commit passes the CI steps; the only failures are load-sensitive timeouts in files the docs change does not touch.

Not approving yet. Still open: the bug and the two risks from my review.

[napetrov]

Not approving yet: open items are 51bcf96, daa7270, c807886.

Nit fixed in 8560de6 (Anton Volkov). Open items: see the threads above.

Nothing to change: novelty.py is 921 lines, as in growth_baseline.json, and the command names in skill-novelty.md match the code.