Errata retracts on two votes and a flag, alignment shortlists its candidates, and each task thinks as hard as it needs - #906
Conversation
…hing the document names Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…runs extraction at minimal Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…es are asked only of bound signatures Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
… structure fits too many Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…and a hidden fitting key still counts Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…bulk tasks keep the workspace's Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…ad of leaving vectors missing Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…hen it is unique Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…ime instead of one after another Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…hink at low effort Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
… endpoint's default Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…le after the rebase Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…cut changed Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…les require The settings page rendered a native select for the four-way effort choice; the style guard refuses a raw select, a rounded-md corner and a py-1.5 gap, and the design says a small bounded enum is a Dropdown so a page never shows two kinds of popup. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
… 0076 and 0090 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
|
Reviewed the code diff in full (the bench data aside) and landed two maintainer edits so this can merge:
On the substance: the two-votes-and-a-flag rule for retractions, the once-per-batch glossary with keys per item, the label-as-key fallback, the shortlist as part of the decision basis, and the per-task reasoning effort all read as the measured cuts the README describes, and each has a test at the layer it changes. One thing to keep an eye on rather than block: Locally on a fresh database after the merge: CLI 13, server errata 4, phrase alignment 13, store errata 6, extract 84 tests pass; clippy clean. Merging once CI agrees. |
…ith tokens by phase (#917) Two groups on dev at 961c3c0 with gemini-3.5-flash, judged and with errata: precision 91.0% / 88.5% before errata and 96.0% / 96.5% after; same-sentence gold recall 11.3% / 13.2% before errata and 18.6% / 22.4% after, with the errata retractions mostly wrong on this binary (72/98, 164/196), which is what #906's two-vote retraction addresses; 158k and 183k tokens per document. The token table by phase shows where they went: the alignment window carried the rule proposals' per-item property tables, the two-vote phrase alignment, and the governance agent deciding the same pairs from up to ten concurrent runs (9481 decisions on 1421 pairs), fixed in #916. The bench judge now fails fast when it would read a sealed key from llm_settings instead of sending the ciphertext and collecting 401s. Signed-off-by: Wayland Yang <wayland0916@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…tokens by phase (#918) Same corpus, model and judge as the pre-#906 baseline, on dev with #906 and #916. Judged precision 81.0% / 79.0% before errata and 83.0% / 80.5% after (the loss against the baseline is misworded facts from minimal reasoning); same-sentence gold recall 14.2% / 14.3% before errata and 17.8% / 17.1% after, with errata now adding without retracting wrongly (2 and 1 retractions, none judged stated); 31k and 34k tokens per document, a fifth of the baseline and still five times 0044's ceiling, seven tenths of it in alignment and rule proposals. 47 and 48 minutes a group against three to five hours. Signed-off-by: Wayland Yang <wayland0916@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The first measured typed-graph runs (#893's README) spent 136,000 tokens per 200-word document and the errata agent retracted mostly what the document did say. This PR is the set of cuts that followed, each measured on the same 20 documents (numbers in the bench README, "第二次真跑"), landing at about 56,000 tokens per document with precision back where the thinking model puts it.
Errata agent (0044 decision 7)
unflagged), and a flagged one is asked about a second time, alone with the document; only an explicit not stated retracts. On 100 documents the agent went from 34/37 wrong retractions to zero (2 retractions, 8 revisions, all confirmed).Alignment cost (0044 decision 3)
p569-style keys the model answersdateOfBirth, and 586 of 1189 signatures in the first run were counted as malformed for it. A key the shortlist hid still counts if it fits structurally.nonesignatures produced under 1% of proposals and half the calls).Reasoning effort
llm_settings.chat_reasoning_effort(migration 0075;minimal | low | medium | high, empty = endpoint default) rides asreasoning_effortin every chat body; settings API and page in both languages. Extraction, readings and errata use it (minimalturns thinking off without changing the answer).low(JUDGEMENT_EFFORT), rule proposals atlow. All-minimal alignment judged 31% misworded, endpoint default 12%, low 76% precision on the small sample, so the constant is a measured choice and one line to change.Bench:
--corpus redocred-20smoke corpus, governance off in the bench base, embedding model configured in setup with property vectors built before extraction.Verified against pgvector/pgvector:pg16 on a fresh database, after rebasing onto #894/#895:
Plus the measured runs: round 3 (100 documents) and smoke5/6/8b/9/10 (20 documents) as tabled in the README.
🤖 Generated with Claude Code