Skip to content

Errata retracts on two votes and a flag, alignment shortlists its candidates, and each task thinks as hard as it needs - #906

Merged
WaylandYang merged 17 commits into
devfrom
feat/errata-two-votes
Sep 25, 2026
Merged

WaylandYang merged 17 commits into
devfrom
feat/errata-two-votes

Conversation

@WaylandYang

Copy link
Copy Markdown
Contributor

The first measured typed-graph runs (#893's README) spent 136,000 tokens per 200-word document and the errata agent retracted mostly what the document did say. This PR is the set of cuts that followed, each measured on the same 20 documents (numbers in the bench README, "第二次真跑"), landing at about 56,000 tokens per document with precision back where the thinking model puts it.

Errata agent (0044 decision 7)

  • A retraction or revision needs a structural flag and two votes: an unflagged fact is never retracted by the agent alone (held for a person as unflagged), and a flagged one is asked about a second time, alone with the document; only an explicit not stated retracts. On 100 documents the agent went from 34/37 wrong retractions to zero (2 retractions, 8 revisions, all confirmed).
  • Additions opened the other way: a name the document contains but the base lacks becomes a new entity with its name fact; a document that came out of extraction with no typed facts is reviewed once with an empty list. Recall on 100 documents 9.3% → 18.4% (same-sentence 13.1% → 26.5%); full-set judged precision 90.2%.
  • The errata prompt carries only the 24 properties nearest the document plus those its facts use (4.7k → ~1k prompt tokens a call).

Alignment cost (0044 decision 3)

  • Shortlist: when the structure fits more than ten properties, the signature's phrase and an example are embedded and the ten nearest properties are shown (label matches always kept); the shortlist is part of the decision's basis. Kind words get the same for rule proposals. 18k → 4k tokens a call.
  • The candidate glossary is written once per batch in both the alignment and the rule prompts; items list keys.
  • A label the model answers instead of a key counts when unique: with p569-style keys the model answers dateOfBirth, and 586 of 1189 signatures in the first run were counted as malformed for it. A key the shortlist hid still counts if it fits structurally.
  • Rule proposals ask only bound signatures (the 1121 none signatures produced under 1% of proposals and half the calls).
  • Batches run four at a time (alignment, rule proposals, the bench judge); 20 documents' alignment 29 min → 5 min.

Reasoning effort

  • llm_settings.chat_reasoning_effort (migration 0075; minimal | low | medium | high, empty = endpoint default) rides as reasoning_effort in every chat body; settings API and page in both languages. Extraction, readings and errata use it (minimal turns thinking off without changing the answer).
  • Judgement tasks think on their own: alignment at low (JUDGEMENT_EFFORT), rule proposals at low. All-minimal alignment judged 31% misworded, endpoint default 12%, low 76% precision on the small sample, so the constant is a measured choice and one line to change.
  • The ontology embedding job fails (and so retries) when a batch fails instead of leaving vectors missing; the alignment reply that carried malformed votes is logged.

Bench: --corpus redocred-20 smoke corpus, governance off in the bench base, embedding model configured in setup with property vectors built before extraction.

Verified against pgvector/pgvector:pg16 on a fresh database, after rebasing onto #894/#895:

cargo fmt --all --check / clippy --workspace --all-targets -D warnings    clean
cargo test -p utopia-store                                                344 passed, 0 failed
cargo test -p utopia-server                                               407 passed, 0 failed
cargo test -p utopia-extract / utopia-llm / utopia-cli                    84 / 37 / 13 passed
cd web && pnpm build && pnpm test                                         tsc clean, 132 passed

Plus the measured runs: round 3 (100 documents) and smoke5/6/8b/9/10 (20 documents) as tabled in the README.

🤖 Generated with Claude Code

WaylandYang and others added 17 commits September 24, 2026 23:07
…hing the document names

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…runs extraction at minimal

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…es are asked only of bound signatures

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
… structure fits too many

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…and a hidden fitting key still counts

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…bulk tasks keep the workspace's

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…ad of leaving vectors missing

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…hen it is unique

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…ime instead of one after another

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…hink at low effort

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
… endpoint's default

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…le after the rebase

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…cut changed

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
…les require

The settings page rendered a native select for the four-way effort choice; the style guard refuses a raw select, a rounded-md corner and a py-1.5 gap, and the design says a small bounded enum is a Dropdown so a page never shows two kinds of popup.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
… 0076 and 0090

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
@WaylandYang

Copy link
Copy Markdown
Contributor Author

Reviewed the code diff in full (the bench data aside) and landed two maintainer edits so this can merge:

On the substance: the two-votes-and-a-flag rule for retractions, the once-per-batch glossary with keys per item, the label-as-key fallback, the shortlist as part of the decision basis, and the per-task reasoning effort all read as the measured cuts the README describes, and each has a test at the layer it changes. One thing to keep an eye on rather than block: documents_due now sends every extracted document with no typed facts through errata once, which is the right call for recall but means a large base's first errata pass touches every document; the errata_runs row keeps it to once.

Locally on a fresh database after the merge: CLI 13, server errata 4, phrase alignment 13, store errata 6, extract 84 tests pass; clippy clean. Merging once CI agrees.

@WaylandYang
WaylandYang merged commit ceaa7df into dev Sep 25, 2026
7 checks passed
@WaylandYang
WaylandYang deleted the feat/errata-two-votes branch September 25, 2026 02:57
WaylandYang added a commit that referenced this pull request Sep 25, 2026
…ith tokens by phase (#917)

Two groups on dev at 961c3c0 with gemini-3.5-flash, judged and with
errata: precision 91.0% / 88.5% before errata and 96.0% / 96.5% after;
same-sentence gold recall 11.3% / 13.2% before errata and 18.6% / 22.4%
after, with the errata retractions mostly wrong on this binary (72/98,
164/196), which is what #906's two-vote retraction addresses; 158k and
183k tokens per document. The token table by phase shows where they
went: the alignment window carried the rule proposals' per-item property
tables, the two-vote phrase alignment, and the governance agent deciding
the same pairs from up to ten concurrent runs (9481 decisions on 1421
pairs), fixed in #916.

The bench judge now fails fast when it would read a sealed key from
llm_settings instead of sending the ciphertext and collecting 401s.

Signed-off-by: Wayland Yang <wayland0916@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
WaylandYang added a commit that referenced this pull request Sep 25, 2026
…tokens by phase (#918)

Same corpus, model and judge as the pre-#906 baseline, on dev with #906
and #916. Judged precision 81.0% / 79.0% before errata and 83.0% / 80.5%
after (the loss against the baseline is misworded facts from minimal
reasoning); same-sentence gold recall 14.2% / 14.3% before errata and
17.8% / 17.1% after, with errata now adding without retracting wrongly
(2 and 1 retractions, none judged stated); 31k and 34k tokens per
document, a fifth of the baseline and still five times 0044's ceiling,
seven tenths of it in alignment and rule proposals. 47 and 48 minutes a
group against three to five hours.

Signed-off-by: Wayland Yang <wayland0916@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant