Give the typed-graph bench a warm-base mode and count tokens by phase from the server log - #920
Merged
Merged
Conversation
… from the server log `--into <kb>` ingests a second batch into a base that already ran, extracts only those documents, runs alignment as the product would (decided signatures with an unchanged basis are not re-asked), and scores only the new documents; a fact counts for every document its evidence names, and the judge samples only facts evidenced in this batch and reads them against this batch's text. `fetch-redocred.mjs` gains `--name` and `--exclude` for a second batch that shares no document with the first and does not rewrite the ontology; batch redocred-100b is committed. With BENCH_SERVER_LOG the result carries model tokens per phase and per document from the server's usage lines. Measured on run 3's base: 25k tokens per document against 31k on a fresh base, alignment 14k against 22k, because the second hundred documents bring about 800 new signatures, nearly as many as the first. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The missing reading named in #918: the per-document cost on a base that already ran, where alignment is paid per signature rather than per document.
typed.mjs --into <kb> --corpus redocred-100b: ingests a second batch into an existing base, extracts only those documents, runs alignment as the product would (decided signatures with an unchanged basis are not re-asked), scores only the new documents (a fact counts for every document its evidence names), and judges only facts evidenced in this batch, read against this batch's text.--into … --scorere-scores an ingested batch.fetch-redocred.mjs --name --exclude: a second batch that shares no document with the first, filenames under its own prefix, ontology file untouched;corpora/redocred-100b.jsonand its truth are committed (3600 gold facts, all on the base's 95 properties).BENCH_SERVER_LOG: the result JSON carries model tokens per phase (extract, align, errata) and per document from the server'sllm usagelines; the console prints the line.Measured on run 3's base (README, 温库第二批): 25k tokens per document against 31k fresh, alignment 14k against 22k. The second hundred documents bring about 800 new signatures (bound 426 → 674, none 1178 → 1698), nearly as many as the first hundred: Re-DocRED's phrase signatures barely repeat across documents, so amortisation is small on this corpus. Judged precision 80.0%; same-sentence recall 11.7% after errata, lower than the first batch because signatures already decided none are not re-asked.
🤖 Generated with Claude Code