Record the post-#906 typed-graph bench: two groups, thresholds read, tokens by phase - #918
Merged
Merged
Conversation
…tokens by phase Same corpus, model and judge as the pre-#906 baseline, on dev with #906 and #916. Judged precision 81.0% / 79.0% before errata and 83.0% / 80.5% after (the loss against the baseline is misworded facts from minimal reasoning); same-sentence gold recall 14.2% / 14.3% before errata and 17.8% / 17.1% after, with errata now adding without retracting wrongly (2 and 1 retractions, none judged stated); 31k and 34k tokens per document, a fifth of the baseline and still five times 0044's ceiling, seven tenths of it in alignment and rule proposals. 47 and 48 minutes a group against three to five hours. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
This was referenced Sep 25, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two full groups (100 documents, judge 200, errata) on dev at b9e3aae, with #906 and #916, same corpus, model and judge as the pre-#906 baseline recorded in #917. Written into the bench README as the formal cut-2 measurement.
minimalreasoning on extraction.🤖 Generated with Claude Code