feat(exam): generate 47 demo items to unlock the full-length GRE mock - #65
Merged
Merged
Conversation
f15cubing
force-pushed
the
agent/exam-demo-items
branch
from
July 5, 2026 22:11
3cd0393 to
68bfb73
Compare
added 3 commits
July 5, 2026 17:15
Exam Mode could only build the 11-item mini mock because the firewalled held-out p0 bank was too small (8 calculus / 7 algebra / 9 additional -> the 50/25/25 blueprint caps a form at 16). This adds 47 deterministic, correct-by-construction demo items (eval-p0-gen-*, gen: generated, demo: true) via generate_eval.gen_demo_p0_items, bringing p0 to 35/18/18 so the official 66-item form (and half/33, third/22) become feasible. The items are SymPy-exact (NOT a live model — same AI-off posture as the AI card pipeline), authored in a high-coefficient / distinct-phrasing regime disjoint from the study deck so assert_firewall holds. demo: true + the id prefix let the scoring layer exclude them from real calibration folds. Adds eval/bank/vendor_exam_items.py (reproducible items.yaml -> vendored anki/qt/aqt/gre/exam_items.json projection; the drift guard stays green) and bumps the anki submodule pin to bc995b88. Data + generator + tests only; no scheduler/undo/store/proto/Rust change.
f15cubing
force-pushed
the
agent/exam-demo-items
branch
from
July 5, 2026 22:16
68bfb73 to
0ef8ab7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Exam Mode could only build the 11-item mini mock — the firewalled held-out
p0bank was too small (8 calc / 7 alg / 9 add; the 50/25/25 blueprint caps a form at 16). This adds 47 deterministic, correct-by-construction demo items (eval-p0-gen-*,gen: generated,demo: true) viagenerate_eval.gen_demo_p0_items, bringing p0 to 35/18/18 so the official 66-item mock (and half/33, third/22) become feasible. Not a live model (same AI-off posture as the AI card pipeline); firewall-safe;demo: true+ id-prefix let the scoring layer exclude them from real calibration folds. Companion submodule PR: f15cubing/anki#4 (bumps the pin tobc995b88).Area(s) touched
eval/bank/{generate_eval.py,items.yaml,vendor_exam_items.py,eval_bank.md,tests/test_generate_eval.py,tests/test_bank_composition.py},docs/{STATUS.md,codebase/INDEX.md},README.md, and theankisubmodule pin bump — engine/Rust PR? yes (bumps theankisubmodule) but data-only: no rslib/proto/pylib/scheduler/undo/store change.Docs updated
eval/bank/eval_bank.md(Composition → 127; new Demo-items section;vendor_exam_items.py),docs/codebase/INDEX.md(eval-bank row →agent/exam-demo-items),README.md(pin →anki@bc995b88),docs/STATUS.md(in-flight entry). Note: the pre-existing INDEX composite-SHA drift for engine areas (rslib/proto/pylib) is left as-is — this change didn't re-verify them.Test evidence
test_exam_items_sync1/1 (repo.venv, sympy 1.14 + PyYAML).test_demo_p0_items_wellformed_and_deterministic(47 items, 27/11/9, deterministic),test_p0_supports_full_length_exam_blueprint(p0 >= 33/17/16).max_feasible=66; full-66 assembles (alleval-*).assert_firewallPASS across 127 items (incl. 47 demo) — no study-deck/ETS collision.vendor_exam_items.pyidempotent (re-run → no diff); reproduces the prior JSON byte-for-byte before the additions.Engine/Rust extra gate (if applicable)
N/A for behavior — the submodule bump carries only vendored MCQ data + one test; no Rust/collection/proto/scheduler/undo change; Exam Mode stays read-only (no
OpChanges). The aqttest_gre_exam.pyruns under CI/./ninja check. Files touched + merge difficulty: LOW (additive).Made with Cursor