Skip to content

feat(gre-exam): vendor 47 generated p0 items so the full-length mock is buildable - #4

Open
f15cubing wants to merge 85 commits into
mainfrom
agent/exam-demo-items
Open

f15cubing wants to merge 85 commits into
mainfrom
agent/exam-demo-items

Conversation

@f15cubing

Copy link
Copy Markdown
Owner

What & why

Enlarges the vendored held-out exam bank so Exam Mode can build the official full-length (66-item) mock. Regenerates qt/aqt/gre/exam_items.json from the outer eval/bank/items.yaml (now 127 items; p0 = 35 calculus / 18 algebra / 18 additional) and updates the feasibility test — every preset (mini/third/half/full) is now feasible and a full 66-item form assembles from bank items only. Companion outer PR bumps the pin to this commit.

Area(s) touched

qt/aqt/gre/exam_items.json (vendored MCQ data, +752/-0 additive), qt/tests/test_gre_exam.py — engine/Rust PR? no (data + test only; no rslib/proto/pylib/scheduler/undo/store; Exam Mode stays read-only).

Docs updated

In the companion outer PR (f15cubing/speedrun): eval/bank/eval_bank.md, docs/codebase/INDEX.md, README.md, docs/STATUS.md.

Test evidence

  • Exam feasibility (direct module load): p0 pool 35/18/18, max_feasible=66, all presets feasible, full-66 assembles (66 distinct, all eval-*).
  • Outer test_exam_items_sync drift guard: PASS (vendored == bank projection; re-vendor is idempotent).
  • test_gre_exam.py updated (test_vendored_p0_supports_full_length_mock); runs under CI/./ninja check (needs the built app env — assertions verified via direct import here).

Engine/Rust extra gate (if applicable)

N/A — no Rust/collection/proto/scheduler/undo change; Exam Mode endpoints remain read-only (no OpChanges). assert_firewall holds; items are deterministic SymPy (not a live model). Files touched + merge difficulty: 2 files, LOW (additive data + one test).

Made with Cursor

abdnh and others added 30 commits May 6, 2026 15:36
…fe_path

_builtin_data previously did no path validation, allowing directory
traversal attacks via crafted bundled file requests. Extract
ensure_safe_path as a reusable guard and apply it to both
_handle_local_file_request and _builtin_data. UnsafePathException is
caught at the top-level request handler.
---------

Co-authored-by: Andrew Sanchez <andrewsanchez@users.noreply.github.com>
The generator expression used 'host' as the loop variable, shadowing
the outer 'host' from request.headers. Rename to 'h' for clarity.
The 'release' job does not exist in this workflow, so the condition
would always fail when draft-release is true. Remove the stale clause.
Move git config into the commit step since it still needs it.
For patch releases run from non-main branches (e.g. 25.09.3).
…4436)

* fix(build): respect .python-version when generating pyenv

* refactor
This will be addressed later on `main`.
This reverts commit 82bfc32cdcd6eab6dcb885e61ef91485651e40d7.

This doesn't work. Fetch requests in iframes are also ResourceTypeXhr
Serve collection media with a restrictive CSP that sandboxes active documents, blocks script execution, and prevents network access. This means SVG/HTML loaded via iframe or object can still render where the browser allows it, but cannot call back into the editor page or Anki's local API.

Keep add-on web exports exempt from the sandbox, since they share the LocalFileRequest path but are trusted extension assets. The editor page CSP is intentionally left limited to script-src so existing user content such as remote iframe embeds, dictionary sites, YouTube embeds, and SVG object tags is not newly blocked by the editor itself.

Add tests for the untrusted media CSP, local API connect blocking, the add-on exemption, and the editor CSP compatibility expectations.
Felipe Caicedo and others added 24 commits July 2, 2026 12:00
Give the GRE readiness dashboard a distinctive, intentional identity in
place of the default-Anki look: a calibrated "instrument" system where
uncertainty is the visual language. Every metric renders through one
signature component — CalibrationStrip — a shaded 95% band + point tick
on a 0..max axis, used at all three scales (exam headline, per-bucket,
per-leaf). n=0 draws an explicit dotted "not yet" rail rather than a
fabricated position.

- tokens.css: 6-token palette + type roles (system tabular mono for all
  numerals), light/dark via Anki's .night-mode; no bundled fonts.
- CalibrationStrip.svelte + lib.ts (pure geometry) + lib.test.ts (8 vitest).
- Restyle +page/MemoryPanel/CoverageMap/ScoreSlot; amber (not red) give-up
  state; single best-next leaf ringed; restrained page-load reveal
  (prefers-reduced-motion respected).

Pure presentation: no view-model/dashboard_data.py change (that stays the
Thursday scoring layer's surface); still read-only; three scores stay
separate; the state guard still forbids a fabricated readiness number.
…ored)

Vendor the faithful GRE Math Subject Test exam-mode core into the fork so it
runs in (and ships with) the app — the app cannot read the outer eval/bank/ at
runtime, so the logic + items are vendored (the dashboard_data.py + taxonomy.json
pattern).

- exam.py: deterministic, blueprint-matched (ETS 50/25/25) form assembly at the
  official pace (2.58 min/item; presets full/half/third/mini = 66/33/22/11) +
  rights-only scoring (no penalty for wrong/omitted) with per-leaf/bucket
  breakdown + Wilson CI + the attempts record for the scoring seam.
  Headless-importable (no aqt deps), mirrors dashboard_data.py.
- exam_items.json: vendored copy of the authored eval bank (80 items),
  drift-guarded against eval/bank/items.yaml by the outer test.
- test_gre_exam.py: pace/blueprint/assembly determinism/insufficient/firewall/
  rights-only/Wilson/attempts. check:pytest:aqt green.

The SvelteKit exam shell + mediasrv endpoints + dialog + mastery gate are B-2.
…hell

The interactive exam shell on the B-1 core, faithful to the computer-delivered
GRE Math Subject Test: one global countdown, one item at a time, five A–E
single-select options, Mark + free Back/Next, a Review-screen navigator grid, no
calculator, no pause, auto-submit at 0:00, and no per-item feedback (results only
after submit). Presets full/half/third/mini at the official 2.58 min/item pace.

- ts/routes/gre-exam/: +page (session state machine + timer), ItemView,
  Countdown, Navigator, Results (reuses the dashboard CalibrationStrip +
  tokens.css); lib.ts (formatClock/tally) + lib.test.ts.
- mediasrv: read-only greExamForm (assembles a blueprint-matched form; never
  sends the keys to the client) + greExamSubmit (rights-only server-side scoring;
  persists an attempts side-file for the scoring layer; reveals keys after
  submit). Route registered in is_sveltekit_page.
- webview: AnkiWebViewKind.GRE_EXAM + api-access allowlist. gre_exam.py QDialog +
  Tools-menu action. Mastery-gate structure present (threshold 0.0 for now).

Read-only: no collection/OpChanges writes (attempts go to a profile side-file).
Firewalled: draws only from the vendored eval items (partition p0).
Verified: check:svelte + check:pytest:aqt green; vitest lib green; headless GUI
smoke of setup/exam/review/results.
Desktop: first-run auto-import of the bundled GRE study deck
…nki-deckauto-integration

# Conflicts:
#	qt/aqt/main.py
The exam webview rendered items as plain ASCII. Now item math is delimited
LaTeX (\(...\) / \[...\]) typeset by MathJax (SVG output — self-contained,
offline, no external font files), matching the study-deck cards.

- add ts/routes/gre-exam/mathjax.ts: sets the MathJax config then dynamically
  imports the engine (code-split, offline chunk) and exposes typesetMath()
- ItemView / Results typeset their content; +page keys ItemView by item id so
  each item gets a fresh DOM (no Svelte/MathJax churn on re-select)
- exam_items.json re-vendored from the LaTeX-migrated eval bank

No engine/Rust/proto change; read-only exam surface unchanged otherwise.
Replace the bundled study-deck asset with the deterministic LaTeX .apkg. Note
GUIDs are now derived from a stable, rendering-independent uid, so a future
version bump re-imports cards in place (no duplicates). No version bump here —
fresh installs pick up LaTeX; existing installs are unaffected until a
coordinated gre_deck_version bump lands in both apps together.

Asset only; no code/engine change.
…isting installs)

The bundled deck is already the LaTeX build (stable GUIDs). Bump the version gate
so existing installs re-import it on next launch; stable GUIDs make it an in-place
update (no duplicates). Coordinated with the same bump in AnkiDroid (the key syncs
via col.conf, so both apps must agree).
Emulator smoke revealed that upgrading an install whose bundled deck predates the
stable-uid GUID scheme DUPLICATES the deck (old content-hash GUIDs can't be
matched by the new package -> import adds instead of updates: 5.4k -> 10.8k cards).

Fix: stamp the imported GUID scheme under `gre_deck_guid_scheme`. On re-import,
if the stored scheme isn't the current uid scheme, remove the previously-bundled
notes (identified by our two note types) once, then import. Fresh installs and
uid->uid re-imports are unaffected (the cleanup is a no-op / skipped), so history
is still preserved going forward.

Tests: pre-uid re-import no longer duplicates; uid re-import updates in place;
scheme stamped on first import (test_gre_autoimport 6/6).
…version

Addresses review feedback on PR ankitects#31: the migration lived behind the version
guard, so an install already stamped at the current version but with a stale/
absent GUID scheme (and thus possibly duplicated) would early-return and never be
repaired. Gate on BOTH version and scheme (`_is_up_to_date`), so such installs
re-import once and the cleanup heals them.

Test: stale-scheme-at-current-version triggers a repair import without dup (7/7).
…ON to 2026-07-03b

Fresh installs auto-import the interactive MCQ template. Documents the existing-install
template-refresh limitation (byte-deterministic notetype mod => IF_NEWER skips).
…t capacity

Exam Mode never loaded a form. Three faults, root-caused end-to-end:

1. 403 (primary): the exam page's post() sent Content-Type: application/json,
   but mediasrv 403s any POST whose Content-Type != application/binary *before*
   the auth check, so greExamForm/greExamSubmit always 403'd (the dashboard
   works because it sends application/binary). Fix: client posts
   application/binary (JSON body still rides in the request bytes).
2. Body parsing: under application/binary, request.get_json(silent=True)
   returns None, dropping preset/seed/answers. Fix: get_json(force=True).
3. Capacity: the vendored held-out p0 bank (24 items: 8/7/9) can't fill
   full/half/third under the 50/25/25 blueprint, so assembly raised after the
   user picked a preset. Fix: pure feasibility helpers + a read-only
   greExamCapacity endpoint; the setup screen disables presets the firewalled
   bank can't build (calm amber "not enough held-out items yet" state) and the
   server returns an honest locked reason. Held-out bank/scoring/partition
   unchanged; mock stays p0-only and read-only.

Tests: +5 in test_gre_exam.py (pool sizes, feasibility boundary, preset flags,
p0-only regression). Verified via a Flask request-context harness on the built
engine: json+Bearer -> 403, binary+Bearer -> allowed; greExamForm{mini} -> 11
items with answer keys withheld; greExamForm{full} -> honest locked reason.
… (clean locked, not 500)

Per independent review: a forged submit for a preset the firewalled bank can't fill would
raise InsufficientItemsError -> a generic 500. The normal flow never reaches it (submit
only follows a successful greExamForm), but return a clean locked reason regardless.
Verified via a request-context harness: forged `full` -> locked; `mini` still scores.
A read-only desktop explainer (Tools ▸ "How this app differs from FSRS"): we
build on FSRS rather than replacing it — interleaving, a timed exam mode, three
separated scores, and the give-up rule. The interleaving section is interactive
and runs the REAL vendored interleave.py on a fixed example queue.

Fast lane (Qt-UI-only): new gre-method SvelteKit route (reuses the dashboard
tokens + CalibrationStrip) + read-only greMethodInterleave endpoint + GreMethod
QDialog + Tools-menu hook + vendored interleave.py (drift-guarded). No
rslib/proto/scheduler; the endpoint never touches col and returns no OpChanges.
Replace the stale "Arrives Thursday" placeholder with a live, observed
Performance score. The read-only greDashboardData handler now pools the
per-item attempts Exam Mode already persists to the gre_exam_results.jsonl
profile side-file (never the collection) and renders rights-only accuracy as a
Wilson range with n, via the existing CalibrationStrip.

Honesty ceilings held: always a range, never a bare point; a give-up
not_available state (never a fabricated 0) with no attempts; the ScoreSlot
guard still collapses any range-less state to not_available. This is the
observed low-n surface, deliberately NOT the calibrated logistic+Platt model in
scoring/performance.py (which needs a multi-student attempt corpus).

- dashboard_data.py: pure load_exam_attempts() + observed_performance();
  build_view_model() gains an exam_attempts param
- mediasrv.py: greDashboardData reads the side-file (best-effort, read-only)
- ScoreSlot.svelte: renders CalibrationStrip for an "observed" state
- 9 new unit tests (dashboard_data suite 20; 23/23 with mediasrv); check_svelte
  + check_pytest_aqt green

Read-only; no OpChanges; three scores stay separate. Pre-existing ruff-format
debt in dashboard_data.py/mediasrv.py at the base is left untouched.
…r (PRD §8, D5)

Off by default (col.conf["gre_interleave"]) -> byte-identical to upstream
(fetch_limit 1, no reorder). When on, the reviewer fetches a small lookahead
window and reorders ONLY the REVIEW cards of the v3 queue for confusable-type
dispersion via the tested aqt.gre.interleave module. Pure presentation:

- never writes scheduling state / the collection / the undo stack (only permutes
  an already-returned QueuedCards batch in memory);
- each QueuedCard (card+states+context) moves as a self-contained unit, so the
  shown card is always paired with its own scheduling states;
- NEW / LEARNING cards keep their scheduler positions (intraday timing intact);
- bails out (no-op) on <3 review cards, a homogeneous cluster, or any review card
  missing a topic:: leaf tag.

A checkable Tools-menu toggle ("GRE: Interleave reviews (ablation)") flips the
flag: this is the ablation's interleaved arm; off is the blocked arm.

9 unit tests (flag gate, fetch-limit switch, dispersion, multiset invariant,
new/learning preserved, all safe fallbacks). Live GUI click-through is the one
human smoke step (offscreen QtWebEngine won't init headlessly here).

No Rust/proto/scheduler change; no OpChanges; undo untouched.
…ch card (PR ankitects#57 review)

Addresses the different-agent review of PR ankitects#57 (verdict CORRECT; two non-blocking
recommendations):

- Split reorder_output into a pure _interleaved_cards (computes the permutation with
  NO mutation) + a thin writer. The write-back (del + extend) happens only after a
  successful pure computation, and the whole thing is wrapped so any unexpected error
  falls back to the scheduler's original order -- an interleaving bug can never empty
  the queue or interrupt the review loop.
- Strengthen the tests: _mk_output now stamps a distinctive context.seed per card and
  a new test asserts every card still carries its own context after the reorder,
  directly proving the load-bearing invariant (each QueuedCard moves as a
  self-contained unit) rather than only checking (queue, note_id).

10/10 tests green; ruff clean. Still pure presentation: no col write, no OpChanges,
no scheduling/undo change.
…esh)

A wrong multiple-choice answer is a lapse: it must only ever grade Again(1),
never Hard/Good/Easy. The graded MCQ template already offered a single Continue
on a wrong tap; this closes the remaining desktop path -- Anki's built-in bottom
answer bar and its keyboard shortcuts, which the card webview cannot touch.

- aqt/gre/mcq_lockdown.py: pure verdict/clamp/restrict helpers (unit-tested).
- reviewer.py: capture the template's guarded gremcq: correctness hint; clamp a
  wrong MCQ's ease to Again (button/keyboard/auto-advance all funnel through
  _answerCard) and collapse the bottom bar to Again. No col write, no OpChanges,
  no scheduler/proto/Rust change -- undo and the collection are untouched.
- deck_autoimport.py: refresh bundled note-type templates in place on existing
  installs (reads the bundled .apkg, applies qfmt/afmt/css via models.update_dict;
  gated by gre_deck_template_revision; no deck re-import, review history preserved).
- data/gre-study-deck.apkg: rebundled so the graded template emits the hint.

Correct MCQ keeps Hard/Good/Easy (FSRS needs the rating); non-MCQ cards untouched;
AnkiDroid (no pycmd) is a no-op and falls back to the built-in ease buttons.
Tests: qt/tests/test_gre_mcq_lockdown.py (13) + pipeline test_mcq_notetype.py.
…ashboard

The GRE dashboard adopts the "Readout" identity from the redesign brief: a calibrated
CAS/terminal printout where the monospaced data face is the hero and every metric is a
range in math notation that shows its work.

- fonts.css + fonts/: bundle JetBrains Mono + Inter (SIL OFL) locally as woff2 — fully
  offline, no CDN (verified emitted into the sveltekit build). Only used weights shipped.
- tokens.css: the Readout token system (light + .night-mode dark) — ink/muted/faint,
  teal signal, amber abstain; JetBrains Mono / Inter families.
- CalibrationStrip.svelte: interval in math notation (∈ [lo, hi]) + a faint method tag.
- +page / MemoryPanel / ScoreSlot: mono-hero masthead + labels; "Wilson" method tag on
  the Memory headline; refactored two nested ternaries (eslint) with no behaviour change.

Pure presentation — no dashboard_data.py / view-model change; the no-fabrication guard,
three-separate-scores, and amber-abstain honesty rules are untouched. Green on changed
files: check:svelte, check:eslint, check:vitest, prettier. Spec:
docs/superpowers/specs/2026-07-05-readout-identity-design.md.
Applies the shipped Readout identity (ankitects#61) to two more surfaces (fast lane, Qt-UI-only):

- Exam mode (ts/routes/gre-exam): imports the bundled fonts.css so the simulator renders
  in JetBrains Mono like the dashboard, and the one global countdown is sized as the focal
  instrument (big mono clock, still amber < 5:00). It already reused the dashboard tokens,
  so the Readout palette flowed automatically.
- Deck browser (qt/aqt/deckbrowser.{py,scss}): finishes the in-progress home-screen
  redesign in the Readout identity — a mono "Study decks" title + eyebrow (replacing the
  serif + graph-paper direction) on the clean elevated deck-list with tabular counts.
  System mono stack (JetBrains Mono if installed); bundling the exact woff2 into this Qt
  webview is a follow-up.

Pure presentation; no collection/scheduler/scoring touch. check:svelte green;
deckbrowser.scss compiles. MCQ card (surface 4C) is the remaining cascade — deferred as
an engine-lane deck rebundle.
The interactive MCQ card adopts the Readout identity: the A-E option letters + key/verdict
render in the monospaced data face on the calibrated instrument palette (hairlines, teal
signal, amber); correct/wrong keep semantic green/red; light + dark. The bundled deck is
re-bundled and `_TEMPLATE_REVISION` bumps to `2026-07-05b-readout-mcq`, so existing installs
refresh the template in place (no re-import; history preserved). Template source is
`pipeline/build_deck.py` (outer). No importer-logic change; no collection/scheduler touch
beyond the existing text-only note-type refresh.
The interactive graded-MCQ card grades in one tap by firing pycmd("ans")
then pycmd("ease<N>") back to back. The normal ans reveal is async
(evalWithCallback -> _onTypedAnswer -> _showAnswer), so the ease grade
arrived while reviewer state was still "question" and _answerCard dropped
it (it requires state == "answer") -- forcing a second grade on the
built-in bottom bar. Reveal synchronously on ans when a graded-MCQ verdict
is already recorded, so the following ease lands. Non-MCQ ans keeps the
normal async type-answer reveal; QWebChannel delivers the two pycmd
messages in order + synchronously, so the reveal completes before the grade.

Pure helper should_reveal_synchronously(verdict, state) in mcq_lockdown.py
+ 2 regression tests (test_gre_mcq_lockdown.py: 15 passing). Qt-UI-only,
no scheduler/undo/store/proto change; grading still flows through the
normal FSRS answer_card path.
Add a GRE Home surface (Tools ▸ GRE Home; auto-opens on startup) that
surfaces the three separated scores + study stats + a one-click "study
next" (a filtered-deck review of the best next topic), and gate timed
Exam Mode behind >=70% studied topic coverage.

Presentation-only, no engine change:
- dashboard_data.py: pure additions (studied_coverage, study_next,
  stats_block, leaf labels); build_view_model emits stats + study_next.
- exam.py: MIN_STUDIED_COVERAGE=0.70 + coverage_meets_threshold /
  coverage_lock_reason; enforced server-side in greExamForm, surfaced by
  greExamCapacity + a locked panel on the gre-exam setup screen.
- mediasrv.py: composed read-only greHomeData endpoint (dashboard vm +
  coverage block + startup flag; one mastery RPC).
- webview.py: AnkiWebViewKind.GRE_HOME + api-access allowlist.
- gre_home.py + ts/routes/gre-home: dialog (bridge actions incl. the
  filtered-deck study-next) + landing page + startup auto-open.

Three scores stay separate; Readiness never a bare number. Writes limited
to a standard undoable filtered-deck op and one col.conf startup flag; no
rslib/proto/scheduler change; read RPCs return no OpChanges.
@f15cubing
f15cubing force-pushed the agent/exam-demo-items branch from bc995b8 to a4e8e61 Compare July 5, 2026 22:10
…buildable

Regenerate the vendored exam_items.json from eval/bank/items.yaml (now 127
items; p0 pool = 35 calculus / 18 algebra / 18 additional), which lets Exam
Mode assemble the official 66-item form under the 50/25/25 blueprint. Update
the feasibility test accordingly: every preset (mini/third/half/full) is now
feasible and a full 66-item form assembles from bank items only.

Data + test only — no engine, scheduler, undo, store, or proto change; Exam
Mode stays read-only. The new items are deterministic, correct-by-construction
SymPy (gen: generated, demo: true), firewalled from the study deck.
@f15cubing
f15cubing force-pushed the agent/exam-demo-items branch from a4e8e61 to e4efa0f Compare July 5, 2026 22:15
@f15cubing

Copy link
Copy Markdown
Owner Author

Integrated via the outer pin bump in f15cubing/speedrun#65 (merged): the outer repo now pins anki@e4efa0f6f, which is this branch's HEAD (rebased onto the current GRE line so ankitects#64's one-click MCQ fix and ankitects#66's 70% coverage lock are preserved). Per this fork's workflow the GRE line is never merged into main (which tracks upstream Anki); the pin bump is the integration. Leaving this PR open as the review/diff artifact; the branch is retained to keep the pinned commit reachable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants