Game Summary on device: measure both runtimes, and fix what was being measured as model quality

[ber4444]

Measures Game Summary on both on-device runtimes, fixes the three bugs that first-pass measurement mistook for model quality, adds the response validator this surface never had, and leaves the attach switched off behind a per-surface constant.

The finding: three bugs were measured before either model was

The first pass reported 7/12 for AICore and 4/12 for Foundation Models, and concluded that Foundation Models "refuses two games in three and truncates every success". All three failure modes were ours.

1. [ErrorCode 30] Background usage is blocked — AICore refuses to generate unless the app is foregrounded. adb shell am start on a locked device produces a full JSONL of sub-second fallbacks that measure nothing: 3 rows of the first run, all 12 of the second. svc power stayon usb does not prevent it, because the keyguard rather than the screen timeout is what stops the activity being resumed. The two rows first reported as "inference exceeded latency budget" at ~46 s were the cold start immediately after the blocked ones.
2. noRepeatNgramSize = 4 — B15's guard, tuned for the coach's single sentence. A summary of three turning points is a parallel list, and a parallel list repeats its connectives by construction, so it cut real answers at the start of the second "so this was another small inaccuracy". 4 of 4 Foundation Models successes, 1 of 7 AICore.
3. The raw PGN in the prompt — the largest effect, in two directions. Foundation Models rejected 8 of 12 prompts with "An unsupported language or locale was used" — the same 8 across two runs, in 15–20 ms, which is an input guardrail and not a model deliberating. And on Android the PGN was the sole source of invention: every unsourced flourish ("in the endgame", "contributed to the loss" on a game with no result) was the model narrating movetext it had been told not to use.

Removing the PGN took Foundation Models from 4/12 to 12/12 and Android's unsourced-narrative rows from 6/12 to 0/12.

The numbers, at 50 games

	Android AICore	iOS Foundation Models
Succeeded	43/43 †	50/50
Cites exactly the code-chosen plies	37/43 (86%)	18/50
Invented a [move-N]	0	2/50
No citation at all	0	28/50
First person ("I played")	0	26/50
Truncated	0	0
Latency median / p95	11.4 s / 13.2 s	1.4 s / 2.2 s

† 7 of 50 rows lost to one contiguous ErrorCode 30 block despite the foreground fix. A contiguous block of sub-second fallbacks is the harness every time.

The two invented tags are the finding. At 12 games iOS had none and the citation gap looked like omission; at 50 it is fabrication, and a fabricated [move-N] is the one defect a user cannot detect — B16 turns it into a board jump to a ply that was never a turning point. Android fabricated none in 43. At 50 games the iOS tail is also worse than the small sample suggested: some rows are not summaries at all, but refusals ("I'm sorry, but I cannot summarize the mistakes in the game") and apologies in character ("I apologize for the mistakes I made… I will try to improve").

What changed in the product

• The summary prompt no longer contains the PGN; it opens with one factual line (side, ply count) and the turning points, which already carry every fact the summary may state.
• noRepeatNgramSize is 8 on this surface, and trimIncompleteSummaryTail drops a ragged tail if anything else ever cuts one.
• GameSummaryResponseValidator — the surface had none, and it is live on iOS today. Five rules: citation set, coverage, move attribution, voice, piece type. A rejection lands on GameSummaryGrounding, which is what ships today, so a false rejection costs the wait and nothing else. Measured: accepts 37/43 AICore, rejects 42/50 Foundation Models.
• The deterministic floor is fixed. It shipped "This was a inaccuracy." and "This was a good." on both platforms, and a MISTAKE could be followed by "This move was slightly inaccurate" one sentence later, because the class comes from cpLoss and the intuition from a win-percent delta.
• MainActivity.attachOnDeviceAi() holds probe → executor → both orchestrators behind ATTACH_MOVE_COACH and ATTACH_GAME_SUMMARY, both false. The probe previously had no caller outside bench/. Three details are load-bearing and documented at the call site: the constant sits inside the probe because probeAvailableLocalVendors() awaits an AICore feature fetch on a DOWNLOADABLE device; isAppForegrounded is read per request so a backgrounded request falls back instead of erroring; and the vendor list is probed once and reused. Modelled on googlesamples/mlkit android/genai, which is also where the Candidate.finishReason == MAX_TOKENS signal we were dropping comes from.
• Bench harness: keepBenchInForeground() (bench-only) holds FLAG_KEEP_SCREEN_ON + setShowWhenLocked.
• ScorecardWriter now matches the scorecard.md note hand-edited in ML Kit on Android: availability was a client-config bug, and the "repetition loop" was ours #137, so the next :evals:run stops reverting it.

Two validator rules were built, measured, and removed

Both passed their own unit tests and produced zero true positives against 43 real summaries:

• Class fidelity (rejecting a BLUNDER described as an inaccuracy) cannot see negation, so it rejected "While these aren't huge blunders…" and "[move-17] with Qc2 wasn't a blunder"; and its per-tag segmentation ran to the next [move-N], so the last citation's segment swallowed the closing paragraph and matched "focusing on those moments of inaccuracy".
• Best-move-described-as-played required the engine's move to appear after "instead of", so the standard counterfactual "opting for e4 instead of Nd2 would have been a stronger choice" — correct English and correct chess — was rejected on six summaries.

Together they accepted only 58% of output the benchmark had already judged good, which would have meant one summary in two costing a 12 s wait and then showing the composed text anyway.

GameSummaryValidatorFieldTest is the check that found this and the reason it cannot recur: it replays 43 AICore and 20 Foundation Models summaries from the 50-game run, with the turning points the device computed, and fails the build if acceptance drops below 85% or if any rule other than incomplete coverage fires on an AICore summary.

Recommendation

Do not flip either constant in this PR. Android is now a genuine trade rather than a loss — 12 s and a coach's voice against instant and robotic, citations intact either way. The remaining question is not quality but whether one summary in seven costing an 11 s wait for the composed text is a good deal; the validator makes that outcome safe, not free.

iOS is not ready, and it already ships: first person in half the answers, no citation in half, and two fabricated board jumps. The validator now catches all of it, which is the argument for merging this regardless of any attach decision. The fix for iOS is the prompt, not the validator — see the plan.

Next steps

docs/plans/on-device-ai-next-steps.md:

1. Write GameSummaryResponseValidator — done in this PR, with the two removed rules recorded.
2. Turn AICore on for Game Summary — gated on a repeat of the 50-game run with the validator in place. Zero invented tags is a hard gate, not a percentage.
3. Benchmark Rules Q&A — the only surface with a model live in production on both phones and the only one never measured. Fixture design, row schema, what to score, and the decision it makes: whether the model turn earns its place at all, given the retrieval floor already answers correctly with a citation.

Leftovers, not addressed here

• DeviceRunScorer has no Game Summary mode — summary runs are scored ad hoc, the "second scorer written next to the data" problem :evals exists to prevent.
• evals/scorecard.md's aicore-nano-fast row is stale (0.0% grounded) — it measured three separate bugs plus a preference that FEATURE_NOT_FOUNDs on a Pixel 10.
• Summary fixtures aren't staged into Android assets like the coach's golden set, so a run needs a manual run-as push.
• Rules Q&A has no availability probe on Android — the only surface that decides availability by failing.

Testing

All CI green on 06c01aa9: Android emulator suite, iOS simulator + Swift tests, the grounding regression gate, and CodeQL.

The field test runs in CI through :ondeviceai:check in the Android job — worth knowing, because the apple job's task list does not include :ondeviceai:desktopTest, so check is the only thing that executes it.

Locally: :ondeviceai:check (all targets), :evals:test, :evals:run (regenerates scorecard.md with no drift) and :androidApp:assembleDebug.

Benchmarks ran on a Pixel 10 Pro XL and an iPhone 17 Pro simulator (iOS 26.5); raw JSONL stays in build/bench/ per the existing convention, with repro commands in the benchmark doc. The 63 rows the validator is tested against are checked in at onDeviceAi/src/desktopTest/resources/game-summary-field-corpus.jsonl.

🤖 Generated with Claude Code