fix(doctor,storage): a broken AI feature nobody was told about (D13, D15) - #293
Merged
Conversation
kevintseng
force-pushed
the
fix/telemetry-and-lesson-history
branch
3 times, most recently
from
September 4, 2026 05:54
0ff665e to
4a82a16
Compare
…ecall history on empty shells D13 — `llm_telemetry` was written by every Smart-Mode flow and read by nothing that could alert anyone. `memesh telemetry` shows it on request, which means a broken flow needed someone to think to ask. Measured on the maintainer's graph on 2026-09-02, over a 7-day trailing window: `guard_proposer` 69/69 failed, `dreamer` 47/47, `auto_tagger` 20/20, `failure_analyzer` 8/8 — all `bad_request`, all starting 2026-08-28, and `memesh doctor` said nothing. The window is 7 days rather than the CLI's own 30-day default for a measured reason: at 30 days, `dreamer`'s stale successes from 2026-08-23 dilute the rate to "36% succeeded" and hide that it has been fully broken all week. `transcript_extractor` (96.6% healthy over its lifetime, real background noise) had no calls inside the window at all, so it stays silent — correctly, not by luck. D15 — `splitFusedLessons` moved a fused bucket's observations into their own entities but left the emptied shell's `recall_hits`/`recall_misses` behind. Of 49 archived shells, 4 carried stale counters (3/61, 1/35, 3/1, 0/1). The counters are not merely inert: `scripts/audit/measure-signals.mjs` sums them over ALL entities with no status filter, so the stale number pollutes that signal forever. Zeroed, not divided and not copied onto the successors. A hit is recorded against the bucket id, never against one observation inside it, so dividing fabricates a fraction nobody earned and copying claims each successor independently earned the whole bucket's record. The real numbers are preserved under `metadata.retired_recall`. The default recall path was never affected — archived shells lose their FTS row and `search()` filters to active — only the audit aggregate and any `includeArchived: true` read. The two archive-point mutations below were green until this commit added the tests that catch them. Both wirings run inside the same `openDatabase`, and the one-shot repair runs SECOND — on a fresh database it silently rescued anything the archive points failed to retire, so every existing test passed with both call sites removed. But the repair is a `runOnceMigration`: after it stamps it never looks again, so a bucket fused after the upgrade — the ordinary case from then on — is protected by the archive point and nothing else. The new tests seed exactly that state (repair already stamped, split still owed) and assert the stamp did not move, so they cannot quietly stop isolating what they claim to isolate. [Verified-By: node scripts/run-tests-isolated.mjs > /tmp/f13.log 2>&1; echo exit=$? -> exit=0; "Test Files 230 passed | 1 skipped (231)", "Tests 3295 passed | 11 skipped (3306)", no Errors line] [Verified-By: npm run verify:release > /tmp/v13b.log 2>&1; echo exit=$? -> exit=0, ending "consumer install has no high-or-worse advisories"] [Verified-By: invariant vs the unrepaired snapshot -> exit=1 naming the 4 shells; vs the repaired copy -> exit=0. Repair on a copy: global SUM(recall_hits)/SUM(recall_misses) 97/561 -> 90/463, exactly the 4 shells and nothing else; 26 successors checked, none touched] [Verified-By: break-tests, mutate -> run -> restore, byte-identical restore asserted by reading the file back: disable the doctor detector entirely (dist/ rebuilt, since that test spawns the built CLI) -> exit=1, restored exit=0; drop retireRecallHistory from archive point A -> exit=1, restored exit=0; from archive point B -> exit=1, restored exit=0; from the one-shot repair -> exit=1, restored exit=0]
`inspectLlmTelemetryHealth` swallows a failed read of `llm_telemetry` and returns undefined — a database from before the table existed has nothing to diagnose. Every test in doctor.test.ts gives the stub an empty table, so the catch was never entered and turning it into a rethrow left all 149 tests green. This adds `telemetryUnreadable` to the stub and one test that asserts the failure neither becomes a finding nor leaks into runDoctor's outer catch as a database error. [Verified-By: node scripts/run-tests-isolated.mjs tests/core/doctor.test.ts --maxWorkers=1 -> Tests 150 passed, exit=0]
The graph-repairs.test.ts comment explaining the archive-point/shell-repair interaction claimed the ordinary case forever after the upgrade is protected by the archive point alone -- but the current code no longer creates new -other buckets at all (learn() moved to digest naming in 4.8.2), and the shell repair is a version-1 runOnceMigration that already stamped, not something still running. What the archive point actually guards is the NEXT time this split logic's own version bumps past 2, not an ongoing steady state. Rewritten to say that instead of a claim the code no longer supports. Added a real-browser (real HTTP server, real Chromium, all 11 locales) check that llm-telemetry.silent-failure renders translated in the dashboard, not as a raw i18n key or a silent English fallback -- something the existing catalogue tests check at the translation-table level but can't confirm a real bundle actually renders. Triaged the new script in verification-audit's baseline as a deliberately eg-invoked QA tool, not a missing CI gate; pruned one baseline entry that no longer matches current line numbers.
The rebase onto origin/main (post-#285 merge)'s own conflict resolution took the wrong side for these two map files; a full rebuild produces the correct byte content.
kevintseng
force-pushed
the
fix/telemetry-and-lesson-history
branch
from
September 4, 2026 06:21
4a82a16 to
39f0637
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
D13 — four AI-backed features had failed every call for days, and nothing said so
llm_telemetryis written by every Smart-Mode flow and was read by nothingthat could alert anyone.
memesh telemetryshows it on request, whichmeans a broken flow needed someone to think to ask.
grep -n "llm_telemetry" src/core/doctor.tsfound nothing before this PR.Measured on the maintainer's graph on 2026-09-02, 7-day trailing window:
guard_proposerdreamerauto_taggerfailure_analyzerAll
bad_request, all starting 2026-08-28.memesh doctorreported a cleanbill of health throughout.
Why 7 days and not the CLI's own 30-day default
At 30 days,
dreamer's stale successes from 2026-08-23 dilute the rate to"36% succeeded" and hide that it has been fully broken all week. The shorter
window is the difference between seeing this and not.
transcript_extractor— 114/118 = 96.6% healthy over its lifetime, realbackground noise — had no calls inside the window, so it stays silent.
Correctly, not by luck: a test pins that an ordinary mixed failure rate is not
flagged, and another pins that failover attempts are counted as calls, not as
attempts (2 calls × 2 providers must not read as 4 and trip a 3-call
threshold).
D15 — emptied lesson shells kept the recall history of the bucket they used to be
splitFusedLessonsmoves a fused bucket's observations into their own entitiesbut left the emptied shell's
recall_hits/recall_missesbehind. Of 49archived shells, 4 carried stale counters: 3/61, 1/35, 3/1, 0/1.
Not merely inert —
scripts/audit/measure-signals.mjssums those columns overall entities with no status filter, so the stale number pollutes that
signal permanently.
Zeroed, not divided and not copied onto the successors. A hit is recorded
against the bucket id, never against one observation inside it: dividing
fabricates a fraction nobody earned, and copying claims each successor
independently earned the whole bucket's track record. The real numbers are
preserved under
metadata.retired_recall.Scope, stated precisely: the default recall path was never affected — an
archived shell loses its FTS row and
search()filters to active by default.What was wrong is the audit aggregate and any
includeArchived: trueread.Repair, on a copy of the snapshot
The part worth reading: two guards that could not fail
Break-testing this PR found that removing
retireRecallHistoryfrom bothlive archive points in
splitFusedLessonsleft every existing test green —including the two tests written specifically for those points.
Both wirings run inside the same
openDatabase, and the one-shot repair runssecond. On a fresh database it silently rescued anything the archive points
failed to retire, so a test that opens once cannot tell the two apart.
But the repair is a
runOnceMigration, already stamped at its current version(1) — it will not fire again as-is. That is not what protects an ordinary
bucket the runtime fuses today: the current code no longer creates new
-otherbuckets at all, so this state only matters for the next timeFUSED_LESSON_SPLIT_KEYbumps past 2 (this split logic changing again) whilethe shell repair stays stamped and can no longer rescue what it misses — the
archive points then have to retire the history on their own.
The two new tests seed exactly that state (shell repair already stamped, split
still owed) and each asserts the stamp did not move during the run, so they
cannot quietly stop isolating the thing they claim to isolate.
Verification, at
a317f5d5in the PR worktreeBreak-tests — mutate, run, restore, byte-identical restore asserted:
dist/rebuilt — that test spawns the built CLI)retireRecallHistoryfrom archive point A (the-otherbucket loop)The middle two were exit=0 before the tests in this PR were added. That is
the finding, not a footnote.
Dashboard strings
The new doctor code has catalogue entries in all 11 locales rather than falling
back to raw English, matching how this project already closed the same gap for
sibling codes.