Skip to content

fix(doctor,storage): a broken AI feature nobody was told about (D13, D15) - #293

Merged
kevintseng merged 6 commits into
mainfrom
fix/telemetry-and-lesson-history
Sep 4, 2026
Merged

fix(doctor,storage): a broken AI feature nobody was told about (D13, D15)#293
kevintseng merged 6 commits into
mainfrom
fix/telemetry-and-lesson-history

Conversation

@kevintseng

@kevintseng kevintseng commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

D13 — four AI-backed features had failed every call for days, and nothing said so

llm_telemetry is written by every Smart-Mode flow and was read by nothing
that could alert anyone. memesh telemetry shows it on request, which
means a broken flow needed someone to think to ask. grep -n "llm_telemetry" src/core/doctor.ts found nothing before this PR.

Measured on the maintainer's graph on 2026-09-02, 7-day trailing window:

flow recent calls succeeded
guard_proposer 69 0
dreamer 47 0
auto_tagger 20 0
failure_analyzer 8 0

All bad_request, all starting 2026-08-28. memesh doctor reported a clean
bill of health throughout.

Why 7 days and not the CLI's own 30-day default

At 30 days, dreamer's stale successes from 2026-08-23 dilute the rate to
"36% succeeded" and hide that it has been fully broken all week. The shorter
window is the difference between seeing this and not.

transcript_extractor — 114/118 = 96.6% healthy over its lifetime, real
background noise — had no calls inside the window, so it stays silent.
Correctly, not by luck: a test pins that an ordinary mixed failure rate is not
flagged, and another pins that failover attempts are counted as calls, not as
attempts (2 calls × 2 providers must not read as 4 and trip a 3-call
threshold).

D15 — emptied lesson shells kept the recall history of the bucket they used to be

splitFusedLessons moves a fused bucket's observations into their own entities
but left the emptied shell's recall_hits/recall_misses behind. Of 49
archived shells, 4 carried stale counters: 3/61, 1/35, 3/1, 0/1.

Not merely inert — scripts/audit/measure-signals.mjs sums those columns over
all entities with no status filter, so the stale number pollutes that
signal permanently.

Zeroed, not divided and not copied onto the successors. A hit is recorded
against the bucket id, never against one observation inside it: dividing
fabricates a fraction nobody earned, and copying claims each successor
independently earned the whole bucket's track record. The real numbers are
preserved under metadata.retired_recall.

Scope, stated precisely: the default recall path was never affected — an
archived shell loses its FTS row and search() filters to active by default.
What was wrong is the audit aggregate and any includeArchived: true read.

Repair, on a copy of the snapshot

global SUM(recall_hits)/SUM(recall_misses): 97/561 → 90/463   (Δ −7/−98)
rows changed: 4 — exactly the 4 shells, nothing else
successors with nonzero recall: 0 before, 0 after (26 checked, none touched)

The part worth reading: two guards that could not fail

Break-testing this PR found that removing retireRecallHistory from both
live archive points in splitFusedLessons left every existing test green —
including the two tests written specifically for those points.

Both wirings run inside the same openDatabase, and the one-shot repair runs
second. On a fresh database it silently rescued anything the archive points
failed to retire, so a test that opens once cannot tell the two apart.

But the repair is a runOnceMigration, already stamped at its current version
(1) — it will not fire again as-is. That is not what protects an ordinary
bucket the runtime fuses today: the current code no longer creates new
-other buckets at all, so this state only matters for the next time
FUSED_LESSON_SPLIT_KEY bumps past 2 (this split logic changing again) while
the shell repair stays stamped and can no longer rescue what it misses — the
archive points then have to retire the history on their own.

The two new tests seed exactly that state (shell repair already stamped, split
still owed) and each asserts the stamp did not move during the run, so they
cannot quietly stop isolating the thing they claim to isolate.

Verification, at a317f5d5 in the PR worktree

node scripts/run-tests-isolated.mjs   -> exit=0
  Test Files  230 passed | 1 skipped (231)
  Tests  3295 passed | 11 skipped (3306)
npm run verify:release                -> exit=0   (all nine steps)
invariant vs unrepaired snapshot      -> exit=1   (names the 4 shells)
invariant vs repaired copy            -> exit=0

Break-tests — mutate, run, restore, byte-identical restore asserted:

Mutation Result
disable the doctor detector entirely (dist/ rebuilt — that test spawns the built CLI) exit=1 → restored exit=0
drop retireRecallHistory from archive point A (the -other bucket loop) exit=1 → restored exit=0
drop it from archive point B (the legacy readable-only loop) exit=1 → restored exit=0
drop it from the one-shot repair exit=1 → restored exit=0

The middle two were exit=0 before the tests in this PR were added. That is
the finding, not a footnote.

Dashboard strings

The new doctor code has catalogue entries in all 11 locales rather than falling
back to raw English, matching how this project already closed the same gap for
sibling codes.

@kevintseng
kevintseng force-pushed the fix/telemetry-and-lesson-history branch 3 times, most recently from 0ff665e to 4a82a16 Compare September 4, 2026 05:54
…ecall history on empty shells

D13 — `llm_telemetry` was written by every Smart-Mode flow and read by nothing
that could alert anyone. `memesh telemetry` shows it on request, which means a
broken flow needed someone to think to ask. Measured on the maintainer's graph
on 2026-09-02, over a 7-day trailing window: `guard_proposer` 69/69 failed,
`dreamer` 47/47, `auto_tagger` 20/20, `failure_analyzer` 8/8 — all
`bad_request`, all starting 2026-08-28, and `memesh doctor` said nothing.

The window is 7 days rather than the CLI's own 30-day default for a measured
reason: at 30 days, `dreamer`'s stale successes from 2026-08-23 dilute the rate
to "36% succeeded" and hide that it has been fully broken all week.
`transcript_extractor` (96.6% healthy over its lifetime, real background noise)
had no calls inside the window at all, so it stays silent — correctly, not by
luck.

D15 — `splitFusedLessons` moved a fused bucket's observations into their own
entities but left the emptied shell's `recall_hits`/`recall_misses` behind. Of
49 archived shells, 4 carried stale counters (3/61, 1/35, 3/1, 0/1). The
counters are not merely inert: `scripts/audit/measure-signals.mjs` sums them
over ALL entities with no status filter, so the stale number pollutes that
signal forever.

Zeroed, not divided and not copied onto the successors. A hit is recorded
against the bucket id, never against one observation inside it, so dividing
fabricates a fraction nobody earned and copying claims each successor
independently earned the whole bucket's record. The real numbers are preserved
under `metadata.retired_recall`. The default recall path was never affected —
archived shells lose their FTS row and `search()` filters to active — only the
audit aggregate and any `includeArchived: true` read.

The two archive-point mutations below were green until this commit added the
tests that catch them. Both wirings run inside the same `openDatabase`, and the
one-shot repair runs SECOND — on a fresh database it silently rescued anything
the archive points failed to retire, so every existing test passed with both
call sites removed. But the repair is a `runOnceMigration`: after it stamps it
never looks again, so a bucket fused after the upgrade — the ordinary case from
then on — is protected by the archive point and nothing else. The new tests
seed exactly that state (repair already stamped, split still owed) and assert
the stamp did not move, so they cannot quietly stop isolating what they claim
to isolate.

[Verified-By: node scripts/run-tests-isolated.mjs > /tmp/f13.log 2>&1; echo
exit=$? -> exit=0; "Test Files  230 passed | 1 skipped (231)", "Tests  3295
passed | 11 skipped (3306)", no Errors line]
[Verified-By: npm run verify:release > /tmp/v13b.log 2>&1; echo exit=$? ->
exit=0, ending "consumer install has no high-or-worse advisories"]
[Verified-By: invariant vs the unrepaired snapshot -> exit=1 naming the 4
shells; vs the repaired copy -> exit=0. Repair on a copy: global
SUM(recall_hits)/SUM(recall_misses) 97/561 -> 90/463, exactly the 4 shells and
nothing else; 26 successors checked, none touched]
[Verified-By: break-tests, mutate -> run -> restore, byte-identical restore
asserted by reading the file back: disable the doctor detector entirely
(dist/ rebuilt, since that test spawns the built CLI) -> exit=1, restored
exit=0; drop retireRecallHistory from archive point A -> exit=1, restored
exit=0; from archive point B -> exit=1, restored exit=0; from the one-shot
repair -> exit=1, restored exit=0]
`inspectLlmTelemetryHealth` swallows a failed read of `llm_telemetry` and
returns undefined — a database from before the table existed has nothing to
diagnose. Every test in doctor.test.ts gives the stub an empty table, so the
catch was never entered and turning it into a rethrow left all 149 tests
green. This adds `telemetryUnreadable` to the stub and one test that asserts
the failure neither becomes a finding nor leaks into runDoctor's outer catch
as a database error.

[Verified-By: node scripts/run-tests-isolated.mjs tests/core/doctor.test.ts
--maxWorkers=1 -> Tests 150 passed, exit=0]
The graph-repairs.test.ts comment explaining the archive-point/shell-repair
interaction claimed the ordinary case forever after the upgrade is
protected by the archive point alone -- but the current code no longer
creates new -other buckets at all (learn() moved to digest naming in
4.8.2), and the shell repair is a version-1 runOnceMigration that already
stamped, not something still running. What the archive point actually
guards is the NEXT time this split logic's own version bumps past 2, not
an ongoing steady state. Rewritten to say that instead of a claim the code
no longer supports.

Added a real-browser (real HTTP server, real Chromium, all 11 locales)
check that llm-telemetry.silent-failure renders translated in the
dashboard, not as a raw i18n key or a silent English fallback -- something
the existing catalogue tests check at the translation-table level but
can't confirm a real bundle actually renders. Triaged the new script in
verification-audit's baseline as a deliberately eg-invoked QA tool, not a
missing CI gate; pruned one baseline entry that no longer matches current
line numbers.
The rebase onto origin/main (post-#285 merge)'s own conflict resolution
took the wrong side for these two map files; a full rebuild produces the
correct byte content.
@kevintseng
kevintseng force-pushed the fix/telemetry-and-lesson-history branch from 4a82a16 to 39f0637 Compare September 4, 2026 06:21
@kevintseng
kevintseng merged commit b95a616 into main Sep 4, 2026
31 of 35 checks passed
@kevintseng
kevintseng deleted the fix/telemetry-and-lesson-history branch September 5, 2026 13:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant