Measure production memory delivery with auditable paired comparisons

CHANGELOG.md

@@ -6,6 +6,17 @@ changelog, see the public [kimetsu CHANGELOG](../CHANGELOG.md).
 
 ## Bench tool
 
+### Unreleased — paired production MCP evidence
+
+- Persistent MCP query measurements, delivered capsule text and optional final
+  answerability; scoped Windows working-set and peak memory observations.
+- Explicit per-side reranker, cutoff, threads and fact-guard settings with
+  effective configuration checks and binary/fixture/runner fingerprints.
+- Separate positive and negative denominators, auditable failures, stale-query
+  accounting, dated compressed-capsule matching and timeout descendant cleanup.
+- Rust harness validation: 132 tests passed; paired-runner validation: 18 Python
+  tests passed. The linked Kimetsu audit evaluates metadata on every repeat.
+
 ### v0.5 — 2026-06-05 — `kstress` brain stress test
 
 New second binary `kstress` profiles the brain (not agent tasks) at scale —

Cargo.toml

@@ -62,3 +62,6 @@ rusqlite = { version = "0.37", features = ["bundled"] }
 
 # Used by the LongMemEval driver for per-instance temp brain workspaces.
 tempfile = "3"
+
+[target.'cfg(windows)'.dependencies]
+windows-sys = { version = "0.61", features = ["Win32_Foundation", "Win32_System_ProcessStatus", "Win32_System_Threading"] }

README.md

@@ -13,6 +13,21 @@ taking ours on faith.
 
 ## What it measures
 
+### Paired memory evidence (unreleased)
+
+BrainBenchmark now measures persistent production MCP context delivery, retaining
+rankings, delivered capsules, optional `answerability`, latency, serialized byte
+cost and supported Windows process-memory measurements. The paired runner verifies
+effective per-side model, floor and guard settings, alternates repeat order, and
+records binary/fixture fingerprints. Errors and unpaired cases are not successful
+quality observations. Historical binaries without answerability remain supported.
+
+See the public [structured-fact report](https://github.com/RodCor/kimetsu/blob/281c6df/docs/audits/2026-09-07-structured-facts.md)
+for the frozen fixture, commands and all-repeat metadata scoring. It reports
+evidence delivery rather than generated-answer accuracy; it does not replace
+older overall BrainBench scores. This branch requires the companion Kimetsu
+hardening changes in its parent checkout for the path dependencies.
+
 `kbench` runs one or more Terminal-Bench tasks under multiple agent
 configurations and produces a side-by-side comparison:
 

datasets/brainbench/agent-memory-contract.json

@@ -0,0 +1,262 @@
+{
+  "scenarios": [
+    {
+      "id": "exact-code-evidence",
+      "dimension": "retrieval",
+      "tier": "easy",
+      "description": "Exact identifiers, multi-fact recall, and unrelated no-answer queries.",
+      "memories": [
+        {
+          "key": "stderr",
+          "text": "MCP diagnostic logs must go to stderr. Stdout is reserved for JSON-RPC messages."
+        },
+        {
+          "key": "foreign-keys",
+          "text": "Enable SQLite foreign key constraints explicitly on every new connection using PRAGMA foreign_keys = ON."
+        },
+        {
+          "key": "blocking",
+          "text": "Move long synchronous operations into tokio::task::spawn_blocking instead of blocking asynchronous executor threads."
+        },
+        {
+          "key": "test-isolation",
+          "text": "Initialize a Git boundary in each temporary test workspace so repository discovery cannot write to the real project."
+        }
+      ],
+      "queries": [
+        {
+          "query": "Where must MCP diagnostic logs be written?",
+          "relevant": [
+            "stderr"
+          ]
+        },
+        {
+          "query": "How do I enable SQLite foreign key constraints on a new connection?",
+          "relevant": [
+            "foreign-keys"
+          ]
+        },
+        {
+          "query": "How should I handle a long synchronous operation in the Tokio executor?",
+          "relevant": [
+            "blocking"
+          ]
+        },
+        {
+          "query": "What is the weather forecast for tomorrow?",
+          "relevant": []
+        },
+        {
+          "query": "What is the current production database password?",
+          "relevant": []
+        }
+      ]
+    },
+    {
+      "id": "related-but-unanswerable",
+      "dimension": "retrieval",
+      "tier": "hard",
+      "description": "Related vocabulary must not imply that a stored fact answers a different question.",
+      "memories": [
+        {
+          "key": "http-port",
+          "text": "The local development HTTP server listens on port 4317."
+        },
+        {
+          "key": "storage",
+          "text": "The project stores local agent memories in a SQLite database."
+        },
+        {
+          "key": "logs",
+          "text": "Development request logs include the route and elapsed milliseconds."
+        }
+      ],
+      "queries": [
+        {
+          "query": "Which port does the local development HTTP server use?",
+          "relevant": [
+            "http-port"
+          ]
+        },
+        {
+          "query": "What authentication password does the local development HTTP server require?",
+          "relevant": []
+        },
+        {
+          "query": "Which encryption key protects the SQLite database?",
+          "relevant": []
+        },
+        {
+          "query": "What is the production request log retention duration?",
+          "relevant": []
+        }
+      ]
+    },
+    {
+      "id": "cross-language-code",
+      "dimension": "retrieval",
+      "tier": "hard",
+      "description": "Small exploratory Spanish query track against English code memories.",
+      "memories": [
+        {
+          "key": "stderr",
+          "text": "Write MCP diagnostic messages to stderr because stdout carries the JSON-RPC protocol."
+        },
+        {
+          "key": "foreign-keys",
+          "text": "SQLite foreign key enforcement must be enabled separately for every connection with PRAGMA foreign_keys = ON."
+        },
+        {
+          "key": "blocking",
+          "text": "Use tokio::task::spawn_blocking for long synchronous work inside an asynchronous application."
+        }
+      ],
+      "queries": [
+        {
+          "query": "¿Dónde deben escribirse los mensajes de diagnóstico MCP para no romper JSON-RPC?",
+          "relevant": [
+            "stderr"
+          ]
+        },
+        {
+          "query": "¿Cómo se activan las claves foráneas de SQLite en cada conexión?",
+          "relevant": [
+            "foreign-keys"
+          ]
+        },
+        {
+          "query": "¿Cómo ejecuto trabajo síncrono largo sin bloquear Tokio?",
+          "relevant": [
+            "blocking"
+          ]
+        },
+        {
+          "query": "¿Cuál es el precio actual de la electricidad?",
+          "relevant": []
+        }
+      ]
+    },
+    {
+      "id": "multi-fact-retrieval",
+      "dimension": "retrieval",
+      "tier": "medium",
+      "description": "Fraction recall requires both expected facts; finding either is only a hit.",
+      "memories": [
+        {
+          "key": "git-boundary",
+          "text": "For safe temporary memory tests, run git init in the temporary workspace to establish a repository boundary."
+        },
+        {
+          "key": "user-brain",
+          "text": "For isolated memory tests, set KIMETSU_USER_BRAIN=0 to prevent cross-project memory leakage."
+        },
+        {
+          "key": "release",
+          "text": "Release packaging produces a compressed archive containing the executable."
+        }
+      ],
+      "queries": [
+        {
+          "query": "How do temporary memory tests establish a Git boundary and prevent user brain leakage?",
+          "relevant": [
+            "git-boundary",
+            "user-brain"
+          ]
+        },
+        {
+          "query": "Which executable signing certificate is used for releases?",
+          "relevant": []
+        }
+      ]
+    },
+    {
+      "id": "live-temporal-applicability",
+      "dimension": "retrieval",
+      "tier": "hard",
+      "description": "Future starts and expired claims must not appear in current context; these are storage validity fields, not text hints.",
+      "memories": [
+        {
+          "key": "expired",
+          "text": "The Atlas staging listener uses port 4001.",
+          "valid_to": "2020-01-01T00:00:00Z"
+        },
+        {
+          "key": "future",
+          "text": "The Atlas staging listener uses port 4003.",
+          "valid_from": "2099-01-01T00:00:00Z"
+        },
+        {
+          "key": "current",
+          "text": "The Atlas staging listener uses port 4002."
+        }
+      ],
+      "queries": [
+        {
+          "query": "Which port does the Atlas staging listener currently use?",
+          "relevant": [
+            "current"
+          ],
+          "stale": [
+            "expired",
+            "future"
+          ]
+        },
+        {
+          "query": "What authentication password does the Atlas staging listener require?",
+          "relevant": [],
+          "stale": [
+            "expired",
+            "future"
+          ]
+        }
+      ]
+    },
+    {
+      "id": "persistent-mcp-observes-new-writes",
+      "dimension": "workflow",
+      "tier": "medium",
+      "description": "A persistent MCP process must see new claims written by another process, while related unknowns remain unanswered.",
+      "workflow": {
+        "seed": [],
+        "episodes": [
+          {
+            "task": "Which port does the Zephyr local development HTTP server use?",
+            "relevant": [],
+            "record": [
+              {
+                "key": "port",
+                "text": "The Zephyr local development HTTP server listens on port 5243."
+              }
+            ]
+          },
+          {
+            "task": "Which port does the Zephyr local development HTTP server use?",
+            "relevant": [
+              "port"
+            ]
+          },
+          {
+            "task": "What authentication password does the Zephyr HTTP server require?",
+            "relevant": []
+          },
+          {
+            "task": "Where should MCP diagnostic logs be written?",
+            "relevant": [],
+            "record": [
+              {
+                "key": "logs",
+                "text": "MCP diagnostic logs go to stderr; stdout is reserved for JSON-RPC messages."
+              }
+            ]
+          },
+          {
+            "task": "Where should MCP diagnostic logs be written?",
+            "relevant": [
+              "logs"
+            ]
+          }
+        ]
+      }
+    }
+  ]
+}

scripts/PAIRED_BRAINBENCH.md

@@ -0,0 +1,36 @@
+# Compare two Kimetsu builds without a reader model
+
+Build the current `kbench` once and use that identical harness for both binaries:
+
+```powershell
+cargo build --release --bin kbench --locked --offline
+python scripts/compare_brainbench.py --kbench target/release/kbench.exe --baseline path/to/baseline/kimetsu.exe --candidate path/to/candidate/kimetsu.exe --dataset datasets/brainbench/agent-memory-contract.json --budget-tokens 512 --repeats 3 --out local/paired-memory
+```
+
+Use an existing shared `FASTEMBED_CACHE_DIR` to avoid model downloads. The binaries must support embeddings. The runner disables the user brain and permits only non-generative dimensions. Every scenario uses a temporary isolated project. It never uses your live brain.
+
+The runner alternates baseline/candidate order, fixes one scenario worker, retains every report and stderr log, and records SHA-256 fingerprints for binaries, the harness, the dataset and referenced fixture files. It does not force identical model defaults: a binary-default comparison intentionally includes changes to those defaults. Pin relevant model/environment settings for an algorithm-only experiment; recorded overrides make that distinction reviewable.
+
+Each scenario is paired by dimension and ID. Repeats are averaged within a scenario, not counted as additional independent cases. A scenario with a skipped or errored observation on either side is reported as unpaired and excluded from quality deltas; execution failures are not scores. The exploratory confidence interval bootstraps scenario IDs. Related scenarios are still correlated: a release claim requires held-out task/repository families and real task-success measurements. Changed/duplicate scenario identities are errors.
+
+`comparison.json` is written as the run progresses. A command failure, timeout, invalid JSON/report response, or final identity-validation failure leaves it with `status: "incomplete"`, structured failure evidence, and all completed run records; the process exits nonzero and does not write a completed Markdown comparison.
+
+The whole-run timeout terminates descendants before reaping `kbench` (Windows `taskkill /T`, Unix process group). This prevents a hung MCP inference child from surviving a timed-out comparison and competing with the next run. The deadline covers seeding and queries together; it is not a per-query latency cutoff.
+
+BrainBench's headline now weights measured dimensions equally; the old scenario-weighted average remains a diagnostic. No-answer queries score abstention, not the vacuous recall of an empty relevant set. Positive recall and negative injection rates use separate denominators, and stale correctness is reported as unavailable when no stale cases exist. Unmatched or ambiguous returned capsules retain their rank and count as injected material.
+
+Retrieval and workflow scenarios query the production `kimetsu_brain_context` tool through a persistent stdio MCP process. Every temporary project sets `broker.warm_start=false`, and requests set `include_ambient=false`: gold labels cover returned capsules, not unsolicited digest/profile/resume text. Measurements therefore cover production MCP retrieval under this controlled session configuration; they exclude warm-start construction/delivery and are not default whole-first-turn costs. Continuity is tested separately with warm starts enabled. Workflow writes happen through a separate process while MCP stays alive, exercising index freshness. The render-contract dimension retains its explicit CLI rendering check. Invalid MCP responses fail the scenario instead of masquerading as abstention. Current-context retrieval scores become zero if any explicitly stale gold item is delivered in the top four, even below the correct answer; the older ordering-only resolution metric remains diagnostic.
+
+Query observations retain hit@4, fraction recall@4, MRR, negative injection and stale injection separately. Repeats are averaged within the same query for quality summaries. Timing reports distinguish the first query in each process from subsequent queries. First-query timing is not a disk-cache-cold measurement; process startup is recorded separately. Subsequent p50/p95 are descriptive pooled measurements, not independent trials or a confidence interval. Full-run timings also include seeding and process startup.
+
+Response sizes include UTF-8 model text, the serialized MCP result, and the full JSON-RPC response line. `reported_used_tokens` is retained as a diagnostic: old heuristic estimates and new conservative byte bounds are not directly comparable. These byte measurements are not provider tokenizer counts or billed-token measurements.
+
+On Windows, each query also samples the MCP child's current and lifetime peak working set via `GetProcessMemoryInfo`, after the latency clock stops. The comparison reports the largest observed process peak; it excludes `kbench`, separate ingestion processes, other agents and system-wide model caches. A working set includes resident shared pages, so it is not private allocation or a sum of whole-agent RAM. Unsupported platforms or failed samples report unavailable. [Windows counter definitions](https://learn.microsoft.com/en-us/windows/win32/api/psapi/ns-psapi-process_memory_counters).
+
+For runtime experiments use `--baseline-threads 0 --candidate-threads 4` with the same binary on both sides (`0` removes `KIMETSU_INTRA_THREADS`; omission inherits it). `KBENCH_RERANKER` sets `embedder.reranker` only inside temporary benchmark projects and is recorded; the binary must actually honor that setting on MCP for a model comparison to be valid. Keep other settings fixed and avoid concurrent builds or other inference during timing runs.
+
+For a paired model comparison, use the same binary on both sides with `--baseline-reranker ms-marco-tinybert-l-2-v2 --candidate-reranker ms-marco-minilm-l-4-v2`. Side overrides take precedence over `KBENCH_RERANKER`; `off` explicitly disables reranking. The harness first sets `retrieval.level=custom` in each temporary project so the new-project `deep` preset cannot overwrite the requested model. Run order still alternates, and the comparison fingerprints the Python runner as well as `kbench` and the binaries. Model comparisons require compatible locally cached models and a binary that honors its MCP reranker configuration.
+
+The small checked-in fixture is a regression and exploratory language track, not a comprehensive held-out benchmark. It includes temporal validity fields and persistent-process write visibility. Final host token consumption and complete-agent success require the corresponding serving and agent evaluations.
+
+Use `--baseline-rerank-floor` and `--candidate-rerank-floor` for explicit final cross-encoder admission thresholds in [0, 1]. Omission inherits `KBENCH_RERANK_FLOOR`, or the binary default when unset. The harness writes `broker.rerank_min_score` only in temporary projects. Older binaries need this option omitted. Thresholds are model-specific scores, not calibrated probabilities; cosine gates remain active. Each side records its effective override alongside the model.