Skip to content

feat(runtime): add Context Quality Eval v0 - #303

Merged
DanielJD1216 merged 1 commit into
mainfrom
feat/context-quality-eval-v0-302
Sep 1, 2026
Merged

DanielJD1216 merged 1 commit into
mainfrom
feat/context-quality-eval-v0-302

Conversation

@DanielJD1216

Copy link
Copy Markdown
Owner

Summary

  • add deterministic metadata-only Context Inbox relevance filtering, current-revision selection, and scoped monotonic freshness
  • add the versioned synthetic Context Quality Eval v0 corpus, exact scorecard, stable JSON report, hard thresholds, and nine isolated mutation probes
  • integrate Eval v0 into the existing protected runtime:skeleton-smoke path without changing package.json, dependencies, workflows, migrations, deployment, or published artifacts
  • document corpus structure, metrics, thresholds, report schema, deterministic execution, failure interpretation, and explicit non-goals

Closes #302

What Eval v0 proves

  • exact expected evidence sets, Precision@K, Recall@K, and irrelevant-evidence exclusion
  • exact allowlisted citation, source identity, revision, locator, digest, and freshness bindings
  • current-revision retrieval and monotonic owner/namespace-scoped freshness
  • zero owner leakage, namespace leakage, source-body leakage, candidate creation, or trusted-memory creation
  • exact retry idempotency and fail-closed request, envelope, source-revision, and cross-identity conflicts
  • deterministic rejection of authorization, malformed-state, descriptor, prototype, array, and Proxy boundary cases
  • mutation sensitivity for retrieval, provenance, source-body leakage, forbidden promotion, replay, authorization, owner-scope leakage, malformed boundaries, and threshold weakening

Deterministic result

  • corpus: 1.0.0
  • cases: 34/34
  • expected-set accuracy: 1.00
  • Precision@K: 1.00
  • Recall@K: 1.00
  • citation/provenance fidelity: 1.00
  • all leakage, lifecycle, replay, freshness, adversarial, and unhandled counters: 0
  • mutation probes detected: 9/9
  • focused evaluator regression tests: 19/19
  • repeated canonical JSON SHA-256: a3a657e5538ee0bef6099a7707b3a5cec9a03285da0cbc7814c09de530d81d24

Local verification

  • npm run build
  • node --test examples/runtime-skeleton/context-quality-eval-v0.test.mjs
  • node examples/runtime-skeleton/context-quality-eval-v0.mjs --self-test
  • npm run runtime:skeleton-smoke
  • node examples/runtime-skeleton/context-inbox-smoke.mjs
  • npm run docs:links
  • npm run docs:anchors
  • npm run docs:command-setup
  • npm run package:content-smoke
  • npm run runtime:gate-b-memory-only:scope
  • npm run runtime:gate-b-memory-only:scope:smoke
  • npm run alpha1:story5:security-gate
  • npm run local-runtime:security-gate
  • npm run safety:scan
  • npm run claims:scan
  • npm run readiness:report
  • npm run ci:check

All listed checks passed locally.

Boundaries

Eval v0 is offline and uses only synthetic public-safe fixtures. It makes no model calls, uses no model judge, starts no service, connects no database or live provider, changes no Alpha MCP profile, and creates no candidate or trusted memory. It does not establish downstream model outcome lift, real-data quality, production performance, hosting, deployment, backup/restore readiness, or release status. Operations issue #296 remains separate.

@DanielJD1216
DanielJD1216 merged commit af15605 into main Sep 1, 2026
6 checks passed
@DanielJD1216
DanielJD1216 deleted the feat/context-quality-eval-v0-302 branch September 1, 2026 05:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add deterministic Context Quality Eval v0

1 participant