framework-drift

Stack-aware framework drift monitor for AI engineers. Reads your stack manifest, surfaces only the agent-framework changes that affect your installed dependencies. Weekly summary; HITL approval gate on migrate / refactor recommendations.

Capstone for the AI-Assisted Coding immersive (9BRAINS / Divergence Academy / Helm), May 4 – June 11, 2026.

Status

Pt 1 shipped · Pt 2 multi-agent build complete (all 5 phases, 2026-06-09). The design contract in docs/pt2/ is now built and running, not just specified. 13 atomic tools, 10 MCP endpoints, 8 CLI commands, 273 tests.

Pt 1 — four pipeline stages (Catch → Enrich → Separate → Compound) ship end-to-end against real GitHub Releases, Pinecone, and the Claude API. Live compound run against langchain-ai/langgraph captured 2026-05-08: 5 real-Claude summaries (Haiku 4.5) + 1 migrate plan (Sonnet 4.6) catching the LANGGRAPH_STRICT_MSGPACK security flag. Phoenix trace: 157 spans under one cli.compound root.

Pt 2 — Curator → HITL Gate → Advisor on LangGraph 1.2.4, with run-keyed thoughts/<run-id>/ state that survives a context clear and resumes without re-paying LLM cost (no checkpointer — disk is the state store, per D-02). Live end-to-end run 2026-06-08: 25 Curator-scored releases → HITL gate → Advisor WeeklyDigest (2 advisory items), full curator → hitl_gate → advisor span tree in Phoenix. A 14-case eval harness (framework-drift eval, T1–T14, rule-based + custom Sonnet judges, no DeepEval) logs to Phoenix experiments; the failure-mode gate is MET 3/5 — F9 (gate bypass), F11 (citation-footer schema), F12 (tool-catalog drift) pass; F1 (action-class false-negative) and F3 (hallucinated plan step) fail and are documented as real Curator/Advisor weaknesses, not harness bugs. Pt 1 demo Thu May 14 ✓.

For demo guests / industry reviewers

If you're reviewing this repo for the capstone (or after a demo), the three highest-signal reads in priority order:

1. docs/pt2/stack-justification.md — 1-page summary of every stack choice with rejected alternatives and rubric coverage. Start here.
2. docs/pt2/architecture.md — Curator → HITL Gate → Advisor topology with three Mermaid diagrams (topology · defense-in-depth vs lethal trifecta · crash-and-resume sequence).
3. docs/pt2/lethal-trifecta-audit.md — the security story. §4 adds the integration-reality threat dimension (malformed third-party data · schema drift · retry-storm · idempotency violations) that a typical RAG-only AI engineer wouldn't carry — fifteen years of enterprise-integration discipline applied to agent design.

The 6 files in docs/pt2/ are the design contract the now-shipped Curator → HITL Gate → Advisor system was built to (all 5 phases complete — see Status); architecture.md, governance.md, and product-brief.md were reconciled to the as-built system in Phase 5. Itemized in the sub-table below.

What's in here

File	Purpose
product-brief.md	User, problem, MVP, success metric (1 page).
context.md	Runtime instructions for the framework-drift agent (under 60 lines).
architecture.md	Pipeline diagram, stack picks, feedback loops, failure modes.
signal-decision.md	What counts as signal, what counts as noise, scoring approach + calibrated threshold.
governance.md	Pillar 3 compliance design — stakes×reversibility map, audit trail, hard nos.
framework-drift.toml.example	Stack manifest schema preview.
docs/rubric/	Capstone grading source-of-truth (extracted text).
docs/signal-harvester-capstone-rubric.pdf	Capstone rubric (PDF).
docs/demo-plan.md	Demo strategy for a weekly-cadence system.
docs/build-log.md	Day-by-day build log — decisions, bugs caught, gotchas, source for the retrospective.
docs/tool-contracts.md	CRUD audit per atomic tool — closes "no bundled judgment" rubric line.
docs/traces/	Phoenix trace exports — Day 5 mock-Claude (127 spans) + Day 8 real-Claude (157 spans, 96s wall).
docs/deck/v0.1-may14.md	10-slide Pt 1 demo deck content + speaker notes + Q&A.
RESUME-2026-05-06.md	Pause/resume handoff doc. Read at restart.

Pt 2 design contract (built and shipped — see Status)

File	Purpose
docs/pt2/part2-plan.md	The 3 picks — Multi-Agent Architecture · Eval Harness · Production-Safety primitives — defended against alternatives; 58/100 rubric points directly addressed.
docs/pt2/architecture.md	Curator → HITL Gate → Advisor topology · 3 Mermaid diagrams (topology · defense-in-depth · crash-and-resume) · production-safety primitive specifics (permission tier per agent, LangGraph sandbox allowlist, autonomy ladder, idempotency contract).
docs/pt2/eval-plan.md	T1–T14 evaluators · custom backends · F10 prompt-injection resistance (LLM-as-judge hybrid scoring) · F12/T14 tool-catalog fidelity.
docs/pt2/lethal-trifecta-audit.md	Security story — private data × untrusted content × exfiltration walked against framework-drift's actual surface · §4 integration-reality threats beyond the deck's frame.
docs/pt2/audit-log-commit-discipline.md	Layer-3 immutability mechanism — operationalizes "immutable" via commit-signed git log (three-step write path + verification primitive + honest limits).
docs/pt2/stack-justification.md	1-page demo Q&A talking-points sheet — every stack choice with course-session anchor + rejected alternatives + scope-discipline excluded-with-reason.

Quickstart

# Install (Python 3.11+)
pip install -e .

# Configure your stack
cp framework-drift.toml.example framework-drift.toml
# Edit it — declare which deps + repos you care about. Or fall back to
# auto-detection from requirements.txt / pyproject.toml / package.json /
# mcp.json (Pt 2 work — Pt 1 reads the explicit manifest).

# Required env vars
cp .env.example .env
# Edit .env — fill in PINECONE_API_KEY (required for enrich), and
# ANTHROPIC_API_KEY (required when Compound stage lands). GITHUB_TOKEN
# is optional but raises rate limits 60 → 5000 req/hr.

# One-time: provision Pinecone index + upsert manifest as embeddings
framework-drift enrich-init

# Run the full pipeline against a repo (catch → enrich → separate → compound)
framework-drift compound --repos langchain-ai/langgraph --gate-mode approve-all

# Or run a single stage in isolation:
framework-drift catch-github langchain-ai/langgraph     # fetch releases only
framework-drift enrich --repos langchain-ai/langgraph   # catch + enrich
framework-drift separate --repos langchain-ai/langgraph # through scoring + classification

# Disable Phoenix tracing for offline / CI runs
framework-drift compound --no-tracing --repos langchain-ai/langgraph

Phoenix tracing emits to localhost:6006 by default — start a local Phoenix instance before running, or set PHOENIX_ENDPOINT to a remote collector. Spans land under project framework-drift. Bind Phoenix to loopback (Phoenix defaults to 0.0.0.0 with auth disabled, which exposes prompt content to anyone on your LAN):

PHOENIX_HOST=127.0.0.1 phoenix serve

Phoenix history retention. Phoenix persists trace history to ~/.phoenix/phoenix.db (SQLite). The DB accumulates every trace captured locally — including prompt and response content. Local-only retention is acceptable while Phoenix stays bound to loopback (see above), but machine handoff is the threat-model boundary: if you loan the machine, plan to transfer it, or otherwise want clean state, wipe the DB before handoff:

rm ~/.phoenix/phoenix.db

Documented as residual risk #3 in docs/pt2/lethal-trifecta-audit.md.

Phoenix install discipline — single-source-of-truth per machine. Phoenix uses Alembic to migrate ~/.phoenix/phoenix.db on startup. The DB is stamped with the migration revision of whichever install ran first — if a second, older Phoenix install (e.g., in a separate venv) is invoked later, its Alembic won't recognize the newer stamp and startup fails with:

alembic.script.revision.ResolutionError: No such revision or branch '<hex>'
phoenix.exceptions.PhoenixMigrationError: Phoenix failed to migrate the database to the latest version.

Pick one install per machine and document which. As of 2026-05-26 on Ricco's daily-driver, the canonical install is the Python 3.11 user site:

C:\Users\ricco\AppData\Local\Programs\Python\Python311\Scripts\phoenix.exe  (v14.5.0)

That binary is what phoenix on bare PATH resolves to, and it's the install that owns the local DB stamp. The separate C:\Users\ricco\phoenix-venv\ install (v13.23.0) is dark — do not invoke it against the shared DB; it will fail to migrate.

To rotate the canonical install (e.g., to a Phoenix major upgrade): either upgrade in place via pip install --upgrade arize-phoenix, or wipe the DB first (rm ~/.phoenix/phoenix.db) before switching to the new install. The DB loss is acceptable since trace history is local-only experimental data, not production state.

A coherent end-to-end trace lives at docs/traces/trace-export-2026-05-08-compound-live.md (157 spans, real Claude API, 96.2 s wall clock).

Compound stage HITL gates:

• --gate-mode approve-all — auto-approve every drafted plan (demo mode)
• --gate-mode deny-all — auto-deny every drafted plan (calibration data)
• --gate-mode prompt — synchronous human prompt per drafted plan (production-default)

GitHub Actions weekly cron lives in the Pt 3 backlog — needs production-traffic evidence to calibrate, which Pt 2's design-contract scope deliberately doesn't generate (see docs/pt2/stack-justification.md §"Scope discipline — what we deliberately don't ship in Pt 2").

Build context

Solo capstone, not a production tool. Throwaway code allowed; throwaway thinking is not — the design artifacts above describe a real system, even if implementation lands incrementally over six weeks.

The repo aims for "a stranger can clone and run" per the rubric's Repository Quality criterion. The full Quickstart (framework-drift compound --repos langchain-ai/langgraph --gate-mode approve-all) is verified end-to-end against real GitHub, Pinecone, and Claude API as of 2026-05-08 — see docs/traces/trace-export-2026-05-08-compound-live.md for the captured trace.