From 4594e0419d4993a3f15d21c0f9d8243facd1562e Mon Sep 17 00:00:00 2001 From: Luca Ostinelli Date: Fri, 16 Oct 2026 10:52:43 -0500 Subject: [PATCH 1/3] docs: add design specification and initial ADRs Record the agreed design for the Agentic-RAG repository: - the domain-agnostic engine extracted from BitPolito Academy; - the AST-aware Code-RAG research track; - milestones with exit criteria; - engineering conventions. ADR-0001 to ADR-0004 capture these foundational decisions: - ports and adapters; - a uv workspace monorepo; - QVAC as an optional adapter; - copy-then-migrate from Academy. --- docs/adr/0001-ports-and-adapters.md | 40 ++ docs/adr/0002-uv-workspace-monorepo.md | 36 ++ docs/adr/0003-qvac-as-optional-adapter.md | 28 ++ .../0004-copy-then-migrate-from-academy.md | 23 ++ docs/adr/README.md | 14 + docs/specs/2026-10-16-agentic-rag-design.md | 361 ++++++++++++++++++ 6 files changed, 502 insertions(+) create mode 100644 docs/adr/0001-ports-and-adapters.md create mode 100644 docs/adr/0002-uv-workspace-monorepo.md create mode 100644 docs/adr/0003-qvac-as-optional-adapter.md create mode 100644 docs/adr/0004-copy-then-migrate-from-academy.md create mode 100644 docs/adr/README.md create mode 100644 docs/specs/2026-10-16-agentic-rag-design.md diff --git a/docs/adr/0001-ports-and-adapters.md b/docs/adr/0001-ports-and-adapters.md new file mode 100644 index 0000000..aeaa1af --- /dev/null +++ b/docs/adr/0001-ports-and-adapters.md @@ -0,0 +1,40 @@ +# 0001. Hexagonal architecture with ports and adapters + +- Status: Proposed +- Date: 2026-10-16 + +## Context + +The Academy RAG code has no interfaces. Embeddings, vector search, the LLM, parent storage and lexical search are all hard-wired: + +- to the QVAC HTTP API; +- to SQLAlchemy models; +- to files under `QVAC_INGEST_DIR`. + +Configuration is read at import time. Models and clients are module-level singletons. + +The engine has to serve two consumers with different backends: + +- Academy, which uses QVAC; +- the Code-RAG research track, which needs GPU-local models and code embeddings. + +## Decision + +The engine is organised as ports and adapters. + +- `agentic_rag.ports` defines `typing.Protocol` interfaces: + - `Embedder`, `VectorStore`, `LexicalIndex`, `ParentStore` + - `Reranker`, `LLM`, `Cache`, `TokenCounter` +- Core packages (`types`, `ingest`, `retrieval`, `evidence`, `generation`, `agent`, `eval`) depend only on ports. +- Every adapter lives under `agentic_rag.adapters` and sits behind an optional extra. +- Dependencies are injected through constructors. +- Configuration is a single `EngineConfig` object built with pydantic-settings. +- Every port has an in-memory fake in `adapters/memory`. +- import-linter enforces the dependency rules in CI. + +## Consequences + +- Backends can be swapped without touching pipeline code. +- Unit tests and the eval smoke test run without models or network access. +- More upfront design. Port changes need an ADR, which slows changes to the interfaces. +- Academy code has to be refactored while it is being ported. Lifting it as-is is not possible. diff --git a/docs/adr/0002-uv-workspace-monorepo.md b/docs/adr/0002-uv-workspace-monorepo.md new file mode 100644 index 0000000..3920521 --- /dev/null +++ b/docs/adr/0002-uv-workspace-monorepo.md @@ -0,0 +1,36 @@ +# 0002. uv workspace monorepo for engine, code-rag and research + +- Status: Proposed +- Date: 2026-10-16 + +## Context + +The repository hosts three things that evolve together: + +- a reusable library; +- an AST-aware extension of that library, which brings heavy dependencies (tree-sitter grammars, GPU stacks); +- research code (benchmark, datasets, experiments). + +We want a single CI and a single lockfile. Code-RAG dependencies must not leak into the engine install. + +## Decision + +The repository is a uv workspace, using Python 3.11. + +| Package | Distribution | Import name | +|---|---|---| +| `packages/engine` | `agentic-rag` | `agentic_rag` | +| `packages/code-rag` | `agentic-rag-code` | `agentic_rag_code` | + +- `agentic-rag-code` depends on `agentic-rag`. +- `research/` holds runners and experiment configs that depend on both packages. It is not a published package. +- `services/qvac-worker` is a Node 22 project with its own `package.json`. +- Tooling: ruff (lint and format), mypy `--strict` on `packages/`, pytest, and import-linter. + +## Consequences + +- Atomic changes can span the engine and code-rag. +- There is one lockfile, so environments are reproducible. +- Each package keeps its own dependency set and extras. +- CI has to run per-package jobs and a Node job. +- Releasing the engine needs package-scoped tags, for example `engine-v0.1.0`. diff --git a/docs/adr/0003-qvac-as-optional-adapter.md b/docs/adr/0003-qvac-as-optional-adapter.md new file mode 100644 index 0000000..d412424 --- /dev/null +++ b/docs/adr/0003-qvac-as-optional-adapter.md @@ -0,0 +1,28 @@ +# 0003. QVAC as an optional adapter, not the engine runtime + +- Status: Proposed +- Date: 2026-10-16 + +## Context + +In Academy, the QVAC SDK (Node 22, llama.cpp, HyperDB) provides embeddings (GTE-Large), dense indexing and LLM inference (Qwen3-4B) behind an HTTP service. Its limits for this project: + +- Embedding models cannot be chosen from Python. +- The service does not use the GPU. +- It lacks index deletion. +- It has had runtime stability issues: native runtime crashes, and long model downloads in CI. + +The AST research needs code-specific embeddings and GPU-local models on an RTX 5080. Academy has to keep working on QVAC. + +## Decision + +- QVAC is one adapter set, `agentic_rag.adapters.qvac` behind the `[qvac]` extra. It implements `Embedder`, `VectorStore` and `LLM` over the documented HTTP contract. +- The Node worker is ported to `services/qvac-worker`. It gets a `DELETE` route and a versioned contract under `services/qvac-worker/contract/`. +- A `local` adapter set is the default for research: sentence-transformers embeddings, a llama.cpp LLM on the GPU, and a vector store chosen by ADR in milestone E3. +- CI does not depend on QVAC. Tests use in-memory fakes, plus a separate worker test job. + +## Consequences + +- Academy can adopt the engine with no behaviour change, through the QVAC adapter. +- Research is not limited by what the QVAC SDK supports. +- Two backend stacks have to be kept at parity. The eval suite has to pass on both (exit criterion of E3). diff --git a/docs/adr/0004-copy-then-migrate-from-academy.md b/docs/adr/0004-copy-then-migrate-from-academy.md new file mode 100644 index 0000000..91ddc32 --- /dev/null +++ b/docs/adr/0004-copy-then-migrate-from-academy.md @@ -0,0 +1,23 @@ +# 0004. Copy from Academy now, migrate Academy later + +- Status: Proposed +- Date: 2026-10-16 + +## Context + +The engine's API will change a lot while it is being extracted. Making Academy depend on it from day one would require coordinated pull requests in two repositories for every interface change. + +## Decision + +- Generic code and its tests are copied from `BitPolito/bitcoin-academy` into this repository and refactored here. +- Every porting pull request records the Academy commit it was copied from, in the form `Source: bitcoin-academy@ `. +- Academy is not modified until the engine reaches a tagged release (milestone E5). +- Milestone E6 then migrates Academy to `agentic_rag[qvac]` and removes the duplicated code. +- Fixes made in Academy in the meantime are ported deliberately. A `source/academy-sync` issue tracks them. + +## Consequences + +- The engine can iterate freely without breaking Academy. +- There are temporarily two copies of the RAG code. +- Drift has to be tracked until E6. +- E6 has to re-baseline Academy's evaluation to prove parity. diff --git a/docs/adr/README.md b/docs/adr/README.md new file mode 100644 index 0000000..abfa77d --- /dev/null +++ b/docs/adr/README.md @@ -0,0 +1,14 @@ +# Architecture Decision Records + +Each ADR records one significant decision: its context, the decision itself, and the consequences. + +- **File name:** `NNNN-kebab-title.md`. +- **Status:** one of Proposed, Accepted, Superseded by NNNN, or Deprecated. +- **When an ADR is required:** any change to a port, a public contract (for example `EvidencePack` or the QVAC HTTP contract), the repository layout, or the core tooling. + +| ADR | Title | Status | +|---|---|---| +| [0001](0001-ports-and-adapters.md) | Hexagonal architecture with ports and adapters | Proposed | +| [0002](0002-uv-workspace-monorepo.md) | uv workspace monorepo for engine, code-rag and research | Proposed | +| [0003](0003-qvac-as-optional-adapter.md) | QVAC as an optional adapter, not the engine runtime | Proposed | +| [0004](0004-copy-then-migrate-from-academy.md) | Copy from Academy now, migrate Academy later | Proposed | diff --git a/docs/specs/2026-10-16-agentic-rag-design.md b/docs/specs/2026-10-16-agentic-rag-design.md new file mode 100644 index 0000000..fbae208 --- /dev/null +++ b/docs/specs/2026-10-16-agentic-rag-design.md @@ -0,0 +1,361 @@ +# Agentic-RAG: Design Specification + +- **Status:** Draft, under review +- **Date:** 2026-10-16 +- **Owners:** @BitPolito/ai-ml +- **Related:** ADR-0001 to ADR-0004 in `docs/adr/` + +## 1. Context + +BitPolito Academy (`BitPolito/bitcoin-academy`) is a local-first study companion. Its core value is the RAG pipeline in `services/ai`. That pipeline does hierarchical ingestion, hybrid retrieval, reranking, parent expansion and an evidence pack with mandatory citations, and today it is coupled to the Bitcoin course domain. The 2026/27 strategy has four steps: + +1. Consolidate the backend ("Agent Harness") and freeze UX work. +2. Use the local hardware for R&D on small language models only. No frontier models. +3. Extract a domain-agnostic engine: courses become namespaces, retrieval becomes a library, and prompts become configuration. +4. Build a second project on top of that engine. + +This repository hosts both the extracted engine and the second project: an **AST-aware agentic RAG for coding tasks**, delivered as an open benchmark and technical report. + +### 1.1 Findings from the Academy codebase (2026-10-15) + +- **Embeddings, the dense index and the LLM all run inside a Node service**, `workers/qvac-service`. + - It uses the QVAC SDK 0.9 with GTE-Large FP16, HyperDB and Qwen3-4B Q4_K_M on llama.cpp. All LLM calls are serialized behind a FIFO lock. + - The Python side reaches it over HTTP. There is no FAISS and no Ollama. +- **The Python side runs everything around the dense index:** + - BM25, stored as a pickle and rebuilt on every ingest. + - Weighted fusion, 0.6 dense + 0.4 lexical. + - A flashrank reranker, MMR, and parent expansion from SQL. + - The `EvidencePack` schema. +- **The generic core is about 2.3k–2.6k LOC of Python**, plus about 3k–4k LOC of portable unit tests. +- **Blocking issues for extraction:** + - The code has no interfaces: no Protocols or ABCs. + - Config is read from the environment at import time, and models and clients are module-level singletons. + - `pipeline.run` mixes ingestion with the application's document state machine. + - Domain vocabulary sits in generic code: Bitcoin synonyms, book-specific cleaning regexes, and `action`/`past_exam` fields in the evidence pack. + - The index lifecycle is incomplete: there is no delete route in QVAC and no BM25 pruning. +- **Current evaluation is weak.** + - A query passes when the top fused score is 0.35 or higher. + - There is no recall@k, MRR or nDCG. + - Page metadata is unreliable on markdown sources. + +## 2. Goals and non-goals + +### Goals + +- **G1.** A domain-agnostic Python RAG engine (`agentic_rag`) with explicit ports and swappable adapters. Academy must be able to adopt it without loss of retrieval quality. +- **G2.** An AST-aware code retrieval extension (`agentic_rag_code`) for Python and TypeScript repositories. +- **G3.** A reproducible open benchmark that relates code-retrieval quality to agent task completion on local small models: + - Recall@K, Precision@K, MRR and nDCG. + - Task Completion Rate (TCR). + - Latency and VRAM. +- **G4.** A public technical report comparing naive text RAG with AST-aware RAG on the lab workstation (RTX 5080, 16 GB VRAM). +- **G5.** A repository that a student team can contribute to in parallel. That means self-contained issues, clear contracts, and CI that enforces the architecture. + +### Non-goals + +- Frontier or commercial hosted models. Everything runs locally. +- UI, gamification, authentication, or course and lesson management. These stay in Academy. +- A conference submission. An arXiv preprint is an optional stretch goal. +- Languages other than Python and TypeScript for AST parsing in this phase. + +## 3. Approach + +**Shared foundation, then two parallel tracks.** A short foundation phase fixes the contracts: types, ports, config and the evaluation harness. After that, two tracks proceed in parallel against the same interfaces: + +- the **Engine** track, which extracts from Academy; +- the **Research** track, which builds Code-RAG. + +Interface changes require an ADR. + +Rejected alternatives: + +- **Engine-first, then research.** This serializes the team and leaves the research contributors idle. +- **Research-first prototype.** This produces a second, divergent copy of the engine. + +## 4. Repository structure + +The repository is a uv workspace with one lockfile and Python 3.11. + +``` +Agentic-RAG/ +├── packages/ +│ ├── engine/ # distribution: agentic-rag import: agentic_rag +│ └── code-rag/ # distribution: agentic-rag-code import: agentic_rag_code +├── services/ +│ └── qvac-worker/ # Node 22 QVAC service, ported from Academy +├── research/ +│ ├── benchmark/ # runners and metrics (IR metrics, TCR, latency/VRAM) +│ ├── datasets/ # build scripts and dataset cards; data is fetched, not committed +│ ├── experiments/ # one folder per experiment: config.yaml + results/*.json +│ └── report/ # technical report sources +├── docs/ +│ ├── architecture.md +│ ├── adr/ # NNNN-title.md +│ └── specs/ +├── .github/ # workflows, issue/PR templates, CODEOWNERS, dependabot +├── AGENTS.md CONTRIBUTING.md README.md LICENSE (MIT) +└── pyproject.toml uv.lock +``` + +## 5. Engine architecture (`agentic_rag`) + +The engine uses a hexagonal (ports and adapters) design. + +``` +agentic_rag/ +├── types/ # Document, Block, Chunk, ParentChunk, CitationAnchor, +│ # EvidenceChunk, EvidencePack, Namespace, IngestResult +├── ports/ # typing.Protocol definitions (no implementations) +├── config/ # EngineConfig (pydantic-settings), no import-time env reads +├── ingest/ # parsers/ (pdf, pptx, docx; registry by extension), clean/ +│ # (boilerplate detection + pluggable rules), chunking/ +│ # (parent/child, injectable TokenCounter), sections/, pipeline.py +├── retrieval/ # HybridRetriever (weighted + RRF fusion), rerank, MMR, +│ # parent expansion, query transforms (HyDE, rewrite), compression +├── evidence/ # EvidencePackBuilder (dedup, ordering, token budget) +├── generation/ # prompt registry (versioned), context formatting ([ref_N]), +│ # citation parsing, strict mode, no-evidence gate +├── agent/ # declarative task graph: system prompt, tools, retrieval policy +├── eval/ # golden-set format, Recall@K, Precision@K, MRR, nDCG, runner +└── adapters/ # optional extras, each importable only if its extra is installed + ├── qvac/ # [qvac] Embedder + VectorStore + LLM over the QVAC HTTP contract + ├── local/ # [local] sentence-transformers embeddings, llama.cpp LLM (GPU) + ├── vector/ # [faiss] or [lancedb], chosen by ADR in milestone E3 + ├── lexical/ # BM25 with JSON persistence (no pickle), injectable tokenizer/synonyms + ├── sql/ # [sql] ParentStore on SQLAlchemy + ├── cache/ # [redis] semantic cache + └── memory/ # in-memory fakes for every port (used by tests and eval smoke) +``` + +### 5.1 Ports + +The signatures below are indicative. They are finalized in milestone M1. + +| Port | Responsibility | +|---|---| +| `Embedder` | `embed(texts) -> vectors`, `dimension`, `max_tokens` | +| `VectorStore` | `upsert(namespace, chunks, vectors)`, `search(namespace, vector, k, filters)`, `delete(namespace, doc_id)`, `drop(namespace)` | +| `LexicalIndex` | `add`, `search(namespace, query, k)`, `delete(namespace, doc_id)` | +| `ParentStore` | `put_many(parents)`, `get_many(ids)`, `delete(doc_id)` | +| `Reranker` | `rerank(query, candidates, k)` | +| `LLM` | `generate(messages, **opts)`, `stream(...)`, `generate_json(messages, schema)` | +| `Cache` | `get(namespace, query)`, `put(namespace, query, value)` | +| `TokenCounter` | `count(text)`, aligned with the active embedder | + +### 5.2 Rules + +- `types` and `ports` import nothing from `adapters`. +- `agentic_rag` never imports `agentic_rag_code`. +- These rules are enforced by import-linter in CI. +- Every component receives its dependencies through its constructor. No module-level singletons. +- Domain specifics are passed in as configuration. Examples: synonym maps, cleaning rules, prompt sets, persona strings. +- `course_id` becomes `namespace` everywhere. +- The `EvidencePack` is a public, versioned contract pinned by a contract test. + - `action` becomes an optional `task` field. + - `doc_name` holds the document name. + - The citation label is a separate field. +- **Strict mode** rejects or flags an answer that has no valid `[ref_N]` citation. It does not fall back to "all chunks cited". + +### 5.3 Data flow + +**Ingestion** + +``` +file -> Parser -> Document(blocks) -> Cleaner -> Chunker -> (parents, children) + -> ParentStore.put_many(parents) + -> Embedder.embed(children) -> VectorStore.upsert + -> LexicalIndex.add(children) + -> IngestResult(stats, section_tree, warnings) +``` + +**Query** + +``` +query -> QueryTransform* -> {VectorStore.search, LexicalIndex.search} -> Fusion + -> Reranker -> MMR -> ParentStore expansion -> EvidencePackBuilder + -> (optional) LLM with strict citation policy -> Answer(evidence, citations) +``` + +Progress and status are reported through callbacks or events. The application maps them to its own state machine. + +### 5.4 Error handling + +- Adapters raise typed errors from `agentic_rag.errors`: + - `BackendUnavailable` + - `LlmDisabled` + - `StructuredGenerationError` + - `IndexNotFound` +- Pipelines degrade explicitly: + - If dense search is unavailable, lexical search runs alone and a warning is attached to the result. + - There is no silent fallback. +- User-facing strings never live in the engine. + +### 5.5 QVAC worker + +The Node service is ported unchanged and then completed. The work covers: + +- a `DELETE` route for documents and workspaces; +- a documented HTTP contract, an OpenAPI or JSON schema under `services/qvac-worker/contract/`; +- indexing of every chunk type. + +The `qvac` adapter is the only Python code that knows this contract. + +## 6. Code-RAG architecture (`agentic_rag_code`) + +- **Parsing.** + - Tree-sitter grammars for Python and TypeScript. + - It extracts modules, classes, functions and methods, with their signatures, docstrings and imports. + - It also extracts call sites and references. +- **Symbol graph.** + - Nodes are symbols and files. + - Edges are `defines`, `calls`, `imports`, `inherits` and `references`. + - The graph is persisted behind a `CodeGraphStore` port. The first adapter is SQLite or networkx, decided by ADR in R1. +- **Chunking.** One chunk per symbol, with its enclosing scope as the parent. Long bodies are split at statement boundaries. +- **Retrieval.** + - `CodeRetriever` combines semantic search (code embeddings via the engine's `Embedder`), lexical search on identifiers, and graph expansion (callers, callees and definitions to depth *d*). + - It returns a standard `EvidencePack` whose anchors carry file, symbol and line range. +- **Agent.** + - An orchestrator-workers pattern built on `agentic_rag.agent`. + - A planner reads the issue, a retrieval worker queries the code index, and a code-generation worker uses a local quantized code model and produces a patch. + - Patches are run in a sandbox, a container with the repository's tests. + +## 7. Evaluation and benchmark + +- **Engine eval** (`agentic_rag.eval`) + - Golden sets list `query`, `relevant_ids` (chunk or section ids), and optionally `ground_truth` and `keywords`. + - Metrics are Recall@K, Precision@K, MRR and nDCG@K, plus latency. + - Baselines are committed as metrics-only JSON. + - The first golden set is ported from Academy (Mastering Bitcoin 3e, CC BY-SA 4.0), with `relevant_ids` re-annotated. +- **Code benchmark** (`research/benchmark`) + - **Tasks** come from real GitHub issues in permissively licensed Python and TypeScript repositories, using SWE-bench-style selection. A task qualifies only if it has a fix commit and a failing-then-passing test. + - **Annotations** list the gold files, symbols and AST paths touched by the fix. + - **Retrieval metrics** are Recall@K and Precision@K over symbols and files. + - **Agent metric** is TCR: the share of tasks whose hidden tests pass after the patch. + - **System metrics** are latency and peak VRAM per stage on the RTX 5080. + - **Experiment matrix** covers {naive text RAG, AST-aware RAG, AST+graph} × {2–3 local code models} × K. +- **Reproducibility.** + - Every experiment is a `config.yaml` that the runner executes. + - Results carry a git SHA, model hashes and hardware info. + - Datasets are rebuilt from scripts. Only the dataset cards and task manifests are committed. + +## 8. Milestones + +Milestones carry no dates. Each one closes when its exit criteria are met. + +| ID | Milestone | Exit criteria | +|---|---|---| +| M0 | Foundation | Workspace scaffold; CI with lint, typecheck, test and quality-gate green on `main`; protected `main`; AGENTS.md, CONTRIBUTING, templates, labels and board in place; ADR-0001–0004 accepted | +| M1 | Core contracts | `types`, `ports`, `EngineConfig` and in-memory fakes; an end-to-end toy pipeline passes on fakes; EvidencePack contract test | +| M2 | Evaluation harness | IR metrics and runner; golden-set format; Academy golden set ported; reproducible baseline report | +| E1 | Ingestion extraction | Parsers, cleaning, chunking, sections and `IngestPipeline`; chunk-count parity with Academy on the open corpus within a stated tolerance; ported tests green | +| E2 | Retrieval extraction | Hybrid retrieval, rerank, MMR, parent expansion, EvidencePackBuilder, strict mode; known Academy defects fixed; metrics ≥ Academy baseline | +| E3 | Adapters | `qvac` adapter plus the completed worker; `local` GPU adapter; vector store chosen by ADR; JSON BM25; SQL ParentStore; Redis cache; eval passes on both the QVAC and local stacks | +| E4 | Agent runtime | Versioned prompt registry; declarative task graph; LLM concurrency control; Academy's 8 study actions expressible as configuration | +| E5 | Engine v0.x release | API docs, changelog, semantic versioning; installable from a git tag | +| E6 | Academy migration | Academy depends on `agentic_rag[qvac]`, duplicated code removed, Academy CI green, Academy eval unchanged or better (cross-repo) | +| R1 | Code ingestion | Python and TypeScript tree-sitter parsing, symbol graph, AST chunker; a mid-size repository indexed and queryable | +| R2 | Code retrieval | CodeRetriever (semantic + lexical + graph); naive vs AST-aware comparison on a small hand-annotated set | +| R3 | Dataset | Reproducible task-building pipeline; dataset card and license checks; v0.1 with a stated task count | +| R4 | Agent and TCR | Orchestrator-workers agent; sandboxed test execution; TCR runner end to end on the dataset | +| R5 | Benchmark and report | Full experiment matrix run; latency and VRAM profile; public technical report reproducible from `research/experiments/` | + +**Dependencies:** + +- M0 → M1 → M2, then E1–E5 and R1–R5 in parallel. +- R2 needs E2. +- R4 needs E3 (`local`) and E4. +- E6 needs E5. + +## 9. Engineering conventions + +### Git and pull requests + +- `main` is protected. +- Work happens on short-lived branches named `/-`. +- Types: feat, fix, refactor, docs, test, chore, perf, research. +- To merge a branch: + 1. Open a pull request against `main` with `Closes #n`. + 2. Wait for a green `quality-gate` and one approval. + 3. Merge with a merge commit (`gh pr merge --merge --delete-branch`). + 4. The branch is deleted. +- Soft limit of about 400 changed lines per PR. Open drafts early. + +### Commits + +- Conventional Commits: `type(scope): summary`. +- Scopes: engine, ingest, retrieval, evidence, generation, agent, adapters, eval, code-rag, dataset, bench, qvac, ci, docs. + +### Quality + +- ruff for lint and format. +- mypy `--strict` on `packages/`. +- pytest with a coverage ratchet starting at 80%. +- import-linter contracts. +- `node --test` for the worker. +- `eval-smoke` on in-memory fakes. + +### Change control + +- Any change that touches retrieval includes a before/after eval summary in the PR. +- Any port or contract change needs an ADR. + +### Labels + +- `type/*`: feature, bug, refactor, docs, test, chore, research. +- `area/*`: ingest, retrieval, adapters, eval, agent, code-rag, dataset, benchmark, qvac, infra. +- `track/*`: foundation, engine, research. +- `priority/*`: p0, p1, p2. +- `size/*`: S, M, L. +- `status/*`: blocked, needs-design. +- Also `good first issue` and `agent-ready`. + +### Board + +GitHub Projects v2. + +- **Fields:** + - Status: Backlog → Ready → In progress → In review → Done. + - Track, Area, Priority, Size, Milestone. +- **Views:** by status, by milestone, and "Ready and unassigned". + +### Issues + +Every issue follows this template: + +- Context +- Goal +- Scope and out of scope +- Files and interfaces +- Acceptance criteria (checklist) +- How to verify (exact commands) +- Dependencies (`Blocked by #n`) +- Size + +**Definition of Ready:** the acceptance criteria are testable, the dependencies are closed, and the size is L or smaller. Anything larger becomes an epic with sub-issues. + +`agent-ready` marks issues precise enough to hand to a coding agent. A human reviews the resulting PR. + +### Ownership + +- `@BitPolito/ai-ml` owns the repository through CODEOWNERS. +- The lead maintainer (@lucaosti) approves ADRs and milestone closures. + +### Cadence + +- A weekly async update on the board covers done, next and blocked. +- A milestone review happens when the exit criteria are met. + +### Language + +All code, documentation, issues, milestones and commits are written in English. + +## 10. Risks + +| Risk | Mitigation | +|---|---| +| Ports designed wrong before both tracks use them | M1 includes a toy pipeline on fakes plus a Code-RAG spike against the ports; changes go through ADRs | +| Parity with Academy drifts while Academy keeps evolving | Pin the Academy commit used as the source; port fixes deliberately; E6 re-baselines | +| QVAC SDK instability (native runtime, model downloads) | The QVAC adapter is optional; the `local` stack is the default for research; CI uses fakes | +| Dataset licensing and contamination | Allow-list of licenses; record repository commit and issue date; prefer issues after the models' training cutoffs | +| 16 GB VRAM limits model choice | Quantized 7–8B code models; one model resident per stage; measure and report VRAM | +| Student availability is bursty | Small, self-contained issues; Definition of Ready; any contributor can pick up any `Ready` issue | From 6c2e722bf3813e56ef4eae78e8749812b60e65b0 Mon Sep 17 00:00:00 2001 From: Luca Ostinelli Date: Fri, 16 Oct 2026 10:58:32 -0500 Subject: [PATCH 2/3] docs(spec): move golden-set re-annotation to E1 and add epic label --- docs/specs/2026-10-16-agentic-rag-design.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/specs/2026-10-16-agentic-rag-design.md b/docs/specs/2026-10-16-agentic-rag-design.md index fbae208..990f6d8 100644 --- a/docs/specs/2026-10-16-agentic-rag-design.md +++ b/docs/specs/2026-10-16-agentic-rag-design.md @@ -246,8 +246,8 @@ Milestones carry no dates. Each one closes when its exit criteria are met. |---|---|---| | M0 | Foundation | Workspace scaffold; CI with lint, typecheck, test and quality-gate green on `main`; protected `main`; AGENTS.md, CONTRIBUTING, templates, labels and board in place; ADR-0001–0004 accepted | | M1 | Core contracts | `types`, `ports`, `EngineConfig` and in-memory fakes; an end-to-end toy pipeline passes on fakes; EvidencePack contract test | -| M2 | Evaluation harness | IR metrics and runner; golden-set format; Academy golden set ported; reproducible baseline report | -| E1 | Ingestion extraction | Parsers, cleaning, chunking, sections and `IngestPipeline`; chunk-count parity with Academy on the open corpus within a stated tolerance; ported tests green | +| M2 | Evaluation harness | IR metrics and runner; golden-set format; reproducible baseline report on the toy set | +| E1 | Ingestion extraction | Parsers, cleaning, chunking, sections and `IngestPipeline`; chunk-count parity with Academy on the open corpus within a stated tolerance; ported tests green; Academy technical golden set re-annotated with section ids | | E2 | Retrieval extraction | Hybrid retrieval, rerank, MMR, parent expansion, EvidencePackBuilder, strict mode; known Academy defects fixed; metrics ≥ Academy baseline | | E3 | Adapters | `qvac` adapter plus the completed worker; `local` GPU adapter; vector store chosen by ADR; JSON BM25; SQL ParentStore; Redis cache; eval passes on both the QVAC and local stacks | | E4 | Agent runtime | Versioned prompt registry; declarative task graph; LLM concurrency control; Academy's 8 study actions expressible as configuration | @@ -307,7 +307,7 @@ Milestones carry no dates. Each one closes when its exit criteria are met. - `priority/*`: p0, p1, p2. - `size/*`: S, M, L. - `status/*`: blocked, needs-design. -- Also `good first issue` and `agent-ready`. +- Also `epic`, `good first issue` and `agent-ready`. ### Board From 10522e648e1fae1e4e16e40c7c20ad3cc4f1c840 Mon Sep 17 00:00:00 2001 From: Luca Ostinelli Date: Fri, 16 Oct 2026 12:26:02 -0500 Subject: [PATCH 3/3] docs(plan): add the project backlog derived from the design spec The backlog follows a rolling-wave model. - M0, M1, M2, E1 and R1 are broken down into ready-to-pick issues with acceptance criteria and verification commands. - E2 to E6 and R2 to R5 are tracked as epics. Their sub-issues are detailed when the preceding milestone closes. --- docs/plans/2026-10-16-backlog.md | 911 +++++++++++++++++++++++++++++++ 1 file changed, 911 insertions(+) create mode 100644 docs/plans/2026-10-16-backlog.md diff --git a/docs/plans/2026-10-16-backlog.md b/docs/plans/2026-10-16-backlog.md new file mode 100644 index 0000000..b421fef --- /dev/null +++ b/docs/plans/2026-10-16-backlog.md @@ -0,0 +1,911 @@ +# Agentic-RAG Backlog + +- **Spec:** `docs/specs/2026-10-16-agentic-rag-design.md` +- **Academy source pin:** `BitPolito/bitcoin-academy@b78d661`. Every porting issue copies code from this commit, and paths are relative to `services/ai/`. +- **Planning model: rolling wave.** + - Milestones M0, M1, M2, E1 and R1 are broken down into issues that are ready to be picked up. + - Later milestones are tracked as one epic issue each, with a sub-issue list. + - An epic's sub-issues are written in full when its predecessor milestone closes, using what was learned by then. Each epic contains the task "Detail the sub-issues of this epic". + +## How to read an issue entry + +Each `### · ` entry becomes one GitHub issue. + +- **Labels:** the labels to apply. Every entry also gets its milestone's `track/*` label. +- **Blocked by:** backlog IDs that must close first. The `Blocked by #n` lines in the issue body are filled in when the issues are created. +- **Body:** the issue template sections: + - Goal + - Scope + - Out of scope + - Files + - Acceptance criteria + - How to verify + +A Definition of Ready applies to every issue: the acceptance criteria are testable, the blockers are closed, and the size is L or smaller. + +--- + +## M0 · Foundation + +Exit criteria: + +- CI is green on `main`, and `main` is protected. +- Contributor docs, templates, labels and the board are in place. +- ADR-0001 to ADR-0004 are accepted. + +### M0-1 · Scaffold the uv workspace and both packages +- **Labels:** type/chore, area/infra, priority/p0, size/M, agent-ready +- **Blocked by:** none + +**Goal:** Create the uv workspace and two importable, typed, empty packages, as defined in spec §4 and ADR-0002. + +**Scope:** +- Root `pyproject.toml`: + - `[tool.uv.workspace] members = ["packages/*"]` + - dev dependency group: `ruff`, `mypy`, `pytest`, `pytest-cov`, `import-linter` +- Python version file, `.python-version` = `3.11`. +- `packages/engine/pyproject.toml`: + - distribution `agentic-rag`, version `0.0.0`, hatchling build + - `src/` layout with `src/agentic_rag/__init__.py` exposing `__version__`, and `py.typed` +- `packages/code-rag/pyproject.toml`: + - distribution `agentic-rag-code` + - depends on `agentic-rag` through `[tool.uv.sources] agentic-rag = { workspace = true }` + - `src/agentic_rag_code/__init__.py` and `py.typed` +- One smoke test per package that asserts `__version__ == "0.0.0"`. +- `.gitignore` covering Python, uv, IDE files, `.env`, `data/`, `research/**/results/raw/` and model caches. +- `.gitattributes` with `* text=auto eol=lf`. +- The directories `research/{benchmark,datasets,experiments,report}`, `services/` and `docs/adr`. Each has a `README.md` stating its purpose in one paragraph. + +**Out of scope:** CI, linters, any engine code. + +**Acceptance criteria:** +- [ ] `uv sync` succeeds from a clean clone. +- [ ] `uv run pytest` runs and passes both smoke tests. +- [ ] `uv run python -c "import agentic_rag, agentic_rag_code"` succeeds. + +**How to verify:** `uv sync && uv run pytest -q` + +### M0-2 · Configure ruff, mypy, pytest coverage and import-linter +- **Labels:** type/chore, area/infra, priority/p0, size/S, agent-ready +- **Blocked by:** M0-1 + +**Goal:** Encode the quality rules from spec §9 and the dependency rules from ADR-0001 as configuration. + +**Scope:** Add the following to the root `pyproject.toml`: + +- **ruff:** + - `target-version = "py311"`, `line-length = 100` + - rule sets `E,F,W,I,B,UP,SIM,RUF,D`, with pydocstyle convention `google` + - `D` ignored under `tests/` +- **mypy:** `strict = true`, `files = ["packages"]`. +- **pytest:** + - `testpaths = ["packages"]` + - `addopts = "--cov=agentic_rag --cov=agentic_rag_code --cov-report=term-missing --cov-fail-under=80"` +- **import-linter**, three contracts: + 1. `agentic_rag.types` and `agentic_rag.ports` must not import `agentic_rag.adapters`. + 2. `agentic_rag` must not import `agentic_rag_code`. + 3. Layers, from top to bottom: `agentic_rag.adapters`, then `agentic_rag.ingest | agentic_rag.retrieval | agentic_rag.generation | agentic_rag.agent | agentic_rag.eval`, then `agentic_rag.evidence`, then `agentic_rag.ports`, then `agentic_rag.types`. Skeleton subpackages with an empty `__init__.py` are created so the contracts resolve. + +**Acceptance criteria:** +- [ ] `uv run ruff check . && uv run ruff format --check .` passes. +- [ ] `uv run mypy` passes. +- [ ] `uv run lint-imports` reports 3 contracts kept. +- [ ] Adding `import agentic_rag.adapters` inside `agentic_rag/types/__init__.py` makes `lint-imports` fail. Check this locally and do not commit the change. + +**How to verify:** `uv run ruff check . && uv run mypy && uv run lint-imports && uv run pytest -q` + +### M0-3 · CI workflow with a single quality gate, plus Dependabot +- **Labels:** type/chore, area/infra, priority/p0, size/M, agent-ready +- **Blocked by:** M0-2 + +**Goal:** Give every pull request one required check that aggregates all quality jobs. + +**Scope:** +- `.github/workflows/ci.yml`: + - Triggers: `pull_request` and `push` to `main`. + - `concurrency` with `cancel-in-progress: true`. + - Jobs, each using `astral-sh/setup-uv` with caching: + - `lint`: ruff check and ruff format --check + - `typecheck`: mypy + - `import-contracts`: lint-imports + - `test`: pytest with coverage, writing a coverage summary to `$GITHUB_STEP_SUMMARY` + - `quality-gate`: `needs` all of the above and fails if any of them did not succeed +- `.github/dependabot.yml`: + - Ecosystems `uv` (root) and `github-actions`. + - Schedule: weekly on Monday, timezone Europe/Rome. + - Minor and patch updates grouped; major updates as separate PRs. + - `npm` for `services/qvac-worker` is added later, in E3. + +**Acceptance criteria:** +- [ ] A PR shows the jobs `lint`, `typecheck`, `import-contracts`, `test` and `quality-gate`. +- [ ] A PR with a ruff violation makes `quality-gate` fail. +- [ ] A push to `main` runs the workflow. + +**How to verify:** open a draft PR containing a deliberate lint error, confirm the gate fails, then remove the error. + +### M0-4 · Contributor documentation and CODEOWNERS +- **Labels:** type/docs, area/infra, priority/p0, size/M +- **Blocked by:** M0-1 + +**Goal:** A new contributor can go from cloning the repository to merging a PR by reading only these documents. + +**Scope:** +- `CONTRIBUTING.md`, covering: + - setup with `uv sync` + - the workflow from spec §9: branch naming `<type>/<issue>-<desc>`, Conventional Commits with the scope list, PR rules, merging with `gh pr merge --merge --delete-branch` + - the Definition of Ready + - how to pick up a `Ready` issue + - the before/after eval block required for retrieval changes + - when an ADR is required +- `AGENTS.md`: the same rules condensed for automated contributors, plus these instructions: + - read the spec and the relevant ADRs first + - never edit outside the issue's scope + - always run the verify commands +- `docs/architecture.md`: a one-page overview with the package map and the data-flow diagram from spec §5.3, linking to the spec and the ADRs. +- `.github/CODEOWNERS`: `* @BitPolito/ai-ml`, and `/docs/adr/ @lucaosti`. +- The README gains a quick start and links to these documents. + +**Acceptance criteria:** +- [ ] Every rule in spec §9 appears in CONTRIBUTING.md. +- [ ] CODEOWNERS requests review from `@BitPolito/ai-ml` on a test PR. + +**How to verify:** open a PR and check the reviewers that were requested automatically. + +### M0-5 · Issue forms and PR template +- **Labels:** type/chore, area/infra, priority/p1, size/S, agent-ready, good first issue +- **Blocked by:** none + +**Goal:** Make the issue structure from spec §9 the default. + +**Scope:** +- `.github/ISSUE_TEMPLATE/`, with these forms: + - `feature.yml`: Context, Goal, Scope, Out of scope, Files and interfaces, Acceptance criteria, How to verify, Dependencies, Size dropdown + - `bug.yml`: Observed, Expected, Reproduction, Environment + - `research-experiment.yml`: Hypothesis, Setup/config, Metrics, Expected outcome, Compute budget + - `epic.yml`: Goal, Exit criteria, Sub-issues + - `config.yml`: `blank_issues_enabled: false` +- `.github/pull_request_template.md`: What, Why, `Closes #`, How verified, Eval before/after (marked required for retrieval changes), Reviewer notes, Checklist. + +**Acceptance criteria:** +- [ ] The "New issue" page on GitHub shows the four forms. +- [ ] A new PR is pre-filled with the template. + +**How to verify:** open the "New issue" page on GitHub. + +### M0-6 · Protect main and configure merge settings +- **Labels:** type/chore, area/infra, priority/p0, size/S +- **Blocked by:** M0-3 + +**Goal:** Enforce the merge rules from spec §9 on GitHub. Repository admins only. + +**Scope:** +- **Branch protection on `main`:** + - require a PR + - require 1 approval, and dismiss stale approvals + - require the status check `quality-gate`, and require branches to be up to date + - block force-pushes and deletions +- **Repository settings:** + - allow merge commits only, with squash and rebase disabled + - automatically delete head branches + +**Acceptance criteria:** +- [ ] A direct push to `main` is rejected. +- [ ] A PR cannot be merged while `quality-gate` is red or before it has an approval. + +**How to verify:** run `gh api repos/BitPolito/Agentic-RAG/branches/main/protection` and check the settings. + +### M0-7 · Accept ADR-0001 to ADR-0004 +- **Labels:** type/docs, area/infra, priority/p1, size/S, good first issue +- **Blocked by:** none + +**Goal:** Change the status of the four foundational ADRs from Proposed to Accepted once the spec PR is merged, and update the index in `docs/adr/README.md`. + +**Acceptance criteria:** +- [ ] All four ADRs and the index show `Accepted`. + +**How to verify:** read `docs/adr/README.md`. + +--- + +## M1 · Core contracts + +Exit criteria: + +- `types`, `ports`, `EngineConfig` and the in-memory fakes exist. +- A toy end-to-end pipeline passes using only the fakes. +- The EvidencePack contract test is in place. + +### M1-1 · ADR-0005: finalize port signatures, including the sync vs async decision +- **Labels:** type/research, area/adapters, priority/p0, size/M, status/needs-design +- **Blocked by:** M0-1 + +**Goal:** Fix the exact Protocol signatures for every port in spec §5.1 before anyone implements them. + +**Scope:** ADR-0005 must decide the following: + +1. Sync or async APIs. The options: + - async-first, with sync wrappers + - sync-first, with LLM streaming as an async iterator + + Weigh Academy's usage (FastAPI, httpx async) against the research scripts. +2. The batch shape of `embed`. +3. The filter grammar for `VectorStore.search`. +4. How namespaces are passed: per call, or a store bound to a namespace. +5. The `LLM` message format and the options object. +6. The error taxonomy in `agentic_rag.errors`. + +Validate the decisions against two call sites: Academy's `rag/retriever.py::unified_retrieval`, and a sketched `CodeRetriever` (spec §6). + +**Out of scope:** writing the code itself (done in M1-3). + +**Acceptance criteria:** +- [ ] ADR-0005 has been merged with status Accepted. +- [ ] It lists every port method with its full signature and docstring summary. + +**How to verify:** an ADR review by at least one engine contributor and one research contributor. + +### M1-2 · Core types +- **Labels:** type/feature, area/ingest, priority/p0, size/M, agent-ready +- **Blocked by:** M0-2 + +**Goal:** Implement the domain-neutral data model in `agentic_rag.types`. + +**Scope:** frozen Pydantic v2 models. + +- `Namespace`, a validated slug matching `^[a-z0-9][a-z0-9_-]{0,63}$`. +- `BlockKind`, an enum: heading, paragraph, table, formula, code, list, figure_caption. +- `Block`, with kind, text, level, page and slide. +- `Document`, with id, namespace, source_name, mime_type, blocks and metadata. Port it from Academy's `schemas/normalized_document.py`, dropping the course and exam fields. +- `ParentChunk`, with id, doc_id, namespace, text, section, page and slide. +- `Chunk`, with id, parent_id (an explicit field: never derive it from the id), doc_id, namespace, text, kind, section, page, slide and token_count. +- `SectionNode`, with title, level, children and parent_ids. +- `IngestResult`, with doc_id, parents, chunks, section_tree, stats and warnings. + +**Files:** `packages/engine/src/agentic_rag/types/{__init__,document,chunk,ingest}.py` and `packages/engine/tests/types/`. + +**Acceptance criteria:** +- [ ] Invalid namespaces are rejected. Tested cases: an empty string, uppercase letters, a slash, and 65 characters. +- [ ] A `Chunk` cannot be built without a `parent_id`. +- [ ] Every model round-trips through `model_dump_json` / `model_validate_json`. +- [ ] mypy passes in strict mode. + +**How to verify:** `uv run pytest packages/engine/tests/types -q && uv run mypy` + +### M1-3 · Ports and errors +- **Labels:** type/feature, area/adapters, priority/p0, size/M, agent-ready +- **Blocked by:** M1-1, M1-2 + +**Goal:** Implement the Protocols exactly as written in ADR-0005. + +**Scope:** +- `agentic_rag/ports/`, one module per port: embedder, vector_store, lexical_index, parent_store, reranker, llm, cache, token_counter. +- Each Protocol is `@runtime_checkable`. +- `agentic_rag/errors.py` defines `AgenticRagError` as the base class, with these subclasses: + - `BackendUnavailable` + - `LlmDisabled` + - `StructuredGenerationError` + - `IndexNotFound` + - `ConfigError` + +**Acceptance criteria:** +- [ ] The signatures match ADR-0005 one to one. +- [ ] `lint-imports` passes. +- [ ] Every error carries the name of the backend that raised it. + +**How to verify:** `uv run mypy && uv run lint-imports` + +### M1-4 · EvidencePack v1 contract +- **Labels:** type/feature, area/retrieval, priority/p0, size/S, agent-ready +- **Blocked by:** M1-2 + +**Goal:** Turn Academy's `schemas/evidence_pack.py` into a versioned, domain-neutral public contract. + +**Scope:** +- `CitationAnchor`: + - fields: doc_id, doc_name (the real document name), citation_label, section, page, slide, chunk_id, kind + - optional code fields for Code-RAG: file_path, symbol, line_start, line_end +- `EvidenceChunk`: as in Academy. +- `EvidencePack`: + - `schema_version: Literal["1"]` + - `task: str | None`, replacing Academy's `action` + - all other fields as in Academy +- `context_block()` keeps the format `[ref_N] text` joined by `\n\n---\n\n`. +- A contract test compares `EvidencePack.model_json_schema()` with the committed snapshot `tests/contracts/evidence_pack.v1.json`. + +**Acceptance criteria:** +- [ ] The snapshot test fails on any change to a field. +- [ ] `context_block()` output matches Academy's for the same passages. Add a ported test fixture. + +**How to verify:** `uv run pytest packages/engine/tests/contracts -q` + +### M1-5 · EngineConfig +- **Labels:** type/feature, area/adapters, priority/p1, size/S, agent-ready +- **Blocked by:** M0-2 + +**Goal:** One typed configuration object, and no reads of the environment at import time. + +**Scope:** +- `agentic_rag/config.py` defines `EngineConfig(BaseSettings)`: + - `env_prefix="ARAG_"`, `env_nested_delimiter="__"` + - nested sections: `chunking`, `retrieval`, `evidence`, `generation` +- Defaults are taken from Academy: + + | Section | Field | Default | + |---|---|---| + | chunking | parent_words | 1200 | + | chunking | child_words | 150 | + | chunking | child_overlap | 30 | + | chunking | child_min_words | 25 | + | chunking | child_max_words | 350 | + | chunking | child_max_tokens | 400 | + | retrieval | retrieve_k | 20 | + | retrieval | top_k | 5 | + | retrieval | dense_weight | 0.6 | + | retrieval | lexical_weight | 0.4 | + | retrieval | mmr_lambda | 0.6 | + | retrieval | rrf_k | 60 | + | evidence | max_evidence | 6 | + | evidence | max_tokens | 2000 | + +- Test that no module under `agentic_rag` calls `os.getenv` or `os.environ`. The test is a grep. + +**Acceptance criteria:** +- [ ] `ARAG_RETRIEVAL__TOP_K=7` overrides `retrieval.top_k`. +- [ ] Invalid weights are rejected: `dense_weight + lexical_weight != 1`. +- [ ] The grep test passes. + +**How to verify:** `uv run pytest packages/engine/tests/config -q` + +### M1-6 · In-memory fakes and the port conformance suite +- **Labels:** type/feature, area/adapters, priority/p0, size/M, agent-ready +- **Blocked by:** M1-3 + +**Goal:** Provide a deterministic fake for every port, plus a reusable conformance suite that every future adapter must pass. + +**Scope:** +- `agentic_rag/adapters/memory/`: + - `HashEmbedder`: deterministic vectors from hashed character n-grams; the dimension is configurable. + - `InMemoryVectorStore`: cosine similarity. + - `InMemoryLexicalIndex`: simple BM25. + - `InMemoryParentStore`. + - `PassthroughReranker`. + - `EchoLLM`: returns the evidence with `[ref_N]` citations; can be set up to fail. + - `DictCache`. + - `WhitespaceTokenCounter`. +- `agentic_rag/testing/conformance.py`: parametrizable pytest suites, one per port, covering upsert/search/delete semantics, namespace isolation and the error types. + +**Acceptance criteria:** +- [ ] Every fake passes its conformance suite. +- [ ] Documents in namespace A are never returned for namespace B. +- [ ] After `delete(doc_id)`, searches return none of that document's chunks. + +**How to verify:** `uv run pytest packages/engine/tests/adapters/memory -q` + +### M1-7 · Toy end-to-end pipeline on fakes +- **Labels:** type/test, area/retrieval, priority/p1, size/M, agent-ready +- **Blocked by:** M1-4, M1-5, M1-6 + +**Goal:** Prove that the ports compose, before any Academy code is ported. + +**Scope:** An integration test and the example `examples/toy_pipeline.py`, both using only the fakes: + +1. Ingest three small markdown documents with a naive paragraph splitter. +2. Run a hybrid query with weighted fusion. +3. Build an `EvidencePack`. +4. Pass it to `EchoLLM`. + +The test asserts that the pack's citations point at the expected document. + +**Acceptance criteria:** +- [ ] The test runs in under 2 seconds with no network access. +- [ ] The example runs through `uv run python examples/toy_pipeline.py`. + +**How to verify:** `uv run pytest packages/engine/tests/integration -q` + +### M1-8 · Code-RAG spike against the ports (throwaway) +- **Labels:** type/research, area/code-rag, priority/p1, size/S +- **Blocked by:** M1-1 + +**Goal:** Check that the ports fit code retrieval before both tracks start depending on them. + +**Scope:** Sketch the following against the draft signatures in ADR-0005: + +- indexing symbols from one small Python repository; +- graph expansion through `ParentStore`, or a new port. + +The output is a short report in the issue, plus proposed amendments to ADR-0005 if any are needed. Do not merge any code. + +**Acceptance criteria:** +- [ ] The report lists every gap found (for example the need for a `CodeGraphStore` port) and how it is resolved. + +--- + +## M2 · Evaluation harness + +Exit criteria: + +- IR metrics and a runner exist. +- The golden-set format is defined. +- The golden-set format is ready for porting the Academy golden set (the re-annotation itself is E1-8). +- The baseline report is reproducible. + +### M2-1 · Golden-set format and loader +- **Labels:** type/feature, area/eval, priority/p0, size/S, agent-ready, good first issue +- **Blocked by:** M1-2 + +**Goal:** Define the format of evaluation datasets. + +**Scope:** +- A JSONL file with one query per line. Fields: + - required: `id`, `query`, `relevant_ids: list[str]` + - optional: `relevance: dict[str, int]` (graded relevance for nDCG), `ground_truth`, `keywords`, `category` +- `agentic_rag/eval/golden.py` provides `GoldenQuery` (a Pydantic model) and `load_golden(path) -> list[GoldenQuery]`. The loader rejects duplicate ids and empty `relevant_ids`. + +**Acceptance criteria:** +- [ ] Valid and invalid fixtures are tested. + +**How to verify:** `uv run pytest packages/engine/tests/eval -q` + +### M2-2 · IR metrics +- **Labels:** type/feature, area/eval, priority/p0, size/S, agent-ready, good first issue +- **Blocked by:** M0-2 + +**Goal:** Pure-function metrics in `agentic_rag/eval/metrics.py`. + +**Scope:** The following functions, all of shape `(retrieved: list[str], relevant: set[str], k: int) -> float`: + +- `recall_at_k` +- `precision_at_k` +- `mrr` +- `ndcg_at_k`, which takes an optional graded `relevance` + +**Acceptance criteria:** +- [ ] The hand-computed fixture `retrieved=[a,b,c,d]`, `relevant={b,d}`, `k=4` gives recall 1.0, precision 0.5 and MRR 0.5. +- [ ] nDCG equals 1.0 for an ideal ranking. +- [ ] Edge cases are handled: `k` larger than `len(retrieved)`, an empty retrieved list, and `k=0`, which raises `ValueError`. + +**How to verify:** `uv run pytest packages/engine/tests/eval/test_metrics.py -q` + +### M2-3 · Eval runner and the arag-eval CLI +- **Labels:** type/feature, area/eval, priority/p0, size/M, agent-ready +- **Blocked by:** M2-1, M2-2, M1-7 + +**Goal:** Produce a reproducible, metrics-only report for any retriever configuration. + +**Scope:** +- `agentic_rag/eval/runner.py` provides `run_eval(retriever, golden, ks=(1,3,5,10)) -> EvalReport`. +- `EvalReport` contains: + - per-query metrics + - aggregate metrics: mean recall, precision, MRR and nDCG for each k + - latency p50 and p95 + - per-category breakdown + - provenance: git SHA, config hash, dataset hash, timestamp, hostname, GPU name if available +- `arag-eval` is a console script that takes `--config <yaml> --golden <jsonl> --out <json>` and prints a SUMMARY block, the one pasted into PRs. +- Reports never contain corpus text. + +**Acceptance criteria:** +- [ ] Two runs with the same seed and the fakes produce identical metrics. +- [ ] The SUMMARY block is at most 20 lines. + +**How to verify:** `uv run arag-eval --config examples/eval/toy.yaml --golden examples/eval/toy.jsonl --out /tmp/r.json` + +### M2-4 · Port the open-corpus fetch script +- **Labels:** type/chore, area/dataset, priority/p1, size/S, agent-ready +- **Blocked by:** M0-1 + +**Goal:** Fetch Mastering Bitcoin 3e reproducibly. Source: `scripts/fetch_rag_corpus.py` at the Academy pin. + +**Scope:** +- `research/datasets/mastering-bitcoin/fetch.py` keeps: + - the pinned upstream commit + - SHA-256 verification + - conversion to `.md` +- Output goes to `data/corpora/mastering-bitcoin/`, which git ignores. +- A dataset card with the license (CC BY-SA 4.0), the source and the commit. + +**Acceptance criteria:** +- [ ] A second run is a no-op. +- [ ] A tampered file makes the checksum verification fail. + +**How to verify:** `uv run python research/datasets/mastering-bitcoin/fetch.py` + +### M2-6 · eval-smoke CI job +- **Labels:** type/chore, area/eval, priority/p1, size/S, agent-ready +- **Blocked by:** M2-3, M0-3 + +**Goal:** Catch retrieval regressions on every PR without models or network access. + +**Scope:** +- A CI job, `eval-smoke`, runs `arag-eval` with the toy config on the fakes. +- It compares the result with the committed `examples/eval/toy.baseline.json` and fails if any metric drops by more than 0.01. +- The job is wired into `quality-gate`. + +**Acceptance criteria:** +- [ ] A PR that breaks fusion ordering makes the gate fail. + +--- + +## E1 · Ingestion extraction + +Exit criteria: + +- Parsers, cleaning, chunking, the section tree and `IngestPipeline` are in place. +- On the open corpus, the chunk count is within ±2% of Academy's. +- The ported tests are green. +- The technical golden set is re-annotated (E1-8). + +### E1-1 · Parser registry and PDF parser +- **Labels:** type/feature, area/ingest, priority/p0, size/M, agent-ready +- **Blocked by:** M1-2 + +**Goal:** Turn files into `Document`s through an extension-based registry. Source: `workers/pipeline.py::parse_pdf_pages`, lines 228–251. + +**Scope:** +- `ingest/parsers/base.py` defines the `Parser` Protocol, `parse(path) -> Document`. +- `ingest/parsers/registry.py` provides `ParserRegistry.register(ext, parser)` and `.for_path(path)`. An unknown extension raises `UnsupportedFormat`. +- `ingest/parsers/pdf.py` uses `pymupdf4llm.to_markdown(page_chunks=True)` and maps each page to blocks with its page number. Dependencies go in the `[pdf]` extra. +- Port the matching tests from `tests/unit/test_pipeline_internals.py`. + +**Acceptance criteria:** +- [ ] The sample PDF parses to the same page count as Academy's parser. +- [ ] Every block carries its page number. + +**How to verify:** `uv run pytest packages/engine/tests/ingest/parsers -q` + +### E1-2 · PPTX and DOCX parsers +- **Labels:** type/feature, area/ingest, priority/p1, size/S, agent-ready, good first issue +- **Blocked by:** E1-1 + +**Goal:** Port `parse_pptx_pages` and `parse_docx_pages`. + +**Scope:** +- The PPTX parser puts the slide number in `Block.slide`, which fixes Academy's `citation_slide=0` defect. It keeps the slide title, body and notes. +- The DOCX parser maps headings to heading blocks with their level. + +**Acceptance criteria:** +- [ ] The sample PPTX produces blocks whose `slide` values run from 1 to N. +- [ ] Heading levels from the DOCX are preserved. + +### E1-3 · Cleaning with pluggable rules +- **Labels:** type/feature, area/ingest, priority/p1, size/S, agent-ready +- **Blocked by:** E1-1 + +**Goal:** Port `detect_boilerplate` and `clean_page` without the book-specific rules. + +**Scope:** +- `ingest/clean/boilerplate.py` keeps Academy's thresholds: a line counts as boilerplate when it appears on at least 30% of pages, and detection runs only on documents with at least 5 pages. +- `ingest/clean/rules.py` defines `CleaningRule(pattern, replacement)` and `apply_rules`. +- Academy's `_STRIP_PATTERNS` and `_SPURIOUS_HEADING_RE` move to `examples/rules/academy_books.py`. + +**Acceptance criteria:** +- [ ] With no rules configured, only the generic cleaning runs. +- [ ] With the Academy rule set, the output matches Academy on the ported fixtures. + +### E1-4 · Block splitter +- **Labels:** type/feature, area/ingest, priority/p0, size/S, agent-ready +- **Blocked by:** E1-1 + +**Goal:** Port `_split_into_blocks` and `_is_spurious_heading` (lines 376–452), operating on `Document` blocks. + +**Acceptance criteria:** +- [ ] Headings, tables, `$$` formulas, code fences and paragraphs are classified like Academy on the ported fixtures. + +### E1-5 · Parent/child chunker with an injectable TokenCounter +- **Labels:** type/feature, area/ingest, priority/p0, size/M, agent-ready +- **Blocked by:** E1-4, M1-5, M1-6 + +**Goal:** Port `build_parent_child_chunks` (lines 604–739), with sizes taken from `EngineConfig.chunking` and token caps measured by the injected `TokenCounter` instead of Academy's `_estimate_tokens`. + +**Scope:** +- Keep Academy's rules: + - parents are sentence-anchored; + - children have overlap; + - tables, formulas and code become a single child; + - a minimum word count applies, set to 4 for tables; + - after the hard word cap, a token cap applies. +- Set `Chunk.parent_id` explicitly. +- Port `filter_chunks`, which drops chunks that are too short or more than 60% figure caption. + +**Acceptance criteria:** +- [ ] No child exceeds `child_max_tokens` according to the injected counter. +- [ ] Every child's `parent_id` exists among the parents. +- [ ] The ported tests from `test_pipeline_internals.py` pass. + +### E1-6 · Section tree +- **Labels:** type/feature, area/ingest, priority/p1, size/S, agent-ready, good first issue +- **Blocked by:** E1-5 + +**Goal:** Port `build_section_tree` and `build_section_events_from_parents`, producing `SectionNode` trees. Section ids must be stable, because golden sets refer to them. + +**Acceptance criteria:** +- [ ] The ported tests from `test_section_tree.py` pass. +- [ ] Section ids are deterministic across runs. + +### E1-7 · IngestPipeline with progress events and a parity test +- **Labels:** type/feature, area/ingest, priority/p0, size/M, agent-ready +- **Blocked by:** E1-2, E1-3, E1-5, E1-6 + +**Goal:** Implement the ingestion data flow from spec §5.3. + +**Scope:** `ingest/pipeline.py` defines `IngestPipeline(parsers, cleaner, chunker, embedder, vector_store, lexical_index, parent_store, token_counter, on_event=None)`. + +- `.ingest(path, namespace, doc_id) -> IngestResult` emits the stage events `parsing`, `cleaning`, `chunking`, `embedding`, `indexing` and `done`. +- `.delete(namespace, doc_id)` removes the document from every store. +- A parity test runs on the open corpus. It is marked `@pytest.mark.corpus` and skipped when the corpus is missing. It compares chunk counts with the recorded Academy numbers in `tests/data/academy_parity.json`, with a tolerance of ±2%. + +**Acceptance criteria:** +- [ ] The pipeline works end to end on the fakes. +- [ ] After delete, all three stores are empty. +- [ ] The parity test passes locally. Record its output in the PR. + +**How to verify:** `uv run pytest -m corpus packages/engine/tests/ingest -q` + +### E1-8 · Re-annotate the technical golden set with relevant ids +- **Labels:** type/research, area/dataset, priority/p1, size/M +- **Blocked by:** M2-1, E1-6, E1-7 + +**Goal:** Turn Academy's 41 technical queries (`tests/data/rag_golden_technical.json`) into a golden set with real `relevant_ids` (section ids from the engine's section tree), so recall, MRR and nDCG become meaningful. + +**Scope:** +- Ingest the corpus with E1's pipeline. +- Annotate one to three relevant sections per query. Two annotators each label half of the queries, and each cross-checks the other's half. +- Record the agreement rate. + +**Acceptance criteria:** +- [ ] `research/datasets/mastering-bitcoin/golden_technical.jsonl` passes `load_golden`. +- [ ] The annotation guidelines are documented in the dataset card. + +--- + +## E2 · Epic: Retrieval extraction +- **Labels:** type/feature, area/retrieval, priority/p0, size/L, epic +- **Blocked by:** M2-3, E1-7 + +**Exit criteria:** +- Hybrid retrieval, reranking, MMR, parent expansion, `EvidencePackBuilder` and strict mode are in place. +- The known Academy defects are fixed. +- Metrics are at least the Academy baseline. + +**Sub-issues:** to be detailed when E1 closes. + +- [ ] Detail the sub-issues of this epic. +- [ ] Lexical tokenizer (CamelCase splitting, injectable `SynonymMap`) and the BM25 algorithm. Source: `services/hybrid_search.py`. +- [ ] Fusion: weighted normalized fusion and RRF, chosen by config. +- [ ] Reranker wrappers: flashrank, then cross-encoder, then passthrough as fallbacks. Source: `services/reranker.py`. +- [ ] MMR selection using the injected `Embedder`. +- [ ] Parent expansion through `ParentStore`, with sibling dedup. Source: `services/parent_expansion.py`. +- [ ] `HybridRetriever` composing the above, with explicit degradation when dense retrieval is unavailable. +- [ ] `EvidencePackBuilder`: dedup, token budget, ordering, and no second rerank or expansion pass. +- [ ] Query transforms: HyDE and rewrite, with the persona injected. Source: `rag/query_rewriter.py`. +- [ ] Opt-in extractive compressor. Source: `rag/compressor.py`. +- [ ] Generation helpers: context formatting, the `[ref_N]` citation parser, strict mode, and a gate that refuses to answer without evidence. +- [ ] Baseline comparison against Academy on the open corpus, as a report in the PR. + +## E3 · Epic: Adapters +- **Labels:** type/feature, area/adapters, priority/p1, size/L, epic +- **Blocked by:** M1-6 + +**Exit criteria:** +- The `qvac` adapter and a completed worker exist. +- The `local` GPU adapter exists. +- A vector store has been chosen by ADR. +- BM25 is persisted as JSON. +- A SQL `ParentStore` and a Redis cache exist. +- The eval passes on both the QVAC and the local stack. + +**Sub-issues:** +- [ ] Detail the sub-issues of this epic. +- [ ] Port `workers/qvac-service` to `services/qvac-worker`. Add a Node CI job and an npm Dependabot entry. +- [ ] Worker: add `DELETE /documents/{id}` and `DELETE /workspaces/{id}`, index every chunk kind, and publish the contract in `contract/`. +- [ ] `adapters/qvac`: Embedder, VectorStore and LLM (generate, stream, generate_json), passing the conformance suites with a mocked HTTP transport. +- [ ] `adapters/local`: a sentence-transformers Embedder (GPU) and a llama.cpp LLM (llama-cpp-python, CUDA). +- [ ] ADR-0006: choose the vector store, FAISS or LanceDB, using a latency and throughput benchmark on the RTX 5080. Then implement it. +- [ ] `adapters/lexical`: BM25 persisted as JSON with incremental add and delete. No pickle. +- [ ] `adapters/sql`: a SQLAlchemy `ParentStore`. +- [ ] `adapters/cache`: a Redis semantic cache. Source: `services/cache_service.py`. +- [ ] A dual-stack eval report: QVAC and local. + +## E4 · Epic: Agent runtime +- **Labels:** type/feature, area/agent, priority/p2, size/L, epic +- **Blocked by:** E2 + +**Exit criteria:** +- A versioned prompt registry exists. +- Tasks are described by a declarative task graph. +- LLM concurrency is controlled. +- Academy's 8 study actions can be expressed as configuration. + +**Sub-issues:** +- [ ] Detail the sub-issues of this epic. +- [ ] ADR-0007: task graph model. Compare a custom state graph, LangGraph and plain chains. +- [ ] Prompt registry with versions and per-task system prompts. Source: `prompts/registry.py`. +- [ ] Task specification: system prompt, tools, retrieval policy (single or two-hop) and output schema. +- [ ] LLM concurrency limiter, equivalent to the worker's FIFO lock. +- [ ] Express Academy's 8 study actions as configuration, in `examples/academy_tasks/`. + +## E5 · Epic: Engine v0.x release +- **Labels:** type/chore, area/infra, priority/p2, size/M, epic +- **Blocked by:** E2, E3 + +**Exit criteria:** +- API docs and a changelog exist. +- The engine follows semantic versioning. +- It can be installed from a git tag. + +**Sub-issues:** +- [ ] API reference site (mkdocs-material with mkdocstrings), published to GitHub Pages. +- [ ] `CHANGELOG.md` following Keep a Changelog, and a documented release process. +- [ ] Tag `engine-v0.1.0`, and verify `uv add "agentic-rag @ git+https://github.com/BitPolito/Agentic-RAG@engine-v0.1.0#subdirectory=packages/engine"`. + +## E6 · Epic: Academy migration (cross-repository) +- **Labels:** type/refactor, area/adapters, priority/p2, size/L, epic +- **Blocked by:** E5 + +**Exit criteria:** +- Academy depends on `agentic_rag[qvac]`, and its duplicated code is removed. +- Academy's CI is green. +- Academy's eval is unchanged or better. + +**Sub-issues:** these are filed in `BitPolito/bitcoin-academy`. + +- [ ] Replace `rag/` and the generic `services/` modules with engine calls. +- [ ] Map engine ingest events to `DocumentProcessingStage`. +- [ ] Re-baseline Academy's eval. + +--- + +## R1 · Code ingestion + +Exit criteria: + +- Python and TypeScript are parsed with tree-sitter. +- A symbol graph and an AST chunker exist. +- A mid-size repository can be indexed and queried. + +### R1-1 · ADR-0008: code graph model and CodeGraphStore port +- **Labels:** type/research, area/code-rag, priority/p0, size/M, status/needs-design +- **Blocked by:** M1-1, M1-8 + +**Goal:** Fix the code data model before anything parses code. + +**Scope:** +- Node types: `File`, `Module`, `Class`, `Function`, `Method`. +- Edge types: `defines`, `calls`, `imports`, `inherits`, `references`. +- A stable `symbol_id` scheme: `<repo>:<path>::<qualified.name>`. +- The `CodeGraphStore` port: `add_nodes`, `add_edges`, `neighbors(symbol_id, edge_types, depth)`, `delete_file`. +- First adapter: SQLite or networkx. Decide using the expected repository sizes. + +**Acceptance criteria:** +- [ ] The ADR has been accepted by both tracks. + +### R1-2 · Tree-sitter Python symbol extractor +- **Labels:** type/feature, area/code-rag, priority/p0, size/M, agent-ready +- **Blocked by:** R1-1 + +**Goal:** Implement `agentic_rag_code/parsers/python.py` with `extract(path, source) -> FileSymbols`. + +**Scope:** +- Extract modules, classes, functions and methods, with qualified name, signature, docstring, line range, decorators and imports. +- Also extract call sites: the callee name and the line. +- Uses the `tree-sitter` and `tree-sitter-python` packages. + +**Acceptance criteria:** +- [ ] The fixture file covers nested classes, async functions, decorators, relative imports, and a syntax error that must not crash the extractor. +- [ ] Line ranges are 1-based and inclusive. + +**How to verify:** `uv run pytest packages/code-rag/tests/parsers -q` + +### R1-3 · Tree-sitter TypeScript symbol extractor +- **Labels:** type/feature, area/code-rag, priority/p1, size/M, agent-ready +- **Blocked by:** R1-2 + +**Goal:** The same `FileSymbols` output for `.ts` and `.tsx` files. + +**Scope:** +- Cover functions, arrow functions assigned to `const`, classes, methods, interfaces, type aliases, ES imports and exports, and call sites. + +**Acceptance criteria:** +- [ ] Fixtures cover default exports, re-exports, generics and JSX. + +### R1-4 · Symbol graph builder and the first CodeGraphStore adapter +- **Labels:** type/feature, area/code-rag, priority/p0, size/M, agent-ready +- **Blocked by:** R1-2 + +**Goal:** Resolve imports and call sites into edges across files, best effort, and persist them in the adapter chosen in R1-1. + +**Acceptance criteria:** +- [ ] On the fixture repository, `neighbors("fixture:pkg/a.py::f", {"calls"}, 1)` returns exactly the expected callees. +- [ ] Unresolved calls are kept with an `unresolved` flag rather than dropped. + +### R1-5 · AST-aware chunker +- **Labels:** type/feature, area/code-rag, priority/p0, size/M, agent-ready +- **Blocked by:** R1-2, M1-4 + +**Goal:** Map symbols to engine `Chunk`/`ParentChunk` objects: + +- each symbol becomes a chunk; +- the enclosing class or module becomes its parent; +- bodies above the token cap are split at statement boundaries; +- the anchor's code fields (`file_path`, `symbol`, `line_start`, `line_end`) are filled. + +**Acceptance criteria:** +- [ ] No chunk splits a statement. +- [ ] Every chunk resolves back to its exact source lines. + +### R1-6 · Repository indexer CLI and demo +- **Labels:** type/feature, area/code-rag, priority/p1, size/M +- **Blocked by:** R1-3, R1-4, R1-5 + +**Goal:** `arag-code index <repo_path> --namespace <ns>` indexes a repository through the engine's ports, plus the graph store. `arag-code query` prints the top symbols together with their graph neighbours. + +**Acceptance criteria:** +- [ ] Indexes a mid-size Python repository and a mid-size TypeScript repository. Suggested targets are `httpx` and `zod`. +- [ ] Indexing time and the node and edge counts are reported in the PR. + +--- + +## R2 · Epic: Code retrieval +- **Labels:** type/research, area/code-rag, priority/p1, size/L, epic +- **Blocked by:** R1, E2 + +**Exit criteria:** +- `CodeRetriever` combines semantic, lexical and graph signals. +- Naive and AST-aware retrieval have been compared on a small hand-annotated set. + +**Sub-issues:** +- [ ] Detail the sub-issues of this epic. +- [ ] Experiment: choose code embedding models that fit 16 GB of VRAM alongside the generator. Candidates include nomic-embed-code and jina-embeddings-v2-base-code. +- [ ] Identifier-aware lexical tokenizer (snake_case and camelCase splitting). +- [ ] Graph expansion policy: depth, edge-type weights and budget. +- [ ] `CodeRetriever` returning `EvidencePack` with code anchors. +- [ ] A hand-annotated mini set of about 20 tasks, and a comparison report: naive vs AST-aware vs AST plus graph. + +## R3 · Epic: Benchmark dataset +- **Labels:** type/research, area/dataset, priority/p1, size/L, epic +- **Blocked by:** R1 + +**Exit criteria:** +- The task-building pipeline is reproducible. +- A dataset card and license checks exist. +- v0.1 is released with a stated task count. + +**Sub-issues:** +- [ ] Detail the sub-issues of this epic. +- [ ] Repository selection criteria and a license allow-list (MIT, Apache-2.0, BSD), for Python and TypeScript. +- [ ] Mine issue and PR pairs that have a linked fix commit and test changes. Record creation dates to limit contamination. +- [ ] Docker-based validation: the tests fail before the fix and pass after it. +- [ ] Derive gold files, symbols and AST paths from the fix diff, using R1's parsers. +- [ ] Dataset card, task manifest format, and release v0.1. + +## R4 · Epic: Agent and TCR harness +- **Labels:** type/research, area/agent, priority/p2, size/L, epic +- **Blocked by:** R2, R3, E3, E4 + +**Exit criteria:** +- An orchestrator-workers agent exists. +- Tests run in a sandbox. +- The TCR runner works end to end on the dataset. + +**Sub-issues:** +- [ ] Detail the sub-issues of this epic. +- [ ] Sandbox runner: a container per task, a time and memory budget, and test execution. +- [ ] Planner, retrieval and code-generation workers on the engine's agent runtime. +- [ ] Patch application and validation. +- [ ] TCR runner and report schema. +- [ ] Experiment: choose the generator model (quantized 7–8B code models that fit 16 GB). + +## R5 · Epic: Benchmark runs and technical report +- **Labels:** type/research, area/benchmark, priority/p2, size/L, epic +- **Blocked by:** R4 + +**Exit criteria:** +- The full experiment matrix has been run. +- Latency and VRAM have been profiled. +- The public technical report can be reproduced from `research/experiments/`. + +**Sub-issues:** +- [ ] Detail the sub-issues of this epic. +- [ ] Experiment config format and matrix runner. +- [ ] Latency and VRAM profiler per pipeline stage. +- [ ] Run the matrix and archive the results with provenance. +- [ ] Write and publish the technical report.