Repository navigation
docs(adr): ADR-6: generation path is a deterministic pipeline; orchestration is eval-gated - #412
Merged
Merged
Conversation
…tration is eval-gated The code carries two answers to "who decides what gets retrieved". The vocabulary says agent (a "ReAct-style" aggregator, a planner that emits tools and hops, a multi-round tool loop); the wiring says pipeline (the planner's tools field is never read, multi-hop sub-questions run in parallel, and api_routes.py:346 disables tool calling whenever retrieval returned anything). ADR-1 ended the same kind of split in the deployment layer. ADR-5 records the pipeline as the product and licenses a bounded re-query as an experiment, off by default, built after P2-3 and shipped only if it beats the P2-3 baseline on multi-hop questions by a margin fixed before the run. Both alternatives were argued at full strength and rejected on the same test: each acts before the evidence exists. Deleting the planner and aggregator now would break the DEL-2 rule (defer, do not delete, when the measurement cannot yet be made); building a harness now has no evidence of lift, runs on the models ADR-4 leaves us, and its strongest stated motive is the one ADR-1's revisit clause excludes. A second revisit trigger (2026-12-31 if P2-3 has not started) keeps "let the eval decide" from becoming indefinite deferral. Found while writing it and listed first among the consequences: get_rag_context_multi_hop drops source_ids and additional_workspace_ids, so on the default path a source-narrowed chat asking a multi-hop question searches the whole workspace. Workspace isolation holds. No code changes in this commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CuUHpys6UBronLzBdtAkS
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
ADR-5 contains a stale evaluation result and contradicts the default model-driven retrieval fallback.
Review effort: Balanced
Findings: 2
Open (2)
What changed in this PR
Adds ADR-5 to establish deterministic retrieval as the product architecture and gate agentic orchestration on answer-level evaluation.
Changes:
- Documents the decision, alternatives, consequences, and revisit criteria.
- Adds ADR-5 to the repository file map.
| File | Description |
|---|---|
docs/ADR.md |
Adds ADR-5. |
.claude/rules/file-map.md |
Registers ADR-5 in the documentation index. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+288
to
+289
| exists, and the one comparable measurement, DEL-2's GraphRAG run, saw clever retrieval | ||
| fire on 3 of 20 questions. ADR-4 confines inference to local or EU OpenAI-compatible |
Comment on lines
+311
to
+314
| **What this rules out:** | ||
| - The model choosing tools or retrieval targets on the default chat path. | ||
| - Unbounded tool rounds. Any experimental loop has a hard round cap and a per-request token | ||
| budget, both enforced in code rather than by `TOOL_MAX_ROUNDS` alone. |
|
|
||
| ## ADR-5: The generation path is a deterministic pipeline; agentic orchestration is an experiment until an answer-level eval says otherwise | ||
|
|
||
| **Proposed 2026-10-01.** Supersedes nothing. Records a choice the code has been making |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jwvanderstam
changed the title
|
|
||
| --- | ||
|
|
||
| ## ADR-6: The generation path is a deterministic pipeline; agentic orchestration is an experiment until an answer-level eval says otherwise |
ADR-5 (row-level security, #404) sits before ADR-6; the file-map row lists both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds ADR-5 to
docs/ADR.mdand its row in.claude/rules/file-map.md. Documentation only; no code changes.Why
The code carries two answers to who decides what gets retrieved. The vocabulary says agent; the wiring says pipeline (planner
toolsnever read, multi-hop sub-questions retrieved in parallel,api_routes.py:346disabling tool calling whenever retrieval returned anything). ADR-5 records the pipeline as the product and licenses a bounded re-query as an experiment, gated on P2-3.Both alternatives are argued at full strength in the record and rejected on the same test: each acts before the evidence exists (the DEL-2 rule).
Owner decisions before merging
ADR-5:rather thanADR-5 —. Normalise either way.Found while writing it
get_rag_context_multi_hop(src/services/chat.py) takes neithersource_idsnoradditional_workspace_ids. On the default path (planner on, RAG on, query of 7 or more words, plan judged multi-hop) a source-narrowed chat searches the whole workspace, and extra workspaces are silently dropped. Workspace isolation holds. Listed first among the ADR's consequences; deserves its own fix PR rather than waiting on this one.🤖 Generated with Claude Code
https://claude.ai/code/session_018CuUHpys6UBronLzBdtAkS
Generated by Claude Code