feat(rag): preserve claim scope in the default prompt, add bounded applicability filter - #176
Open
n0nuser wants to merge 4 commits into
Open
feat(rag): preserve claim scope in the default prompt, add bounded applicability filter#176n0nuser wants to merge 4 commits into
n0nuser wants to merge 4 commits into
Conversation
build_prompt read only source and chunk_index off each context and never touched context["metadata"], so heading_path reached the API sources list but never the model. The structural chunker computes heading_path precisely so a chunk carries its position in the document hierarchy; discarding it at the prompt boundary threw away the signal that distinguishes a passage about long-term risk from one about immediate effects. Each context block header now carries a section= segment when heading_path is populated, and is unchanged when it is not, so text_block and code_block chunks keep the previous format. Heading paths are bounded by MAX_SECTION_CHARS. The bound is derived rather than picked: compression budgets measure only chunk body text and reserve nothing for the block scaffolding, so worst-case headers have to fit inside reserved_prompt_tokens. A test pins that invariant, since a looser bound silently pushes the prompt past what any budget counted. Refs #172 Claude-Session: https://claude.ai/code/session_01PhFVTFLaxhmP8icwzmcm5Q
The default prompt constrained the model to the retrieved context but said nothing about how faithfully to represent that context's scope. A passage about habitual exposure measured across years and one about a single occurrence are both just context, so the model flattened them together and presented long-term epidemiological findings as the result of one event — grounded in real retrieved text, with the scope silently changed. Two imperative sentences now require preserving the scope each source states, and require saying so when the question asks about a single instance the context cannot support. Length is part of the contract: this ships on every query and small local models follow long instructions unevenly, so a test bounds it rather than leaving the constraint implicit. The string had five hand-maintained copies with no automated consistency check — the settings default, both cloud provider keyword defaults, .env.example, and prose in the docs. Every in-code copy now derives from DEFAULT_SYSTEM_PROMPT in localrag/rag/prompt.py, and tests pin both the provider defaults and .env.example against it so the drift cannot return. Refs #173 Claude-Session: https://claude.ai/code/session_01PhFVTFLaxhmP8icwzmcm5Q
The pipeline was retrieve -> concatenate -> generate, with nothing between retrieval and generation asking whether a retrieved claim applies at the scope the question asks about. Retrieval ranks by topical similarity, which is blind to the qualifier that decides which passage actually answers the question: a question about a single occurrence and a passage about habitual exposure over years are the same topic, so both rank and both reach the model. CLAIM_FILTER_ENABLED (default false) inserts one bounded provider call that asks which numbered passages do not apply at the question's scope, and removes those. Removal of an already-retrieved context is the stage's only power: it never rewrites, adds, or reorders a passage, and never contributes answer text. The filter prompt carries heading_path, because scope usually lives in the heading rather than the sentence. Filtering runs before compression so the budget is spent only on applicable passages, and reported sources come from the filtered set, since citing a discarded passage would misattribute the answer. Every failure degrades to the unfiltered contexts: provider error, unparseable output, out-of-range indices, and any verdict that would discard all context. Answering with more context than necessary is the behavior that shipped before this stage; answering with too little because one judgment call went wrong is a regression. Prompt construction sits outside that guard on purpose, so a bug in this module surfaces instead of masquerading as degradation. Unlike HyDE this is not Ollama-only: it calls generate_from_prompt, which is on the BaseLLMProvider contract, so all three backends work without a guard. Refs #173. Contract: ADR 041. Claude-Session: https://claude.ai/code/session_01PhFVTFLaxhmP8icwzmcm5Q
A manual check against gemma3:4b on the motivating acute-vs-chronic example produced a wrong verdict: the model marked both passages inapplicable. The all-discarded guard caught it and degraded, so the answer path was unharmed, but the stage did no useful work. Rewording the instruction to bias harder toward keeping made it worse — the model then discarded exactly the acute passage that answered the question — so that reword was reverted rather than kept. Recording this because shipping an optional stage without saying what it did when actually run would be the more misleading choice. The safety design holds; the judge model is the limiting factor, and #174 is what makes it measurable. Claude-Session: https://claude.ai/code/session_01PhFVTFLaxhmP8icwzmcm5Q
Owner
Author
|
Merge order (this PR is stacked). Base is Commit 1 ( |
This was referenced Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #173
Two independent commits, reviewable separately.
Commit 1 — scope-preserving default prompt
The default constrained the model to the context but said nothing about how faithfully to represent that context's scope. A passage about habitual exposure across years and one about a single occurrence are both just context, so the model flattened them together.
New default adds two imperatives: preserve the scope each source states, and say so when the question asks about one instance the context only supports long-term. Length is part of the contract — this ships on every query and small local models follow long instructions unevenly — so a test bounds it.
The string had five hand-maintained copies with no consistency check (settings default, both cloud provider keyword defaults,
.env.example, docs prose). All in-code copies now derive fromDEFAULT_SYSTEM_PROMPT, with tests pinning the provider defaults and.env.exampleagainst it.Commit 2 — bounded claim-applicability filter (ADR 041)
This is the "reasoning pipeline" gap: nothing between retrieval and generation asked whether a retrieved claim applies at the question's scope.
CLAIM_FILTER_ENABLED(default false) makes one provider call — regardless of context count — asking which passages don't apply. Removal of an already-retrieved context is its only power: no rewriting, adding, reordering, or answer text.heading_pathinto the filter prompt — judging scope without the heading is the blind case RAG prompt drops heading_path, so the model cannot see where a chunk came from #172 describes.generate_from_promptis on theBaseLLMProvidercontract, so all three backends work with no guard.What happened when I actually ran it
Worth reading before merging commit 2.
Against
gemma3:4bon the motivating example (one chronic passage, one acute), the model marked both inapplicable. The all-discarded guard caught it and degraded — answer path unharmed, but the stage did nothing useful.Rewording the instruction to bias harder toward keeping made it worse: the model then discarded exactly the acute passage that answers the question. I reverted that reword rather than keeping it.
So the safety design holds, but the judge model is the limiting factor and a 4B local model does not clear the bar. Recorded in ADR 041 and flagged in
docs/rag-retrieval.mdrather than left for someone to discover. Whether this wants a dedicatedCLAIM_FILTER_MODELis the first thing to test once #174 provides a regression case.If you'd rather not ship an unproven stage at all, commit 2 can be dropped and commit 1 merged alone — they're independent.
Verification
CLAIM_FILTER_*→ flat → grouped →with_overrides.None.https://claude.ai/code/session_01PhFVTFLaxhmP8icwzmcm5Q