Skip to content

feat: provenance — track source corroboration per entity + relationship #84

Description

@rajnavakoti

Part of #82 (calibrated honesty).

Problem

The over-fit errors ("Quarkus underlies Argo for Cleveron") come from the LLM asserting global claims from a single document's local context. Today there's no record of how many / which docs support a given entity or relationship — so "asserted from 1 doc, uncorroborated" is indistinguishable from "corroborated across 5 docs."

Goal

Track and expose provenance — the source documents supporting each entity and each relationship — as the raw material for confidence (#calibration) and a trust signal in its own right.

Acceptance Criteria

  • Each entity records the set of source documents it was derived from (entities already have source_documents; ensure it accumulates correctly across the dedup/merge step)
  • Each relationship records its source document(s) — a relationship seen in 1 doc vs many is distinguishable
  • A corroboration count (how many independent docs assert this) is available per entity + relationship
  • Single-source assertions are flaggable (uncorroborated / low-corroboration)
  • Tests: an entity/relationship mentioned in 3 docs shows corroboration 3; a single-doc one shows 1

Out of scope

  • Using provenance to compute confidence (that's the calibration issue)
  • Surfacing it to consumers (that's the retrieval/MCP propagation issue)

Metadata

Metadata

Assignees

No one assigned

    Labels

    frameworkScientific frameworks

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions