Skip to content

Add local VLM second-opinion / consensus stage (cross-check cloud LLM against offline model) #55

Description

@Liohtml

Part of #51. Depends on #18 (local VLM provider).

Summary

Once a local, offline vision-language model provider exists (#18), add an
optional stage that runs it in parallel with the primary cloud provider
(Claude/GPT/Gemini) and produces a consensus / second-opinion comparison:
where both agree, confidence is reinforced; where they diverge, the report flags
it for the radiologist. This complements the heuristic cross-validation in #52
with a second independent model.

Why (vision)

Two independent models disagreeing on a finding is exactly the signal a reviewing
radiologist most wants surfaced. A cloud frontier model and a medical-domain
local model have different failure modes; comparing them is cheap defense in
depth. It also gives users who cannot send data to the cloud (consent gate /
offline, see #27) a fully local path that still benefits from cross-checking.

Recommended model (license-verified — corrects #18)

  • Lingshu-7BMIT license (code + weights), explicitly supports MRI
    among 12+ modalities. This is the recommended default for Add local LLaVA-Med vision model provider #18 instead of
    LLaVA-Med, whose weights are research-only ("any deployed use case —
    commercial or otherwise — is out of scope") and therefore unsuitable as a
    default in an Apache-2.0 tool. Alternatives: HuatuoGPT-Vision-7B-Qwen2.5VL
    (Apache-2.0). See comment on Add local LLaVA-Med vision model provider #18.
  • Runs behind the optional local-models extra (torch + transformers).

Where

  • Builds on the LocalLLMProvider from Add local LLaVA-Med vision model provider #18 (currently a stub in
    src/medcheck/llm/local.py).
  • New optional step src/medcheck/pipeline/consensus.py
    ConsensusStep(PipelineStep), OR a flag on vision_analysis to run a second
    registered provider and diff the results.
  • Extend PipelineContext with second_opinion: AnalysisResult | None and a
    consensus: list[...] comparison.
  • Report renderers show an "Independent model comparison" section (agreement per
    structure, divergences highlighted).

Design constraints

Acceptance criteria

  • Depends on Add local LLaVA-Med vision model provider #18 landing a working local provider (Lingshu-7B recommended).
  • Optional ConsensusStep (or vision_analysis flag) compares two providers'
    structured findings and records agreements/divergences.
  • Report includes an independent-comparison section when a second opinion exists.
  • Honors the consent gate; supports a fully-offline consensus configuration.
  • Tests with two mocked providers (agreeing case, diverging case); no real
    model download in CI.

Effort

~1 day on top of #18.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions