Allow image snippet checking

[cmungall]

Just as we provide for checking of extracted text against an abstract/full text, we could also allow for "quoting" a cropped image, and allow for validation that it is an "excerpt".

This is obviously a bit more fraught than text

• would the data model use hex encoded images (ugh) or something like html url references plus a checksum?
• what libraries to use [these would be optional dependencies]

Another approach that would require no real changes to LRV would be to use a trusted 3rd party service that provides annotations of images (which checksums guaranteeing proof of work etc); LRV just treats the textual annotation as any other external reference. Could be coupled with the legend somehow. This is obviously bringing nondeterminism into the mix, but it's a controlled, external determinism.

[github-actions]

Some notes to help frame the design space for this, based on how the rest of LRV currently works (I didn't find any other open issue that overlaps this one — closed #12 "Create generic URL checking" is the closest precedent, since it establishes the pattern of validating a reference by URL rather than by fetched full text).

Tension with the core design principle

The README is explicit that LRV's value proposition is deterministic substring matching of supporting_text against fetched full text/abstract (src/linkml_reference_validator/validation/supporting_text_validator.py, with fuzzy-repair fallback in fuzzy_text_utils.py / repairer.py). Both routes floated in the issue trade away some of that determinism:

• Third-party annotation service — as you note, this imports nondeterminism. It's arguably fine if it's scoped as a new, distinct capability rather than folded into the existing ValidationSeverity.ERROR semantics — e.g. surfaced only as INFO/WARNING with the annotation service's identity + a content hash recorded in the result, so a human can audit why it passed rather than trusting it blindly. This keeps LRV itself deterministic (same inputs → same LRV output) even though the upstream annotation isn't.
• Embedding image data in the record (hex/base64) — I'd push back on this even harder than text: LinkML data files are typically YAML/JSON, and base64-blob-in-YAML is unpleasant to diff, review in a PR, or store in git at any scale. If image data has to travel with the record at all, base64 is not it.

A third option that stays closer to LRV's existing model

There's a middle path that preserves determinism and mirrors how text validation already works: instead of "quoting" a crop by embedding pixels, treat the crop as a claim that region R of source figure F is what's shown, and verify that claim deterministically:

• Reference = (PMID/PMCID/DOI, figure_id_or_url) — for PMC-derived papers you likely already have a path to the figure list via the OA/figures endpoints, reusable from the existing etl/fulltext/pmc.py / epmc_preprint.py providers.
• Excerpt = either a bounding box [x, y, w, h] (resolution-normalized) into that source figure, or a cropped image + checksum.
• Validation = deterministic image containment check: template/feature matching (OpenCV matchTemplate or ORB/SIFT keypoint matching) to confirm the crop is genuinely a sub-region of the cited figure, with a perceptual hash (imagehash, pHash/dHash) to tolerate lossless re-encoding/recompression the way fuzzy_text_utils.py tolerates whitespace/unicode noise in text. No AI judgment call involved — it's the image analogue of "is this substring in that string," just with a similarity threshold instead of exact match.
• This would need optional deps (opencv-python-headless, Pillow, imagehash) — pyproject.toml doesn't currently define an extras group, so this would be a good opportunity to introduce linkml-reference-validator[images] rather than making these hard dependencies for all users.

Data model note

Whatever the excerpt representation, a checksum field (algorithm-tagged, e.g. sha256:...) on the crop is worth having regardless of which validation strategy is used — it's cheap, it lets you detect "this crop changed since it was checked" independent of re-running image matching, and it composes with the URL+checksum approach you floated.

Suggested scoping

Given the open questions you've flagged, this reads like it needs a design decision before implementation (not yet at the "write tests first" stage per the repo's TDD convention). If it's helpful, a reasonable phase 1 would be just the deterministic bounding-box/checksum containment check against a PMC-sourced figure (no third-party service, no new severity semantics) — that's a self-contained, testable slice that doesn't foreclose adding the annotation-service route later as an opt-in, lower-trust check.