Skip to content

VLM analysis does not respect ignore regions - baseline image sent without ignore area masking #348

@sumeetp26

Description

@sumeetp26

Problem

When using VLM analysis (e.g. Google Gemini), ignore regions defined in the
UI are not respected. This is because the original unmasked baseline image is
sent to the VLM, rather than a version with ignore regions applied.

As a result, the VLM flags differences in areas that have been explicitly
marked as ignore regions, leading to false positives.

Expected Behavior

When sending images to the VLM for analysis, the baseline image should have
ignore regions masked/blacked out before being sent. This way the VLM will
not analyze or flag differences in those areas.

Suggested Fix

Before sending the baseline image to the VLM:

  1. Retrieve the ignore regions for the corresponding TestVariation
  2. Apply the ignore region masks to the baseline image (e.g. fill with black
    or grey overlay)
  3. Send the masked baseline image to the VLM instead of the original

Current Behavior

  • Ignore regions are applied at pixel diff level only
  • VLM receives the original unmasked baseline image
  • VLM flags changes in ignored areas as regressions

Environment

  • VRT API version: 5.1.1
  • VRT UI version: 5.1.2
  • VLM: Google Gemini

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions