Problem
When using VLM analysis (e.g. Google Gemini), ignore regions defined in the
UI are not respected. This is because the original unmasked baseline image is
sent to the VLM, rather than a version with ignore regions applied.
As a result, the VLM flags differences in areas that have been explicitly
marked as ignore regions, leading to false positives.
Expected Behavior
When sending images to the VLM for analysis, the baseline image should have
ignore regions masked/blacked out before being sent. This way the VLM will
not analyze or flag differences in those areas.
Suggested Fix
Before sending the baseline image to the VLM:
- Retrieve the ignore regions for the corresponding TestVariation
- Apply the ignore region masks to the baseline image (e.g. fill with black
or grey overlay)
- Send the masked baseline image to the VLM instead of the original
Current Behavior
- Ignore regions are applied at pixel diff level only
- VLM receives the original unmasked baseline image
- VLM flags changes in ignored areas as regressions
Environment
- VRT API version: 5.1.1
- VRT UI version: 5.1.2
- VLM: Google Gemini
Problem
When using VLM analysis (e.g. Google Gemini), ignore regions defined in the
UI are not respected. This is because the original unmasked baseline image is
sent to the VLM, rather than a version with ignore regions applied.
As a result, the VLM flags differences in areas that have been explicitly
marked as ignore regions, leading to false positives.
Expected Behavior
When sending images to the VLM for analysis, the baseline image should have
ignore regions masked/blacked out before being sent. This way the VLM will
not analyze or flag differences in those areas.
Suggested Fix
Before sending the baseline image to the VLM:
or grey overlay)
Current Behavior
Environment