Skip to content

Fix(SAM2): mask post-processing with Transformers 5.x - #9555

Open
hlista wants to merge 5 commits into
invoke-ai:mainfrom
hlista:segm2-remove-reshaped-input-sizes-fix
Open

Fix(SAM2): mask post-processing with Transformers 5.x#9555
hlista wants to merge 5 commits into
invoke-ai:mainfrom
hlista:segm2-remove-reshaped-input-sizes-fix

Conversation

@hlista

@hlista hlista commented Aug 28, 2026

Copy link
Copy Markdown

Summary

Fixes SAM2 segmentation when using the current Transformers SAM2 processor.

Sam2Processor no longer returns reshaped_input_sizes in its processed inputs, but the InvokeAI SAM2 pipeline attempts to access that field during mask post-processing

Removing the obsolete argument allows SAM2 mask post-processing to complete successfully.

Processing an image with the SAM2 processor:

processed_inputs = processor(images=image, return_tensors="pt")
print(processed_inputs.keys())

returns:

pixel_values
original_sizes

There is no reshaped_input_sizes

Related Issues / Discussions

Closes #9551

QA Instructions

Tested locally with:

  • InvokeAI 6.14.0
  • Transformers 5.5.4
  • facebook/sam2.1-hiera-large

Before the change, SAM2 segmentation consistently failed with
KeyError: 'reshaped_input_sizes'

After the change, SAM2 segmentation completes successfully

Merge Plan

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • ❗Changes to a redux slice have a corresponding migration
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

@github-actions github-actions Bot added python PRs that change python files backend PRs that change backend files labels Aug 28, 2026
@Pfannkuchensack Pfannkuchensack self-assigned this Aug 28, 2026
@Pfannkuchensack

Copy link
Copy Markdown
Member

Findings

No correctness defect found in the change itself. The one-line removal is verified correct against the pinned dependency. Two coverage/accuracy issues remain.

  • Medium (test gap): invokeai/backend/image_util/segment_anything/segment_anything_2_pipeline.py:101-104 — this PR fixes a third-party contract drift (Sam2Processor dropped reshaped_input_sizes from its output) and adds no test. There is no test anywhere covering either SAM pipeline: tests/backend/image_util/ contains only test_color_conversion.py and test_vendor_mutable_defaults.py, and a repo-wide search for *segment* / *sam* test files returns nothing. The identical construct still lives in the SAM1 path at invokeai/backend/image_util/segment_anything/segment_anything_pipeline.py:92, so the next upstream transformers bump that drops reshaped_input_sizes from SamProcessor reproduces issue [bug]: Segment Anything V2 fails every time #9551 verbatim with zero CI signal. The failure mode is a runtime AttributeError inside an invocation, i.e. a user-visible generation failure, not an import-time or type error, so neither lint nor typecheck catches it.
    To expose this issue, add a test that runs SegmentAnything2Pipeline.segment() against a randomly initialized Sam2Model(Sam2Config()) plus a real Sam2Processor(image_processor=Sam2ImageProcessor()) and asserts the returned tensor is torch.bool with shape [num_masks, 3, original_height, original_width]. This needs no downloaded weights and runs in seconds. The same test shape should be added for SegmentAnythingPipeline so the still-present reshaped_input_sizes access in the SAM1 path is pinned by a test rather than by luck.

  • Low (stale, now actively misleading comment): invokeai/backend/image_util/segment_anything/segment_anything_2_pipeline.py:6 — the comment reads # Import SAM2 components - these should be available in transformers 4.56.0+. After this change the code depends on the 5.x processor output contract (no reshaped_input_sizes), and pyproject.toml:67 pins transformers>=5.5,<5.6. The comment advertises support for a version range the patched code no longer targets. Relatedly, the constructor docstring at invokeai/backend/image_util/segment_anything/segment_anything_2_pipeline.py:22 claims the processor "can be Sam2Processor or Sam2VideoProcessor", but the type annotation is Sam2Processor and invokeai/app/invocations/segment_anything.py:127 only ever constructs Sam2Processor. Both should be corrected while the file is open; this is a documentation-accuracy issue, not a runtime defect.

  • Ruff formats is needed.

@github-actions github-actions Bot added the python-tests PRs that change python tests label Aug 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend PRs that change backend files python PRs that change python files python-tests PRs that change python tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug]: Segment Anything V2 fails every time

2 participants