Add user_vision_size in VLM's get_specializations for chunked embedding in vLLM v1 by quic-sanising · Pull Request #1059 · qualcomm/efficient-transformers

Sanidhya Singal (quic-sanising) · 2026-06-10T01:12:25Z

Same as described in #996

Background

vLLM v1 introduces chunked prefill with encoder cache, where the model runner processes requests in scheduled token windows. For each window:

Overlap detection determines which multimodal embeddings are required

Only the relevant sub-tensors are gathered and used

From a QEfficient perspective, this is analogous to:

Identifying the image indices for the current chunk, and

Gathering the corresponding vision embeddings accordingly

Additionally, v1 eliminates the strict prefill/decode distinction, allowing:

Prefill chunks of one request to be interleaved with decode steps of other requests

However, if we reuse the full vision embedding for every prefill chunk, which is what currently QEfficient supports, this:

Breaks the intended vLLM v1 design

Introduces unnecessary overhead due to repeated large set_buffer calls

Proposal

This PR proposes adding user_vision_size to get_specializations for VLM models, with the following scope:

✅ Apply to all current VLM models ❌ Exclude mllama due to it's cross attention ❌ Exclude molmo (not yet supported in vLLM on QAIC, not sure about how the team is going to support it)

For models onboarded by QEfficient in the future, it should also follow the same way to enable user_vision_size.

Benefits

This change enables:

✅ In vLLM v1, align vision embedding size with prefill sequence length and enable efficient chunked prefill for multimodal inputs ✅ Better alignment with vLLM v1 scheduling model ✅ Reduced overhead from repeated large buffer updates ✅ Easier support for multi-resolution input and multiple images per request

Release Plan

🚫 Not intended for 1.21 release ✅ Target release: 1.22

This draft PR is currently based on the 1.21.6 branch because multi-resolution and multi-frame support for Qwen2.5-VL / Qwen3-VL is only available in that branch, and these features are prerequisites for this change

Next Steps

Once multi-resolution and multi-frame support are available in the QEfficient main branch, this PR will be rebased and migrated to the main branch.

…ng in vLLM v1 (qualcomm#996) vLLM v1 introduces chunked prefill with encoder cache, where the model runner processes requests in scheduled token windows. For each window: - Overlap detection determines which multimodal embeddings are required - Only the relevant sub-tensors are gathered and used From a QEfficient perspective, this is analogous to: - Identifying the image indices for the current chunk, and - Gathering the corresponding vision embeddings accordingly Additionally, v1 eliminates the strict prefill/decode distinction, allowing: - Prefill chunks of one request to be interleaved with decode steps of other requests However, if we reuse the full vision embedding for every prefill chunk, which is what currently QEfficient supports, this: - Breaks the intended vLLM v1 design - Introduces unnecessary overhead due to repeated large set_buffer calls This PR proposes adding user_vision_size to get_specializations for VLM models, with the following scope: ✅ Apply to all current VLM models ❌ Exclude mllama due to it's cross attention ❌ Exclude molmo (not yet supported in vLLM on QAIC, not sure about how the team is going to support it) For models onboarded by QEfficient in the future, it should also follow the same way to enable user_vision_size. This change enables: ✅ In vLLM v1, align vision embedding size with prefill sequence length and enable efficient chunked prefill for multimodal inputs ✅ Better alignment with vLLM v1 scheduling model ✅ Reduced overhead from repeated large buffer updates ✅ Easier support for multi-resolution input and multiple images per request 🚫 Not intended for 1.21 release ✅ Target release: 1.22 This draft PR is currently based on the 1.21.6 branch because multi-resolution and multi-frame support for Qwen2.5-VL / Qwen3-VL is only available in that branch, and these features are prerequisites for this change Once multi-resolution and multi-frame support are available in the QEfficient main branch, this PR will be rebased and migrated to the main branch. --------- Signed-off-by: quic-xiyushi <xiyushi@qti.qualcomm.com> Signed-off-by: sanising <sanising@qti.qualcomm.com> Co-authored-by: sanising <sanising@qti.qualcomm.com>

…ng in vLLM v1 Signed-off-by: sanising <sanising@qti.qualcomm.com>

Signed-off-by: sanising <sanising@qti.qualcomm.com>

…ng in vLLM v1 (#1059) Same as described in #996 > ### Background > vLLM v1 introduces chunked prefill with encoder cache, where the model runner processes requests in scheduled token windows. For each window: > > * Overlap detection determines which multimodal embeddings are required > * Only the relevant sub-tensors are gathered and used > > From a QEfficient perspective, this is analogous to: > > * Identifying the image indices for the current chunk, and > * Gathering the corresponding vision embeddings accordingly > > Additionally, v1 eliminates the strict prefill/decode distinction, allowing: > > * Prefill chunks of one request to be interleaved with decode steps of other requests > > However, if we reuse the full vision embedding for every prefill chunk, which is what currently QEfficient supports, this: > > * Breaks the intended vLLM v1 design > * Introduces unnecessary overhead due to repeated large set_buffer calls > > ### Proposal > This PR proposes adding user_vision_size to get_specializations for VLM models, with the following scope: > > ✅ Apply to all current VLM models ❌ Exclude mllama due to it's cross attention ❌ Exclude molmo (not yet supported in vLLM on QAIC, not sure about how the team is going to support it) > > For models onboarded by QEfficient in the future, it should also follow the same way to enable user_vision_size. > > ### Benefits > This change enables: > > ✅ In vLLM v1, align vision embedding size with prefill sequence length and enable efficient chunked prefill for multimodal inputs ✅ Better alignment with vLLM v1 scheduling model ✅ Reduced overhead from repeated large buffer updates ✅ Easier support for multi-resolution input and multiple images per request > > ### Release Plan > 🚫 Not intended for 1.21 release ✅ Target release: 1.22 > > This draft PR is currently based on the 1.21.6 branch because multi-resolution and multi-frame support for Qwen2.5-VL / Qwen3-VL is only available in that branch, and these features are prerequisites for this change > > ### Next Steps > Once multi-resolution and multi-frame support are available in the QEfficient main branch, this PR will be rebased and migrated to the main branch. --------- Signed-off-by: quic-xiyushi <xiyushi@qti.qualcomm.com> Signed-off-by: sanising <sanising@qti.qualcomm.com> Co-authored-by: quic-xiyushi <xiyushi@qti.qualcomm.com> Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>

…upport chunked embedding Instead of simply popping vision_size from compiler_options and ignoring it, use vision_size when provided (following the same pattern as gemma3, llava, internvl, qwen2_5_vl, etc. from PR qualcomm#1059) to allow chunked embedding in vLLM v1. - Pop vision_size from compiler_options - If user provides vision_size, validate it (must be < ctx_len) and use it as the vision token count in specializations - Otherwise fall back to mm_tokens_per_image (default 256 for Gemma4) - Replace mm_tokens_per_image with vision_size in build_lang_prefill_spec and build_lang_decode_spec - Remove unused logger import (no longer needed) Signed-off-by: Gokkulnath Thirukonda Srinivasan <gthiruko@qti.qualcomm.com>

…bedding in vLLM v1 (#1135) ## Summary To keep Gemma4 consistent with the other VLMs, use `vision_size` instead of `mm_tokens_per_image` for the number of vision tokens when building specializations. This allows chunked embedding in vLLM v1. Follows the same pattern introduced in PR #1059 (quic-sanising) for gemma3, llava, internvl, qwen2_5_vl, etc. ## Changes - Pop `vision_size` from `compiler_options` (same as all other VLMs) - If user provides `vision_size`, validate it (`vision_size < ctx_len`) and use it as the vision token count in specializations - Otherwise fall back to `mm_tokens_per_image` (default 256 for Gemma4) - Replace `mm_tokens_per_image` with `vision_size` in `build_lang_prefill_spec` and `build_lang_decode_spec` ## Testing Tested with vLLM flow for the following models: - google/gemma-4-E2B-it & google/gemma-4-E4B-it → 2 QPC (Dense) - google/gemma-4-26B-A4B-it → 3 QPC (MoE) Signed-off-by: Gokkulnath Thirukonda Srinivasan <gthiruko@qti.qualcomm.com> --------- Signed-off-by: Gokkulnath Thirukonda Srinivasan <gthiruko@qti.qualcomm.com>

…bedding in vLLM v1 (qualcomm#1135) ## Summary To keep Gemma4 consistent with the other VLMs, use `vision_size` instead of `mm_tokens_per_image` for the number of vision tokens when building specializations. This allows chunked embedding in vLLM v1. Follows the same pattern introduced in PR qualcomm#1059 (quic-sanising) for gemma3, llava, internvl, qwen2_5_vl, etc. ## Changes - Pop `vision_size` from `compiler_options` (same as all other VLMs) - If user provides `vision_size`, validate it (`vision_size < ctx_len`) and use it as the vision token count in specializations - Otherwise fall back to `mm_tokens_per_image` (default 256 for Gemma4) - Replace `mm_tokens_per_image` with `vision_size` in `build_lang_prefill_spec` and `build_lang_decode_spec` ## Testing Tested with vLLM flow for the following models: - google/gemma-4-E2B-it & google/gemma-4-E4B-it → 2 QPC (Dense) - google/gemma-4-26B-A4B-it → 3 QPC (MoE) Signed-off-by: Gokkulnath Thirukonda Srinivasan <gthiruko@qti.qualcomm.com> --------- Signed-off-by: Gokkulnath Thirukonda Srinivasan <gthiruko@qti.qualcomm.com>

…bedding in vLLM v1 (#1135) (#1146) Cherry-pick of commit [1f16d42](1f16d42) (PR #1135) from main onto release/v1.22.0. ## Summary To keep Gemma4 consistent with the other VLMs, use `vision_size` instead of `mm_tokens_per_image` for the number of vision tokens when building specializations. This allows chunked embedding in vLLM v1. Follows the same pattern introduced in PR #1059 (quic-sanising) for gemma3, llava, internvl, qwen2_5_vl, etc. ## Changes - Pop `vision_size` from `compiler_options` (same as all other VLMs) - If user provides `vision_size`, validate it (`vision_size < ctx_len`) and use it as the vision token count in specializations - Otherwise fall back to `mm_tokens_per_image` (default 256 for Gemma4) - Replace `mm_tokens_per_image` with `vision_size` in `build_lang_prefill_spec` and `build_lang_decode_spec` ## Testing Tested with vLLM flow for the following models: - google/gemma-4-E2B-it & google/gemma-4-E4B-it → 2 QPC (Dense) - google/gemma-4-26B-A4B-it → 3 QPC (MoE) --------- Signed-off-by: Gokkulnath Thirukonda Srinivasan <gthiruko@qti.qualcomm.com>

…bedding in vLLM v1 (qualcomm#1135) ## Summary To keep Gemma4 consistent with the other VLMs, use `vision_size` instead of `mm_tokens_per_image` for the number of vision tokens when building specializations. This allows chunked embedding in vLLM v1. Follows the same pattern introduced in PR qualcomm#1059 (quic-sanising) for gemma3, llava, internvl, qwen2_5_vl, etc. ## Changes - Pop `vision_size` from `compiler_options` (same as all other VLMs) - If user provides `vision_size`, validate it (`vision_size < ctx_len`) and use it as the vision token count in specializations - Otherwise fall back to `mm_tokens_per_image` (default 256 for Gemma4) - Replace `mm_tokens_per_image` with `vision_size` in `build_lang_prefill_spec` and `build_lang_decode_spec` ## Testing Tested with vLLM flow for the following models: - google/gemma-4-E2B-it & google/gemma-4-E4B-it → 2 QPC (Dense) - google/gemma-4-26B-A4B-it → 3 QPC (MoE) Signed-off-by: Gokkulnath Thirukonda Srinivasan <gthiruko@qti.qualcomm.com> --------- Signed-off-by: Gokkulnath Thirukonda Srinivasan <gthiruko@qti.qualcomm.com>

Rishin Raj (quic-rishinr) added the 1.22 Release 1.22 candidate label Jun 10, 2026

Xiyue Shi (quic-xiyushi) and others added 3 commits June 10, 2026 17:17

Add user_vision_size in VLM's get_specializations for chunked embeddi…

2c6bab1

…ng in vLLM v1 Signed-off-by: sanising <sanising@qti.qualcomm.com>

Fix typos

e968d6c

Signed-off-by: sanising <sanising@qti.qualcomm.com>

Rishin Raj (quic-rishinr) force-pushed the user_vision_sizes branch from 4480271 to e968d6c Compare June 10, 2026 11:47

Rishin Raj (quic-rishinr) changed the base branch from release/v1.22.0_tmp to intermediate_branch June 10, 2026 11:48

Rishin Raj (quic-rishinr) merged commit 0fbe652 into qualcomm:intermediate_branch Jun 10, 2026
4 checks passed

quic-gthiruko mentioned this pull request Jun 29, 2026

fix: use vision_size for Gemma4 specializations to support chunked embedding in vLLM v1 #1135

Merged

quic-gthiruko mentioned this pull request Jul 3, 2026

fix: use vision_size for Gemma4 specializations to support chunked embedding in vLLM v1 (#1135) #1146

Merged

Navigation Menu

Repository navigation

Add user_vision_size in VLM's get_specializations for chunked embedding in vLLM v1 - #1059

Add user_vision_size in VLM's get_specializations for chunked embedding in vLLM v1#1059
Rishin Raj (quic-rishinr) merged 3 commits into
qualcomm:intermediate_branchfrom
quic-sanising:user_vision_sizes

Sanidhya Singal (quic-sanising) commented Jun 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Sanidhya Singal (quic-sanising) commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

Proposal

Benefits

Release Plan

Next Steps

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Sanidhya Singal (quic-sanising) commented Jun 10, 2026 •

edited

Loading