Repository navigation
Add user_vision_size in VLM's get_specializations for chunked embedding in vLLM v1 - #996
Merged
Rishin Raj (quic-rishinr) merged 6 commits intoJun 2, 2026
Conversation
… in vLLM v1 Signed-off-by: quic-xiyushi <xiyushi@qti.qualcomm.com>
Xiyue Shi (quic-xiyushi)
marked this pull request as draft
May 18, 2026 18:53
Xiyue Shi (quic-xiyushi)
marked this pull request as ready for review
May 22, 2026 15:22
Rishin Raj (quic-rishinr)
requested a review
from Dhiraj Kumar Sah (quic-dhirajku)
May 25, 2026 10:05
Rishin Raj (quic-rishinr)
requested changes
May 27, 2026
| mm_tokens_per_image = getattr(self.config, "mm_tokens_per_image", 256) | ||
| user_vision_size = compiler_options.pop("vision_size", None) | ||
| if user_vision_size: | ||
| assert user_vision_size < ctx_len, "vision_size must be less than ctx_len" |
Contributor
There was a problem hiding this comment.
nit: raise an exception instead of assert in all modelling changes
Contributor
There was a problem hiding this comment.
Done for all models
| assert user_vision_size < ctx_len, "vision_size must be less than ctx_len" | ||
| vision_size = user_vision_size | ||
| else: | ||
| vision_size = getattr(self.config, "mm_tokens_per_image", 256) |
Contributor
There was a problem hiding this comment.
Add a warning mm_tokens_per_image will be deprecated in the next release. Having two input arguments for a single input introduces ambiguity.
Contributor
There was a problem hiding this comment.
Done
| f"vision_size={user_vision_size}. Please adjust the image resolution or " | ||
| "increase the vision_size." | ||
| ) | ||
| if vision_size * f >= user_vision_size: |
Contributor
There was a problem hiding this comment.
nit: Do we want the warning to be raised even if vision_size * f == user_vision_size? I think we should raise the warning only when the user_vision_size is strictly less than the calculated size.
Contributor
There was a problem hiding this comment.
Yes, thanks for pointing it out. Fixed the bug for Qwen2.5-VL, Qwen3-VL, and Qwen3-VL MoE.
…ion_size Signed-off-by: sanising <sanising@qti.qualcomm.com>
Signed-off-by: sanising <sanising@qti.qualcomm.com>
Signed-off-by: sanising <sanising@qti.qualcomm.com>
Signed-off-by: sanising <sanising@qti.qualcomm.com>
Rishin Raj (quic-rishinr)
requested a review
from Dhiraj Kumar Sah (quic-dhirajku)
June 2, 2026 09:26
Rishin Raj (quic-rishinr)
approved these changes
Jun 2, 2026
Rishin Raj (quic-rishinr)
left a comment
Rishin Raj (quic-rishinr)
left a comment
Contributor
There was a problem hiding this comment.
LGTM!! Approved
Dhiraj Kumar Sah (quic-dhirajku)
approved these changes
Jun 2, 2026
Dhiraj Kumar Sah (quic-dhirajku)
left a comment
Dhiraj Kumar Sah (quic-dhirajku)
left a comment
Contributor
There was a problem hiding this comment.
LGTM
Rishin Raj (quic-rishinr)
merged commit Jun 2, 2026
a2266a7
into
qualcomm:release/v1.21.6
4 checks passed
Rishin Raj (quic-rishinr)
pushed a commit
to quic-sanising/efficient-transformers
that referenced
this pull request
Jun 10, 2026
…ng in vLLM v1 (qualcomm#996) vLLM v1 introduces chunked prefill with encoder cache, where the model runner processes requests in scheduled token windows. For each window: - Overlap detection determines which multimodal embeddings are required - Only the relevant sub-tensors are gathered and used From a QEfficient perspective, this is analogous to: - Identifying the image indices for the current chunk, and - Gathering the corresponding vision embeddings accordingly Additionally, v1 eliminates the strict prefill/decode distinction, allowing: - Prefill chunks of one request to be interleaved with decode steps of other requests However, if we reuse the full vision embedding for every prefill chunk, which is what currently QEfficient supports, this: - Breaks the intended vLLM v1 design - Introduces unnecessary overhead due to repeated large set_buffer calls This PR proposes adding user_vision_size to get_specializations for VLM models, with the following scope: ✅ Apply to all current VLM models ❌ Exclude mllama due to it's cross attention ❌ Exclude molmo (not yet supported in vLLM on QAIC, not sure about how the team is going to support it) For models onboarded by QEfficient in the future, it should also follow the same way to enable user_vision_size. This change enables: ✅ In vLLM v1, align vision embedding size with prefill sequence length and enable efficient chunked prefill for multimodal inputs ✅ Better alignment with vLLM v1 scheduling model ✅ Reduced overhead from repeated large buffer updates ✅ Easier support for multi-resolution input and multiple images per request 🚫 Not intended for 1.21 release ✅ Target release: 1.22 This draft PR is currently based on the 1.21.6 branch because multi-resolution and multi-frame support for Qwen2.5-VL / Qwen3-VL is only available in that branch, and these features are prerequisites for this change Once multi-resolution and multi-frame support are available in the QEfficient main branch, this PR will be rebased and migrated to the main branch. --------- Signed-off-by: quic-xiyushi <xiyushi@qti.qualcomm.com> Signed-off-by: sanising <sanising@qti.qualcomm.com> Co-authored-by: sanising <sanising@qti.qualcomm.com>
Rishin Raj (quic-rishinr)
pushed a commit
that referenced
this pull request
Jun 10, 2026
…ng in vLLM v1 (#1059) Same as described in #996 > ### Background > vLLM v1 introduces chunked prefill with encoder cache, where the model runner processes requests in scheduled token windows. For each window: > > * Overlap detection determines which multimodal embeddings are required > * Only the relevant sub-tensors are gathered and used > > From a QEfficient perspective, this is analogous to: > > * Identifying the image indices for the current chunk, and > * Gathering the corresponding vision embeddings accordingly > > Additionally, v1 eliminates the strict prefill/decode distinction, allowing: > > * Prefill chunks of one request to be interleaved with decode steps of other requests > > However, if we reuse the full vision embedding for every prefill chunk, which is what currently QEfficient supports, this: > > * Breaks the intended vLLM v1 design > * Introduces unnecessary overhead due to repeated large set_buffer calls > > ### Proposal > This PR proposes adding user_vision_size to get_specializations for VLM models, with the following scope: > > ✅ Apply to all current VLM models ❌ Exclude mllama due to it's cross attention ❌ Exclude molmo (not yet supported in vLLM on QAIC, not sure about how the team is going to support it) > > For models onboarded by QEfficient in the future, it should also follow the same way to enable user_vision_size. > > ### Benefits > This change enables: > > ✅ In vLLM v1, align vision embedding size with prefill sequence length and enable efficient chunked prefill for multimodal inputs ✅ Better alignment with vLLM v1 scheduling model ✅ Reduced overhead from repeated large buffer updates ✅ Easier support for multi-resolution input and multiple images per request > > ### Release Plan > 🚫 Not intended for 1.21 release ✅ Target release: 1.22 > > This draft PR is currently based on the 1.21.6 branch because multi-resolution and multi-frame support for Qwen2.5-VL / Qwen3-VL is only available in that branch, and these features are prerequisites for this change > > ### Next Steps > Once multi-resolution and multi-frame support are available in the QEfficient main branch, this PR will be rebased and migrated to the main branch. --------- Signed-off-by: quic-xiyushi <xiyushi@qti.qualcomm.com> Signed-off-by: sanising <sanising@qti.qualcomm.com> Co-authored-by: quic-xiyushi <xiyushi@qti.qualcomm.com>
Rishin Raj (quic-rishinr)
pushed a commit
that referenced
this pull request
Jun 10, 2026
…ng in vLLM v1 (#1059) Same as described in #996 > ### Background > vLLM v1 introduces chunked prefill with encoder cache, where the model runner processes requests in scheduled token windows. For each window: > > * Overlap detection determines which multimodal embeddings are required > * Only the relevant sub-tensors are gathered and used > > From a QEfficient perspective, this is analogous to: > > * Identifying the image indices for the current chunk, and > * Gathering the corresponding vision embeddings accordingly > > Additionally, v1 eliminates the strict prefill/decode distinction, allowing: > > * Prefill chunks of one request to be interleaved with decode steps of other requests > > However, if we reuse the full vision embedding for every prefill chunk, which is what currently QEfficient supports, this: > > * Breaks the intended vLLM v1 design > * Introduces unnecessary overhead due to repeated large set_buffer calls > > ### Proposal > This PR proposes adding user_vision_size to get_specializations for VLM models, with the following scope: > > ✅ Apply to all current VLM models ❌ Exclude mllama due to it's cross attention ❌ Exclude molmo (not yet supported in vLLM on QAIC, not sure about how the team is going to support it) > > For models onboarded by QEfficient in the future, it should also follow the same way to enable user_vision_size. > > ### Benefits > This change enables: > > ✅ In vLLM v1, align vision embedding size with prefill sequence length and enable efficient chunked prefill for multimodal inputs ✅ Better alignment with vLLM v1 scheduling model ✅ Reduced overhead from repeated large buffer updates ✅ Easier support for multi-resolution input and multiple images per request > > ### Release Plan > 🚫 Not intended for 1.21 release ✅ Target release: 1.22 > > This draft PR is currently based on the 1.21.6 branch because multi-resolution and multi-frame support for Qwen2.5-VL / Qwen3-VL is only available in that branch, and these features are prerequisites for this change > > ### Next Steps > Once multi-resolution and multi-frame support are available in the QEfficient main branch, this PR will be rebased and migrated to the main branch. --------- Signed-off-by: quic-xiyushi <xiyushi@qti.qualcomm.com> Signed-off-by: sanising <sanising@qti.qualcomm.com> Co-authored-by: quic-xiyushi <xiyushi@qti.qualcomm.com> Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Rishin Raj (quic-rishinr)
pushed a commit
that referenced
this pull request
Jun 10, 2026
…ng in vLLM v1 (#1059) Same as described in #996 > ### Background > vLLM v1 introduces chunked prefill with encoder cache, where the model runner processes requests in scheduled token windows. For each window: > > * Overlap detection determines which multimodal embeddings are required > * Only the relevant sub-tensors are gathered and used > > From a QEfficient perspective, this is analogous to: > > * Identifying the image indices for the current chunk, and > * Gathering the corresponding vision embeddings accordingly > > Additionally, v1 eliminates the strict prefill/decode distinction, allowing: > > * Prefill chunks of one request to be interleaved with decode steps of other requests > > However, if we reuse the full vision embedding for every prefill chunk, which is what currently QEfficient supports, this: > > * Breaks the intended vLLM v1 design > * Introduces unnecessary overhead due to repeated large set_buffer calls > > ### Proposal > This PR proposes adding user_vision_size to get_specializations for VLM models, with the following scope: > > ✅ Apply to all current VLM models ❌ Exclude mllama due to it's cross attention ❌ Exclude molmo (not yet supported in vLLM on QAIC, not sure about how the team is going to support it) > > For models onboarded by QEfficient in the future, it should also follow the same way to enable user_vision_size. > > ### Benefits > This change enables: > > ✅ In vLLM v1, align vision embedding size with prefill sequence length and enable efficient chunked prefill for multimodal inputs ✅ Better alignment with vLLM v1 scheduling model ✅ Reduced overhead from repeated large buffer updates ✅ Easier support for multi-resolution input and multiple images per request > > ### Release Plan > 🚫 Not intended for 1.21 release ✅ Target release: 1.22 > > This draft PR is currently based on the 1.21.6 branch because multi-resolution and multi-frame support for Qwen2.5-VL / Qwen3-VL is only available in that branch, and these features are prerequisites for this change > > ### Next Steps > Once multi-resolution and multi-frame support are available in the QEfficient main branch, this PR will be rebased and migrated to the main branch. --------- Signed-off-by: quic-xiyushi <xiyushi@qti.qualcomm.com> Signed-off-by: sanising <sanising@qti.qualcomm.com> Co-authored-by: quic-xiyushi <xiyushi@qti.qualcomm.com> Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Rishin Raj (quic-rishinr)
pushed a commit
that referenced
this pull request
Jun 10, 2026
…ng in vLLM v1 (#1059) Same as described in #996 > ### Background > vLLM v1 introduces chunked prefill with encoder cache, where the model runner processes requests in scheduled token windows. For each window: > > * Overlap detection determines which multimodal embeddings are required > * Only the relevant sub-tensors are gathered and used > > From a QEfficient perspective, this is analogous to: > > * Identifying the image indices for the current chunk, and > * Gathering the corresponding vision embeddings accordingly > > Additionally, v1 eliminates the strict prefill/decode distinction, allowing: > > * Prefill chunks of one request to be interleaved with decode steps of other requests > > However, if we reuse the full vision embedding for every prefill chunk, which is what currently QEfficient supports, this: > > * Breaks the intended vLLM v1 design > * Introduces unnecessary overhead due to repeated large set_buffer calls > > ### Proposal > This PR proposes adding user_vision_size to get_specializations for VLM models, with the following scope: > > ✅ Apply to all current VLM models ❌ Exclude mllama due to it's cross attention ❌ Exclude molmo (not yet supported in vLLM on QAIC, not sure about how the team is going to support it) > > For models onboarded by QEfficient in the future, it should also follow the same way to enable user_vision_size. > > ### Benefits > This change enables: > > ✅ In vLLM v1, align vision embedding size with prefill sequence length and enable efficient chunked prefill for multimodal inputs ✅ Better alignment with vLLM v1 scheduling model ✅ Reduced overhead from repeated large buffer updates ✅ Easier support for multi-resolution input and multiple images per request > > ### Release Plan > 🚫 Not intended for 1.21 release ✅ Target release: 1.22 > > This draft PR is currently based on the 1.21.6 branch because multi-resolution and multi-frame support for Qwen2.5-VL / Qwen3-VL is only available in that branch, and these features are prerequisites for this change > > ### Next Steps > Once multi-resolution and multi-frame support are available in the QEfficient main branch, this PR will be rebased and migrated to the main branch. --------- Signed-off-by: quic-xiyushi <xiyushi@qti.qualcomm.com> Signed-off-by: sanising <sanising@qti.qualcomm.com> Co-authored-by: quic-xiyushi <xiyushi@qti.qualcomm.com> Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
vLLM v1 introduces chunked prefill with encoder cache, where the model runner processes requests in scheduled token windows. For each window:
Overlap detection determines which multimodal embeddings are required
Only the relevant sub-tensors are gathered and used
From a QEfficient perspective, this is analogous to:
Additionally, v1 eliminates the strict prefill/decode distinction, allowing:
However, if we reuse the full vision embedding for every prefill chunk, which is what currently QEfficient supports, this:
Proposal
This PR proposes adding user_vision_size to get_specializations for VLM models, with the following scope:
✅ Apply to all current VLM models
❌ Exclude mllama due to it's cross attention
❌ Exclude molmo (not yet supported in vLLM on QAIC, not sure about how the team is going to support it)
For models onboarded by QEfficient in the future, it should also follow the same way to enable user_vision_size.
Benefits
This change enables:
✅ In vLLM v1, align vision embedding size with prefill sequence length and enable efficient chunked prefill for multimodal inputs
✅ Better alignment with vLLM v1 scheduling model
✅ Reduced overhead from repeated large buffer updates
✅ Easier support for multi-resolution input and multiple images per request
Release Plan
🚫 Not intended for 1.21 release
✅ Target release: 1.22
This draft PR is currently based on the 1.21.6 branch because multi-resolution and multi-frame support for Qwen2.5-VL / Qwen3-VL is only available in that branch, and these features are prerequisites for this change
Next Steps
Once multi-resolution and multi-frame support are available in the QEfficient main branch, this PR will be rebased and migrated to the main branch.