Repository navigation
fix: use vision_size for Gemma4 specializations to support chunked embedding in vLLM v1 (#1135) - #1146
Merged
Rishin Raj (quic-rishinr) merged 1 commit intoJul 6, 2026
Conversation
Rishin Raj (quic-rishinr)
approved these changes
Jul 3, 2026
Contributor
Author
|
Tested with vLLM flow for the following models: server_logs_gemma-4-26B-A4B-it_text_vision Corresponding logs files have been attached to jira. |
…bedding in vLLM v1 (qualcomm#1135) ## Summary To keep Gemma4 consistent with the other VLMs, use `vision_size` instead of `mm_tokens_per_image` for the number of vision tokens when building specializations. This allows chunked embedding in vLLM v1. Follows the same pattern introduced in PR qualcomm#1059 (quic-sanising) for gemma3, llava, internvl, qwen2_5_vl, etc. ## Changes - Pop `vision_size` from `compiler_options` (same as all other VLMs) - If user provides `vision_size`, validate it (`vision_size < ctx_len`) and use it as the vision token count in specializations - Otherwise fall back to `mm_tokens_per_image` (default 256 for Gemma4) - Replace `mm_tokens_per_image` with `vision_size` in `build_lang_prefill_spec` and `build_lang_decode_spec` ## Testing Tested with vLLM flow for the following models: - google/gemma-4-E2B-it & google/gemma-4-E4B-it → 2 QPC (Dense) - google/gemma-4-26B-A4B-it → 3 QPC (MoE) Signed-off-by: Gokkulnath Thirukonda Srinivasan <gthiruko@qti.qualcomm.com> --------- Signed-off-by: Gokkulnath Thirukonda Srinivasan <gthiruko@qti.qualcomm.com>
Rishin Raj (quic-rishinr)
force-pushed
the
cherry-pick/vision-size-gemma4-v1.22.0
branch
from
July 6, 2026 05:03
ae674c3 to
436763b
Compare
Rishin Raj (quic-rishinr)
merged commit Jul 6, 2026
5b7fa7a
into
qualcomm:release/v1.22.0
5 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of commit 1f16d42 (PR #1135) from main onto release/v1.22.0.
Summary
To keep Gemma4 consistent with the other VLMs, use
vision_sizeinstead ofmm_tokens_per_imagefor the number of vision tokens when building specializations. This allows chunked embedding in vLLM v1.Follows the same pattern introduced in PR #1059 (quic-sanising) for gemma3, llava, internvl, qwen2_5_vl, etc.
Changes
vision_sizefromcompiler_options(same as all other VLMs)vision_size, validate it (vision_size < ctx_len) and use it as the vision token count in specializationsmm_tokens_per_image(default 256 for Gemma4)mm_tokens_per_imagewithvision_sizeinbuild_lang_prefill_specandbuild_lang_decode_specTesting
Tested with vLLM flow for the following models: