Skip to content

fix(onnx): export vocabulary-quantized models with their token mapping and weights - #364

Merged
stephantul merged 2 commits into
MinishLab:mainfrom
serhiizghama:fix/onnx-export-vocab-quantized-models
Sep 12, 2026
Merged

fix(onnx): export vocabulary-quantized models with their token mapping and weights#364
stephantul merged 2 commits into
MinishLab:mainfrom
serhiizghama:fix/onnx-export-vocab-quantized-models

Conversation

@serhiizghama

Copy link
Copy Markdown
Contributor

Exporting a vocabulary-quantized model to ONNX gives a graph that can't actually run. TorchStaticModel builds its embedding table from model.embedding, which for a quantized model has one row per cluster, and then gathers it with the raw token ids — so any id at or above the cluster count blows up at inference (Gather ... indices element out of data bounds from onnxruntime, IndexError from the torch module itself), and weights were never applied at all. The export itself succeeds because the dummy inputs are all zeros.

StaticModel._encode_helper looks ids up through token_mapping first and multiplies by weights; the exporter now does the same. Both are kept as buffers, so the exported graph stays cluster-sized instead of materializing a full vocab × dim table.

Reproduced with quantize_model(model, vocabulary_quantization=3) and with a classifier trained on such a model and exported via to_pipeline() — both go through the same module. Added a test for each; they fail on main with the out-of-bounds gather and match encode / predict_proba with the fix.

@codecov

codecov Bot commented Sep 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines Coverage Δ
model2vec/onnx.py 99.30% <100.00%> (+0.04%) ⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@stephantul

Copy link
Copy Markdown
Contributor

@serhiizghama thanks, very cool!

@stephantul
stephantul merged commit a8ce637 into MinishLab:main Sep 12, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants