Fast State-of-the-Art Static Embeddings in Rust

Quickstart • Features • Models • Performance • Relation to Python Model2Vec

model2vec-rs is a Rust crate providing an efficient implementation for inference with Model2Vec static embedding models. Model2Vec is a technique for creating compact and fast static embedding models from sentence transformers, achieving significant reductions in model size and inference speed. This Rust crate is optimized for performance, making it suitable for applications requiring fast embedding generation.

Quickstart

You can utilize model2vec-rs in two ways:

1. As a library in your Rust projects
2. As a standalone Command-Line Interface (CLI) tool for quick terminal-based inferencing

---

1. Using model2vec-rs as a Library

Integrate model2vec-rs into your Rust application to generate embeddings within your code.

a. Add model2vec-rs as a dependency:

cargo add model2vec-rs

b. Load a model and generate embeddings:

use anyhow::Result;
use model2vec_rs::model::StaticModel;

fn main() -> Result<()> {
    // Load a model from the Hugging Face Hub or a local path.
    // Arguments: (repo_or_path, hf_token, normalize_embeddings, subfolder_in_repo)
    let model = StaticModel::from_pretrained(
        "minishlab/potion-base-8M", // Model ID from Hugging Face or local path to model directory
        None,                       // Optional: Hugging Face API token for private models
        None,                       // Optional: bool to override model's default normalization. `None` uses model's config.
        None                        // Optional: subfolder if model files are not at the root of the repo/path
    )?;

    let sentences = vec![
        "Hello world".to_string(),
        "Rust is awesome".to_string(),
    ];

    // Generate embeddings using default parameters
    // (Default max_length: Some(512), all sentences in one batch)
    let embeddings = model.encode(&sentences);
    // `embeddings` is a Vec<Vec<f32>>
    println!("Generated {} embeddings.", embeddings.len());

    // To generate embeddings with custom arguments:
    let custom_embeddings = model.encode_with_args(
        &sentences,
        Some(256), // Optional: custom max token length for truncation
        512,       // Custom batch size for processing
    );
    println!("Generated {} custom embeddings.", custom_embeddings.len());

    // `encode_flat` returns one row-major `Vec<f32>` of
    // `sentences.len() * model.dim()` values. Use it when you want contiguous
    // output or only read the rows; prefer `encode` if you need an owned
    // `Vec<f32>` per sentence.
    let flat = model.encode_flat(&sentences);
    for row in flat.chunks_exact(model.dim()) {
        println!("First value: {}", row[0]);
    }

    Ok(())
}

Alternative: Loading from in-memory bytes (wasm / embedded use cases):

When filesystem access is unavailable, use from_bytes instead of from_pretrained:

use model2vec_rs::model::StaticModel;

let model = StaticModel::from_bytes(
    include_bytes!("path/to/tokenizer.json").as_ref(),
    include_bytes!("path/to/model.safetensors").as_ref(),
    include_bytes!("path/to/config.json").as_ref(),
    None, // normalize: None reads the value from config.json
)?;

let embeddings = model.encode(&["Hello world".to_string()]);

Quantizing an f32 model to int8:

use model2vec_rs::model::StaticModel;

let model = StaticModel::from_pretrained("minishlab/potion-base-8M", None, None, None)?.quantize_int8();

quantize_int8 stores the embeddings as int8 with one scale per row, using a quarter of the memory and encoding faster. Outputs stay very close to the f32 model (cosine similarity above 0.9998 on the potion models).

See the feature flags section for wasm and local-only build configurations.

---

2. Using the model2vec-rs CLI

a. Install the CLI tool: This command compiles the crate in release mode (for speed) and installs the model2vec-rs executable to Cargo's binary directory ~/.cargo/bin/.

cargo install model2vec-rs --features cli

Ensure ~/.cargo/bin/ is in your system's PATH to run model2vec-rs from any directory. The CLI is opt-in; default library builds do not include Clap. From a checkout, use cargo run --release --features cli -- ....

b. Generate embeddings via CLI: The compiled binary installed via cargo install is significantly faster (often >10x) than running via cargo run -- ... without release mode.

• Encode a single sentence:

  model2vec-rs encode-single "Hello world" "minishlab/potion-base-8M"
  

  Embeddings will be printed to the console in JSON format. This command should take less than 0.1s to execute.

• Encode multiple lines from a file and save to an output file:

  echo -e "This is the first sentence.\nThis is another sentence." > my_texts.txt
  model2vec-rs encode my_texts.txt "minishlab/potion-base-8M" --output embeddings_output.json
  

Features

• Fast Inference: Optimized Rust implementation for fast embedding generation.
• Hugging Face Hub Integration: Load pre-trained Model2Vec models directly from the Hugging Face Hub using model IDs, or use models from local paths.
• Model Formats: Supports models with f32, f16, and i8 weight types stored in safetensors files. Int8 models (model2vec's quantize_to="int8") stay int8 in memory, and f32 models can be quantized to int8 at load time with quantize_int8.
• Flexible Loading: Load models from the Hub (from_pretrained), raw bytes (from_bytes), or owned/borrowed data (from_owned, from_borrowed, and from_owned_i8, from_borrowed_i8 for int8 embeddings) for embedded and WASM use cases.
• Batch Processing: Encodes multiple sentences in batches.
• Configurable Encoding: Allows customization of maximum sequence length and batch size during encoding.

Feature flags

The crate exposes a few feature combinations for different runtimes:

• default: native build with onig tokenization and optional Hugging Face Hub downloads
• cli: build the command-line executable with Clap (not enabled by default)
• fancy-regex: alternative tokenizer backend for native builds
• local-only: disable remote model downloads and restrict loading to local paths or from_bytes(...)
• wasm: minimal WebAssembly-oriented feature set for in-memory loading via from_bytes(...)

Typical invocations:

# native local-only build
cargo build --no-default-features --features onig,local-only

# wasm (requires getrandom backend config)
RUSTFLAGS='--cfg getrandom_backend="wasm_js"' \
cargo check --no-default-features --features wasm --target wasm32-unknown-unknown

What is Model2Vec?

Model2Vec is a technique to distill large sentence transformer models into highly efficient static embedding models. This process significantly reduces model size and computational requirements for inference. For a detailed understanding of how Model2Vec works, including the distillation process and model training, please refer to the main Model2Vec Python repository and its documentation.

This model2vec-rs crate provides a Rust-based engine specifically for inference using these Model2Vec models.

Models

A variety of pre-trained Model2Vec models are available on the HuggingFace Hub (MinishLab collection). These can be loaded by model2vec-rs using their Hugging Face model ID or by providing a local path to the model files.

Model	Language	Sentence Transformer	Params	Task
potion-base-32M	English	bge-base-en-v1.5	32.3M	General
potion-multilingual-128M	Multilingual	bge-m3	128M	General
potion-retrieval-32M	English	bge-base-en-v1.5	32.3M	Retrieval
potion-base-8M	English	bge-base-en-v1.5	7.5M	General
potion-code-16M-v2	Code	CodeRankEmbed	16.2M	Code

Performance

We compared model2vec-rs with the Python version of Model2Vec by encoding the same 666,808 texts with potion-base-8M in both.

Implementation	Single thread	All cores	Model load
Rust	575K texts/second	2.60M texts/second	24 ms
Python	38K texts/second	110K texts/second	436 ms
Speedup	15.2×	23.6×	18.2×

On a single thread, the Rust version is roughly 15× faster than the Python version, and about 24× faster when using all cores. With int8 models the speedup is larger:

To reproduce these numbers, run this from the repository root (it needs Rust and uv):

uv run scripts/python_compare.py --no-python-multiprocessing

The script downloads the corpus, builds benches/python_compare.rs, runs both implementations and prints the table above. For the int8 bars of the chart, add --quantize int8, then pass both results to scripts/speedup_chart.py. See scripts/python_compare.py for the options.

Benchmark details:

• Corpus: 28,721 random articles from English Wikipedia: shard 20231101.en/train-00017-of-00041.parquet of wikimedia/wikipedia, shuffled with seed 0 and taken until they reach 11M words, joined by blank lines (70.0 MB, SHA-256 b6535c025aca457f71ef010b7c4d06f54f564981cb6db5683437ccc8e1ae9710).
• Texts: each of the 666,808 non-empty lines (paragraphs and section headings, 16.5 words and 104 bytes on average) is one text. Every timed pass encodes all of them in a single encode call with the default max_length of 512.
• Hardware: Apple M3 Pro (12 cores), 18 GB RAM.
• Software: model2vec-rs 0.4.0 (release build, Rust 1.99.1); Python 3.14.3 with model2vec 0.10.0, tokenizers 0.23.3 and numpy 2.5.4.
• Single thread: RAYON_NUM_THREADS=1 and TOKENIZERS_PARALLELISM=false for both; Python with use_multiprocessing=False.
• All cores: Rust with Rayon parallelism; Python with tokenizer parallelism and use_multiprocessing=False. Python's default use_multiprocessing=True spreads the texts over a process pool, which makes it slower.
• Method: 5 alternating rounds of Rust and Python processes, 5 timed passes each after one warm-up pass; throughput is the median over the 25 passes.
• Model load: median time over all 10 process launches per side to load the model from the local Hugging Face cache; for Python this includes importing model2vec.
• Output: for float32, the Rust and Python embeddings differ by at most 3.0e-7 per value (minimum cosine similarity 0.9999998).
• int8: the model is quantized at load time on both sides, with quantize_int8 in Rust and quantize_to="int8" in Python. Rust keeps one scale per row and Python one scale for the whole table, so the int8 embeddings differ between the two (minimum cosine similarity 0.27); these bars compare speed only.

Relation to Python model2vec

• model2vec-rs (This Crate): High-performance Rust engine for up to 28x faster Model2Vec inference.
• model2vec (Python-based): Handles model distillation, training, fine-tuning, and slower Python-based inference.

License

MIT

Citing Model2Vec

If you use Model2Vec in your research, please cite the following:

@software{minishlab2024model2vec,
  author       = {Stephan Tulkens and {van Dongen}, Thomas},
  title        = {Model2Vec: Fast State-of-the-Art Static Embeddings},
  year         = {2024},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.17270888},
  url          = {https://github.com/MinishLab/model2vec},
  license      = {MIT}
}