[backend:feature] Client-side model-aware tokenization for Nova embeddings MCDB-98680 #closes

[KarishS2]

Summary

For Nova models the SDK chunks long text itself, because tiktoken is the wrong tokenizer
for them. It had no way to know how a model tokenizes or how big its window is, so it
split on a 24,000-character guess.

This adds a registry of that per-model information: which tokenizer, the real context
window, and which platforms accept token IDs. Registered models are tokenized with their
own tokenizer, split on the real limit, and sent as token IDs. Supporting a new model
later is one entry.

Entries are added by hand after checking against a live deployment, because wrong token
IDs return a normal-looking vector, not an error. Today: Qwen/Qwen3-Embedding-0.6B at
32,768 tokens, where a 119,000-character document drops from 5 pieces to 1 (Test Plan).

Azure and Bedrock unchanged. Unregistered Nova models keep the 24,000-character split and
now warn.

Test Plan

• pytest singlestoredb/tests/test_embeddings.py — 14 unit tests. They cover which
  model gets which path, the special tokens we detect, the chunk size math, the
  platform check, and each way the code falls back.
• singlestoredb/tests/test_embeddings_live.py — tests that need a real deployment, so
  they are marked management and CI skips them. Sending token IDs gives the same
  vector as sending text (cosine > 0.9999). Two control tests confirm the server really
  is reading Qwen token IDs: tiktoken IDs give a different vector, and so does dropping
  the special tokens.
• Ran against the prod shared-qwen3-embed-0-6b deployment (NovaMultiTenant,
  Qwen/Qwen3-Embedding-0.6B) from a Nova notebook.

Original MCDB-98680 repro, versus a local sentence_transformers reference:

	this PR	local Qwen
paraphrase	0.8305	0.8297
same_intent	0.7517	0.7524
unrelated	0.3662	0.3664
unrelated2	0.3625	0.3667

Long input, 118,923 characters with one distinctive fact buried in the middle.
vs truth is cosine against the whole document embedded in a single chunk.
query match is cosine between a query asking for that fact and the document vector,
i.e. how well vector search would rank it.

	chunks	vs truth	query match
character path (unchanged)	5	0.9771	0.3927
token path (this PR)	1	0.9999	0.4366

TODO: single-tenant Nova has no parity run and is allowed by inference from
NovaMultiTenant serving the same image. The token path also requires transformers
and tokenizer files at runtime; if either is unavailable it warns and falls back to
current behavior.

Deployment Plan

• Customer impact: Yes. Registered Nova models change from raw text to token IDs.
  Azure and Bedrock paths are untouched.
• Rollout Plan: SDK release, then update helios entry for SDK release
• Rollback Plan: Revert the PR and release a new SDK version
• Rollback Tested: Yes. The fallback path is the current production behavior

Subscribers

JIRA Issues

MCDB-98680

[Copilot]

Pull request overview

This PR enhances singlestoredb.ai.embeddings to support model-aware, client-side tokenization for Nova-hosted embedding models by introducing a per-model tokenization policy registry and a token-based chunking path that can send model-native token IDs when supported.

Changes:

• Added a per-model policy registry (_MODEL_POLICIES) with context-window sizing and token-ID allowlisting, plus tokenizer loading and special-token affix derivation.
• Extended _ChunkedOpenAIEmbeddings to support token-based chunking (token IDs on the wire) and to weight-reduce chunk embeddings by content-token count (excluding special-token affixes).
• Expanded unit coverage and added a new “live server-contract” test suite (pytest management-marked) to validate token-ID parity against a real deployment.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
singlestoredb/ai/embeddings.py	Adds model policy registry, tokenizer loading/inspection, token chunker, and token-ID sending path for Nova models.
singlestoredb/tests/test_embeddings.py	Adds offline unit tests for registry behavior, affix derivation, chunk sizing, weighting, and fallback warnings via injected fake tokenizers/modules.
singlestoredb/tests/test_embeddings_live.py	Introduces live parity tests (management-marked) to validate server interpretation of token IDs and special-token requirements.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.