[Feature]: BM25 Lexical + Dense Hybrid Fusion Retriever

[shivamm-gupta]

Problem

In production RAG systems, dense vector retrieval alone suffers from well-known blind spots:

1. Exact keyword matches & identifiers: Part numbers, SKUs (e.g. SKU-892-XZ), error codes (ConnectionResetError 104), IDs, and version tags often perform poorly with dense embeddings because embeddings prioritize semantic generalization over exact token matches.
2. Rare named entities & acronyms: Unseen or niche domain-specific abbreviations and proper nouns often get diluted in high-dimensional embedding spaces.

Industry-standard vector databases (Pinecone, Qdrant, Weaviate) address this via Hybrid Search, combining dense semantic vectors with sparse BM25 lexical scores.

While Dynavec has Reciprocal Rank Fusion (reciprocal_rank_fusion) and Learned RRF Weights (RRFWeightFitter), it currently only fuses dense embeddings with reformulations (Multi-Query/HyDE) or Knowledge Graph traversals. There is currently no native sparse/BM25 retriever in Dynavec, forcing users to either run a dedicated OpenSearch cluster or accept lower retrieval recall for keyword-specific queries.

Proposed solution

Implement a native, lightweight BM25Retriever (and hybrid search wrapper BM25HybridRetriever) that computes BM25 scores over document text and fuses them with S3 Vectors dense search using Dynavec's existing RRF:

1. BM25Retriever

• Dependency-light BM25 scoring algorithm (Okapi BM25) computed over the corpus text stored in DynamoDB or an in-memory inverted index.
• Tokenization with standard stopword filtering and term-frequency inverse-document-frequency (TF-IDF) statistics.
• Exposes standard .search(query, top_k) returning list[SearchResult] with lexical scores.

2. BM25HybridRetriever / db.hybrid_search(...)

• Executes parallel retrieval:
  1. Dense ANN vector search via S3 Vectors (query_vectors)
  2. Sparse lexical search via BM25Retriever
• Fuses ranks using reciprocal_rank_fusion with configurable weights (e.g., dense_weight=1.0, sparse_weight=0.8).
• Compatible with RRFWeightFitter for learning optimal dense vs. sparse weighting on labeled queries.

from dynavec.retrievers import BM25HybridRetriever

retriever = BM25HybridRetriever(
    db.namespace("products"),
    dense_weight=1.0,
    sparse_weight=0.8,
    top_k=5,
)

# Retrieves exact SKU matches via BM25 AND conceptual matches via S3 Vectors
results = retriever.search("replacement charger for model XPS-13-9310")

3. Ergonomics on Dynavec & NamespaceView

• Add convenience method db.as_bm25_retriever() and ns.as_bm25_retriever().
• Add db.hybrid_search(query, top_k=...) method on the main client.

Alternatives considered

• Amazon OpenSearch Serverless / Elasticsearch: Defeats Dynavec's zero-idle-compute cost model by requiring costly 24/7 OCUs ($700+/mo).
• Relying only on dense vectors: Leaves customers vulnerable to vocabulary mismatch on exact code, SKU, or entity lookups.

Additional context

• Can be implemented with zero mandatory third-party dependencies (using standard library or optional extras).
• 100% offline unit-testable against in-memory corpus fixtures.
• I am happy to take this up and implement it with full test coverage, benchmark comparisons, and documentation!

[shivamm-gupta]

Hi @codeforstartups Please let me know your thoughts on this , Once you provide your insights and approval i will start working on this