docs(rfc): RFC-104 native vector search support in Apache Hudi#19291
docs(rfc): RFC-104 native vector search support in Apache Hudi#19291chrevanthreddy wants to merge 1 commit into
Conversation
Propose native approximate nearest-neighbor (ANN) vector search in Hudi, stored in the Metadata Table. The base table stays the source of truth; the MDT holds routing, pruning, and approximate-scoring metadata; final ranking reads exact base-table vectors. Core storage innovation: the posting block -- ~1-4K vectors packed into one MDT record, columnar (structure-of-arrays), keyed 0x10|gen|cluster|shard|blockId so one IVF cluster is one contiguous prefix-scannable range. ~1000x fewer MDT records, single-range cluster scans, pay-per-promise column access. Combines IVF routing + multibit RaBitQ (unbiased estimator with per-vector error bound for safe pruning) + two-pass scan + RLI freshness arbitration + positional exact rerank + LIRE maintenance + generation lifecycle for blue/green rebuilds. Docs-only RFC PR (adds rfc-104.md + diagrams, README entry). Tracks umbrella issue apache#19094; §11 maps sections to child issues apache#19095-apache#19105. Resolves apache#19095
hudi-agent
left a comment
hudi-agent
left a comment
There was a problem hiding this comment.
Thanks for the RFC! This proposes native IVF + RaBitQ approximate nearest-neighbor vector search stored in the Metadata Table, with a column-packed "posting block" as the core storage innovation and exact re-ranking from the base table. The motivation and algorithm are clearly written, but a few design areas would benefit from more detail before implementation — notably multi-writer/concurrency behavior, the feasibility of column-selective reads within a single MDT record, positional exact fetch under MOR/clustering, the RLI dependency, and a couple of missing sections (alternatives, migration). Details are in the inline comments; once addressed, a Hudi committer or PMC member can take it from here for a deeper design review.
| ### 5.2 Incremental inserts and vector updates | ||
|
|
||
| For an inserted row or a vector-changing update, the metadata writer's vector hook computes | ||
| the nearest centroid, cluster id, shard id, RaBitQ code, optional scalar, and base-table |
There was a problem hiding this comment.
🤖 The write path describes appending posting deltas to a cluster's key range each commit, but I don't see how this behaves under multiple concurrent writers. How do two writers appending deltas into the same cluster range interact with MDT's OCC / commit model, and can there be lost updates or ordering issues in delta supersession? @nsivabalan could you weigh in on whether this is safe under multi-writer?
| P|<gen>|... -> posting blocks + deltas | ||
| ``` | ||
|
|
||
| A query uses one generation consistently. Publishing a generation is atomic from a reader's |
There was a problem hiding this comment.
🤖 "write the generation's rows, commit them to MDT, then flip manifest" — is the flip a single atomic MDT commit distinct from the generation-row writes, and what does a reader see if it loads the manifest between those commits? It might be worth spelling out how snapshot pinning guarantees a reader never sees a half-published generation.
| needs: | ||
|
|
||
| ```text | ||
| POSTING BLOCK (~512 KB target, one MDT record) |
There was a problem hiding this comment.
🤖 The "pay only for promise" column access assumes a query can read S1/S3 without materializing S2/S4/S6, but a posting block is a single MDT record (one HFile cell value of ~512KB). Wouldn't reading any part of the record deserialize the whole value from storage? Could you clarify how column-selective reads avoid pulling the full block, and whether MDT/HFile handles ~512KB values well at compaction time? @yihua this touches MDT storage assumptions.
|
|
||
| ### 1.1 Goals | ||
|
|
||
| 1. Keep authoritative vector values in the base table (`ARRAY<FLOAT>` / `VECTOR(D)` column). |
There was a problem hiding this comment.
🤖 The doc uses ARRAY<FLOAT> and VECTOR(D) somewhat interchangeably. Is VECTOR(D) a new logical type being introduced, or just an alias/annotation over ARRAY<FLOAT>? If it's new, that has schema-evolution and cross-engine (Flink/Spark/Avro) implications worth calling out. @bvaradar for the public-type/API angle.
| Cost is proportional to promise, not to table size. Delta records encountered in the same | ||
| scan are scored identically and supersede matching packed entries. | ||
|
|
||
| ### 6.3 Freshness arbitration and exact re-rank |
There was a problem hiding this comment.
🤖 Freshness arbitration and delete handling both depend on the Record Level Index. Is RLI a hard prerequisite for this feature, and what happens if it's not enabled? It'd also help to quantify the per-query cost of batched RLI point lookups for the finalist set, since that's on the critical query path.
| Posting locators are treated as *hints*. The finalists' record keys are validated against the | ||
| Record Level Index in batched point lookups on the same pinned snapshot: | ||
|
|
||
| - moved file group → re-resolve location by key; |
There was a problem hiding this comment.
🤖 The exact re-rank fetches candidate rows "by position" / page-level positional reads. Positional access isn't generally stable in Hudi — row positions shift under compaction, clustering, and MOR log merges. How are locators kept valid, and does this fall back to key-based lookups (via RLI) when positions are stale? A bit more detail on the positional reader would help.
|
|
||
| --- | ||
|
|
||
| ## Table of Contents |
There was a problem hiding this comment.
🤖 The RFC justifies IVF-over-HNSW and RaBitQ-over-PQ inline, but there's no dedicated Alternatives Considered section, and no explicit Migration/Upgrade section (MDT version bump, enabling the feature on an existing table, downgrade/rollback for readers on older versions). Could you add these? They're standard for Hudi RFCs that touch MDT.
| routing, pruning, and approximate-scoring metadata; final ranking always reads exact vectors | ||
| from the base table. | ||
|
|
||
| Prototype measurements on a **1-billion-row, 128-dimensional table** show exact-reranked |
There was a problem hiding this comment.
🤖 The 0.985 recall@10 / low-single-digit-second prototype numbers are compelling — could you add the benchmark setup (cluster size, D=128 dataset, refineFactor, num_clusters, whether RLI arbitration and positional fetch were included in the measured latency)? That makes the claim reproducible and clarifies what's baked into the number.
chrevanthreddy
changed the title
3 participants
Describe the issue this Pull Request addresses
Proposes RFC-104: native approximate nearest-neighbor (ANN) vector search for Apache Hudi, with the index stored in the Metadata Table (MDT). Tracks umbrella issue #19094.
Closes #19095
Summary and Changelog
Docs-only PR that adds the RFC document, three diagrams, and the
rfc/README.mdentry. No code changes.The RFC proposes vector search as a table capability: the base table keeps authoritative vectors, the MDT holds routing/pruning/approximate-scoring metadata, and final ranking reads exact base-table vectors.
Core storage innovation — the posting block: instead of one MDT record per indexed vector, ~1–4K vectors are packed into a single MDT record laid out column-wise (structure-of-arrays), keyed
0x10|generation|cluster|shard|blockIdso one IVF cluster is one contiguous, prefix-scannable range. This yields ~1000× fewer MDT records, single-range cluster scans, and pay-per-promise column access. §4.6 contains a small worked example (D=4, 3 vectors).Approach: IVF routing + multibit RaBitQ (unbiased estimator with a per-vector error bound for safe pruning) + a two-pass scan (sign-plane prune → multibit refine) + RLI-arbitrated freshness + positional exact rerank + LIRE maintenance + a generation lifecycle for zero-downtime rebuilds. §11 maps RFC sections to the child implementation issues #19096–#19105.
Changes:
rfc/rfc-104/rfc-104.mdrfc/rfc-104/diagrams/01-architecture-overview.svg,02-read-path.svg,03-write-maintenance.svgrfc/README.mdImpact
None (documentation only). No public API, behavior, or performance change. Establishes the design contract for the subsequent implementation PRs (#19096–#19105).
Risk Level
none
Documentation Update
none (this PR is itself the RFC design document). User-facing docs for the feature will accompany the implementation PRs.
Contributor's checklist