Skip to content

feat(index): add vector index option surface and validation#19310

Open
chrevanthreddy wants to merge 1 commit into
apache:masterfrom
chrevanthreddy:rfc-109-pr01-ddl-options
Open

feat(index): add vector index option surface and validation#19310
chrevanthreddy wants to merge 1 commit into
apache:masterfrom
chrevanthreddy:rfc-109-pr01-ddl-options

Conversation

@chrevanthreddy

Copy link
Copy Markdown
Contributor

Describe the issue this Pull Request addresses

Resolves #19096 (umbrella: #19094, RFC-109)

First foundational slice of RFC-109 (Hudi Native Vector Index): the option surface and validation for vector indexes. This PR is intentionally scoped to the self-contained option/config layer so it compiles and tests standalone; the DDL command dispatch, MDT schema/payload, and bootstrap execution land in dependent follow-up PRs (#19097, #19099, …) where their partition-type and executor dependencies are introduced.

Summary and Changelog

  • Add org.apache.hudi.common.index.vector.VectorIndexOptions — the full OPTIONS(...) key surface for CREATE INDEX ... USING VECTOR (dimension, metric, algorithm, quantizer, cluster/probe/refine counts, RaBitQ bits/seed/storage/posting layout, search-time flags) with typed accessors, defaults, and early validation.
  • Add VectorIndexType — routing algorithm enum (IVFFLAT, IVFPQ_HNSW) with case/separator-insensitive parsing.
  • Add VectorDistanceMetricCOSINE / L2 / DOT_PRODUCT distance enum (min-heap-consistent distances) with parsing.
  • Add PARTITION_NAME_VECTOR_INDEX / PARTITION_NAME_VECTOR_INDEX_PREFIX constants to HoodieTableMetadataUtil.
  • Add TestVectorIndexOptions — 9 unit tests covering defaults, metric/algorithm parsing, and early-failure on invalid configs (missing dimension, invalid metric, out-of-range RaBitQ bits).

Impact

No behavioral impact on existing tables or code paths. Adds new, unreferenced option/enum classes plus two constants. No schema, writer, reader, or DDL execution changes in this PR.

Risk Level

none

New self-contained classes + two constants + unit tests; nothing wires into existing runtime paths yet.

Documentation Update

Option semantics are documented as Javadoc on each option key. The user-facing RFC document is tracked separately (see #19309). No site docs change required until the feature is user-invokable.

Contributor's checklist

@github-actions github-actions Bot added the size:L PR with lines of changes in (300, 1000] label Jul 16, 2026
@hudi-bot

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L PR with lines of changes in (300, 1000]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add vector index options and Spark DDL scaffolding

2 participants