feat(ingest): ingest unstructured files into BLOB+VECTOR Hudi tables via Hudi Streamer

[vinothchandar]

Describe the issue this Pull Request addresses

Landing unstructured data (documents, images, videos) in a lakehouse today means gluing Spark's binaryFile reader to hand-rolled parsing UDFs, with no incremental ingestion, no upsert semantics, no blob management, and no embedding lifecycle. With the BLOB and VECTOR types available since 1.2, HoodieStreamer can offer this as one command: point at a folder on cloud storage, get a queryable Hudi table with parsed text, chunks, and embedding vectors that stay current as files arrive or change.

Summary and Changelog

Two additive pieces in hudi-utilities, no changes to existing behavior:

UnstructuredFileDFSSource (RowSource)

• Reuses DFSPathSelector for modification-time-checkpointed incremental discovery: each sync ingests only new/changed files; record key = path with modification_time ordering makes re-ingested files upsert in place.
• Per-record blob placement by configurable size threshold (default 1 MiB): small files store bytes INLINE in the BLOB column; larger files store an OUT_OF_LINE reference to the original file in place. Large-file bytes never enter Spark rows, so memory and shuffle stay bounded regardless of file sizes (--source-limit already caps total inline bytes per batch).
• Embedded text extraction behind a pluggable DocumentParser; default TikaDocumentParser (Apache Tika, in-process, no services). Parsing never fails the job: per-row parse_status/parse_error record SUCCESS/TRUNCATED/EMPTY/SKIPPED/FAILED. Extracted text is chunked (size/overlap configurable) into a nested chunks column.
• Only tika-core (~800 KB) ships in the utilities bundles; format parser modules (PDF, Office, ...) are supplied at runtime (e.g. spark-submit --packages org.apache.tika:tika-parsers-standard-package), degrading gracefully to EMPTY parses when absent.

EmbeddingTransformer (chained via --transformer-class)

• Appends a VECTOR(dimension) column by calling an embeddings API, batching at the record level within each partition (up to batch.size, default 1024, records per request) — executors stay busy, large request batches keep request rates low, and retry with Retry-After-honoring backoff is the only flow control.
• Pluggable EmbeddingProvider; default targets any OpenAI-compatible /v1/embeddings endpoint (Ollama, TEI, vLLM, OpenAI, Voyage). API keys come from an environment variable named in config, never config values. Errors after bounded retries fail the batch loudly; rows without text (images, videos, failed parses) get a null vector and are never sent to the API.
• Since only new/changed records flow through each sync, embeddings stay current with the data incrementally.

Also: tika.version property in the root pom; tika-core added to both utilities bundle include lists. No code copied from other projects.

Impact

New optional source + transformer in hudi-utilities; no existing Hudi APIs, configs, storage format, or bundles change behavior. New user-facing configs under hoodie.streamer.source.unstructured.* and hoodie.streamer.transformer.embedding.* (documented via @ConfigClassProperty). Utilities bundles grow by ~800 KB (tika-core).

Risk Level

low — additive code paths only. Verified by 10 offline tests (chunker boundaries, Tika status mapping on generated fixtures, source-level inline/out-of-line + checkpoint semantics, BLOB logical type surviving Row-to-Avro schema conversion, embedding batching/429-retry/fail-fast against a stub HTTP server) plus a streamer E2E writing a Parquet COW table across two syncs (blob struct round-trip for both placements, upsert refresh, vector logical type present in the committed table schema). Additionally validated end to end locally with real PDFs/DOCX/HTML/images, live Ollama embeddings, and hudi_vector_search queries across insert/update/delete cycles.

Documentation Update

Website docs for the new source/transformer configs and a quickstart example are planned as a follow-up PR (config reference tables generate from the config classes). Two notes to include: hudi_vector_search requires spark-mllib on the classpath (present in standard Spark distributions), and Office-format parsing needs a commons-compress version aligned with Tika's POI.

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[hudi-bot]

CI report:

• 1b5ec8f UNKNOWN

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build