docs: add v10 into the tech-spec page#19282
Conversation
|
cc @vinothchandar for taking a look~ |
hudi-agent
left a comment
hudi-agent
left a comment
There was a problem hiding this comment.
Thanks for the docs update! This change extends the technical specification to table version 10, documenting the native log format (v2), the optional LSM-tree storage layout, L0/L1 compaction, and v9→v10 compatibility rules. The additions are well-structured and internally consistent; my comments concern a few areas where the "native logs by default" scope (non-Parquet formats and the metadata table) and the VERSION header encoding could confuse implementers. Please have a Hudi committer or PMC member confirm the format-support details against RFC-103 and the v10 implementation.
| physical log-format version stored in the block envelope or native-file footer. | ||
| * **write_token** - Token identifying the task and attempt that produced the file. It distinguishes retries and lets | ||
| cleaning remove unsuccessful attempts. | ||
| * **native_format** - Storage format of a native log file, such as `parquet`, `orc`, or `hfile`. The current writer uses |
There was a problem hiding this comment.
🤖 This bullet says native_format may be parquet, orc, or hfile and that "the current writer uses the effective base-file format," but the Native Log Format section states "The current LSM file-group reader expects native Parquet logs," and RFC-103 is described as defining Parquet as the primary layout. It might help to clarify whether ORC/HFile native logs are actually written and readable in table version 10 today, or whether v2 is Parquet-only for now with ORC/HFile listed as future formats. As written, a user with an ORC- or HFile-based table could reasonably expect native-log support that may not yet exist.
| * **native_format** - Storage format of a native log file, such as `parquet`, `orc`, or `hfile`. The current writer uses | ||
| the effective base-file format; Lance is the exception described below. | ||
|
|
||
| Table versions below 10 write inline logs. Writers targeting table version 10 or later write native logs by default, |
There was a problem hiding this comment.
🤖 "Writers targeting table version 10 or later write native logs by default, except for Lance-backed tables" — the metadata table is HFile-based and relies on inline HFile log blocks with point-lookup merge semantics. Since the default native format follows the effective base-file format, this rule as stated would imply the metadata table writes native HFile logs at v10, yet the reader is described as expecting native Parquet. Could the spec clarify whether the metadata table is exempt (continues writing inline HFile logs) or migrates to native logs at v10? @yihua could you confirm the intended metadata-table log behavior under table version 10 so the exception list here is complete?
| | BLOCK\_IDENTIFIER | 6 | Block sequence number used to detect duplicate log blocks (e.g. due to task retries). | | ||
| | IS\_PARTIAL | 7 | Boolean indicating whether the data block encodes partial updates. | | ||
| | BASE\_FILE\_INSTANT\_TIME\_OF\_RECORD\_POSITIONS | 8 | Begin (requested) instant time of the base file that the `RECORD_POSITIONS` bitmap is relative to. Used by the reader to validate that positions still refer to the same base file when reconstructing a file slice. | | ||
| | VERSION | 9 | Physical log-format version. Native log writers always add this value to `hudi.log.format.metadata`. | |
There was a problem hiding this comment.
🤖 The VERSION row is added to the inline Headers table with Encoding ID 9, but the description says it is added by native log writers to the hudi.log.format.metadata JSON — and the note just above the table (and the Block types section) states native logs use symbolic names and "readers must not interpret the numeric IDs." So Encoding ID 9 would only ever apply to inline logs. Does the inline v0/v1 format actually serialize a VERSION header? Note the inline envelope already carries a 4-byte version field (see the Inline Log Format spec table), so a VERSION header there would be redundant. It might help to clarify whether VERSION is inline-serialized at all, or whether this row belongs conceptually to the native metadata (in which case an integer Encoding ID is a bit misleading).
3 participants
Describe the issue this Pull Request addresses
This closes #19292
The site technical specification currently stops at table version 9 and does not describe the table version 10 storage changes introduced by RFC-103. In particular, it lacks the native log format, LSM-tree storage layout, and version 9/10 compatibility requirements needed by reader and writer implementers.
This is a documentation-only change. It does not modify the storage format or runtime behavior.
Summary and Changelog
Updates the current technical specification for Hudi 1.3.0 and table version 10.
hoodie.table.storage.layout, the optional LSM-tree layout, sorted writes, and k-way merge behavior.Impact
Documentation only. There are no public API, configuration, runtime, compatibility, or performance changes introduced by this PR.
The updated specification provides storage-format guidance for users and engine implementers working with table version 10 and RFC-103.
Risk Level
low
Only
website/learn/tech-specs.mdis changed. The production site was validated with:npm run buildThe build completed successfully. It reported existing broken-anchor warnings from unrelated documentation pages.
Documentation Update
This PR is the documentation update. It brings the site technical specification up to date with table version 10 and RFC-103. No additional documentation changes are required.
Contributor's checklist