feat(format)!: add file metadata size hint by Ali2Arslan · Pull Request #9168 · lance-format/lance

Ali2Arslan · 2026-09-11T22:46:33Z

Proposal

Add one advisory file_metadata_size_bytes value to DataFile and IndexFile. For a Lance file of format version 2.0 or later, it is the file size minus the position of global buffer 0, which holds the file's FileDescriptor. That position is stored in the first entry of the global buffer offset table, which the footer locates. The value is always zero for legacy (0.1) files and for index files in other formats; zero also means unknown. Writers record either this exact value or zero.

Footer offsets and counts remain authoritative. Readers ignore a value larger than the file and read any metadata the footer references outside the hinted range.

Rationale

A full metadata open always needs this complete suffix. Recording its exact size lets a reader schedule the required bytes immediately instead of waiting for a footer read before scheduling the metadata prefix. This optimizes dependency depth independently of physical request count; the scheduler may still split a large logical range into concurrent requests.

Projected data-file opens do not need another persisted value: an implementation can estimate the CMO table from the manifest's existing physical-column mappings and retain a prefix-only fallback.

Compatibility

Existing readers ignore the new protobuf fields.
Existing writers encode zero by omission.
New readers retain the ordinary footer-directed path when the value is unusable.
The Lance file bytes and footer grammar do not change.
No feature flag or file-format version bump is required because the value is advisory.
The in-memory representation uses Option<NonZeroU64>, so the hint occupies one u64 slot.
DataFile JSON always includes file_metadata_size_bytes (null when unknown), like file_size_bytes and base_id. Python's DataFile accepts and carries the value because FragmentMetadata.from_json builds it from that JSON, for example for write progress and distributed compaction.
Java conversions and the Python IndexFile conversion initialize the new fields as unavailable until the follow-up implementation propagates them.

Per the format-change process, this PR contains only the durable contract, documentation, and minimum compilation/round-trip updates. The reader/writer implementation is prepared in #9169 and remains draft until this change merges.

Maintainer action

Gatekeeper requested the breaking-change label. GitHub does not permit fork contributors to apply repository labels, so a maintainer must add it.

Validation

cargo test -p lance-table --lib -- test_to_json test_intern_fragment_file_metadata_size_bytes test_rewritten_index_roundtrips test_roundtrip_fragment test_index_metadata_codec_roundtrip
uv run pytest python/tests/test_fragment.py python/tests/test_optimize.py python/tests/test_dataset.py::test_dataset_progress
cargo clippy --all --tests --benches -- -D warnings
uv run make lint (from python/)
python3 ci/check_proto_comments.py protos/table.proto protos/fragment_metadata.proto

Made with Cursor

Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions · 2026-09-11T22:55:09Z

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer), at least one of them on the latest commit, and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Approvals carry over across pushes, so a rebase or a typo fix does not send everyone back to re-vote. Whoever approves the latest commit is vouching that nothing substantive has changed since the earlier approvals; if something has, ask for fresh votes.

Status: ❌ Blocked — 1 of 3 required approvals


Approvals	@wjones127 (1/3)
Latest commit approved by	@wjones127
Vetoes	none
Voting period	elapsed — ended Wed 2026-09-16 22:46 UTC (15:46 PDT)

_{Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).}

Co-authored-by: Cursor <cursoragent@cursor.com>

…ze-hint-spec DataFile now lives in fragment_metadata.proto, so the metadata size hint is added there. Co-authored-by: Cursor <cursoragent@cursor.com>

wjones127 · 2026-10-08T21:10:38Z

   * Lance use it as key of the base_paths field in Manifest to determine the actual base path of the data file.
   */
  optional uint32 base_id = 7;
+


issue (blocking): "Schema descriptor" isn't a defined term in the file spec, and this is the durable contract. Acceptance: the proto comment and docs state exactly which byte offset the suffix starts at (derived from footer fields for v2.x) and whether v1/legacy files get a value or stay 0, so independent writers produce identical values.

wjones127 · 2026-10-08T21:10:38Z


+### File Metadata Size Hints
+
+`DataFile.file_metadata_size_bytes` records the suffix length from the Lance


issue (blocking): Same definition gap: "suffix length from the Lance schema descriptor" needs a precise anchor (offset in file-format terms). Also reconcile "Lance-format" here with "current-format" in the PR body.

wjones127 · 2026-10-08T21:10:38Z


 </details>

+### File Metadata Size Hints


nitpick: This section lands between the DataFile message block and the "Field-to-column mapping" note, which belongs to the Data Files section. Consider placing it after that note.

wjones127 · 2026-10-08T21:10:38Z

            file_minor_version: proto.file_minor_version,
            file_size_bytes: CachedFileSize::new(proto.file_size_bytes),
            base_id: proto.base_id,
+            file_metadata_size_bytes: NonZero::new(proto.file_metadata_size_bytes),


issue (blocking): The interner path (intern_fragment) is what manifest loads use, and nothing tests that it carries the hint. Acceptance: a test builds a pb::DataFragment with a non-zero file_metadata_size_bytes, runs it through intern_fragment, and asserts the value survives (and that 0 gives None).

wjones127 · 2026-10-08T21:10:38Z

                        .map(|f| IndexFile {
                            path: f.path.clone(),
                            size_bytes: f.size_bytes,
+                            file_metadata_size_bytes: NonZero::new(f.file_metadata_size_bytes),


issue (blocking): The RewrittenIndex ↔ proto conversion (this line and the From impl below) changed with no test. Acceptance: a roundtrip test with a non-zero hint on a rewritten index's files.

wjones127 · 2026-10-08T21:10:38Z

    fn serialize<S: Serializer>(&self, serializer: S) -> std::result::Result<S::Ok, S::Error> {
        use serde::ser::SerializeStruct;
-        let mut s = serializer.serialize_struct("DataFile", 7)?;
+        let field_count = 7 + usize::from(self.file_metadata_size_bytes.is_some());


question (non-blocking): Why omit the field from serde output when None rather than always emitting it? Non-self-describing formats (bincode-style) would then mismatch the deserialize helper. Is DataFile serialized that way anywhere? Also, please add a JSON test for both present and absent cases.

Anchor the hint at global buffer 0, which holds the FileDescriptor, in both proto comments and the table format docs. Legacy (0.1) files and non-Lance index files always record 0, and writers record either the exact value or 0. Move the docs section after the field-to-column note. Always serialize DataFile.file_metadata_size_bytes, like file_size_bytes and base_id. Python's DataFile now accepts and carries the value because FragmentMetadata.from_json builds it from Rust-serialized fragment JSON. Cover the hint in the manifest interner, the RewrittenIndex conversion, and JSON serialization. Co-authored-by: Cursor <cursoragent@cursor.com>

lance-gatekeeper

✅ Gate recommendation: approve.

2aaa5f4e addresses the suffix-definition concern: the proto comments and spec now identify the exact FileDescriptor boundary and when the value must be zero. JSON and Python carry the optional value consistently, with regression coverage for absence and round trips.

The contract remains advisory, with footer-authoritative reads and no change to the file grammar. This format contract can land before #9169, which supplies the reader/writer optimization.

wjones127

This looks good to me. Thanks for adding the clarification. 😄

feat(format): add file metadata size hint

41b5c4e

Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions Bot added enhancement New feature or request A-index Vector index, linalg, tokenizer A-format On-disk format: protos and format spec docs format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo). labels Sep 11, 2026

Ali2Arslan mentioned this pull request Sep 11, 2026

perf(file): reduce metadata read scheduling rounds #9169

Draft

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 11, 2026

lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. and removed K-approved Latest Gatekeeper recommendation permits acceptance. labels Sep 11, 2026

fix(bindings): initialize metadata size hints

af9f600

Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions Bot added A-python Python bindings A-java Java bindings + JNI labels Sep 12, 2026

lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 12, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 12, 2026

Ali2Arslan changed the title ~~feat(format): add file metadata size hint~~ feat(format)!: add file metadata size hint Sep 14, 2026

github-actions Bot added the breaking-change label Sep 14, 2026

lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. and removed K-approved Latest Gatekeeper recommendation permits acceptance. labels Sep 14, 2026

Merge remote-tracking branch 'origin/main' into feat/file-metadata-si…

a152480

…ze-hint-spec DataFile now lives in fragment_metadata.proto, so the metadata size hint is added there. Co-authored-by: Cursor <cursoragent@cursor.com>

lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 8, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 8, 2026

wjones127 self-requested a review October 8, 2026 19:46

wjones127 requested changes Oct 8, 2026

View reviewed changes

lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 8, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Oct 8, 2026

lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Oct 9, 2026

lance-gatekeeper Bot reviewed Oct 9, 2026

View reviewed changes

lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 9, 2026

wjones127 approved these changes Oct 10, 2026

View reviewed changes


		### File Metadata Size Hints

		`DataFile.file_metadata_size_bytes` records the suffix length from the Lance

Conversation

Ali2Arslan commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposal

Rationale

Compatibility

Maintainer action

Validation

Uh oh!

This comment was marked as outdated.

Uh oh!

github-actions Bot commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Format specification vote

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

wjones127 Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

wjones127 Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

wjones127 Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

wjones127 Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

wjones127 Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

wjones127 Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

lance-gatekeeper Bot left a comment

Choose a reason for hiding this comment

Uh oh!

wjones127 left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Ali2Arslan commented Sep 11, 2026 •

edited

Loading

github-actions Bot commented Sep 11, 2026 •

edited

Loading