Canonicalize the default null value and intern per-column strings at metadata parse time - #19473
Open
xiangfu0 wants to merge 1 commit into
Open
Conversation
This was referenced Sep 6, 2026
Open
xiangfu0
added
performance
…metadata parse time
A server retains one ColumnMetadataImpl + FieldSpec per (segment, column) for as long as the segment is loaded.
For wide segments (1000+ columns, tens of thousands of segments per server) two per-column allocations made
on the metadata.properties parse path add up to ~120 bytes per column that carry no information:
- The segment creator writes `column.<c>.defaultNullValue` for every column, so `extractFieldSpec` handed the
literal to the FieldSpec constructor, which boxed it (`Integer.valueOf("-2147483648")`) and kept the literal
in the transient `_stringDefaultNullValue` - a fresh Integer, String and byte[] per column per segment even
though the value is the type default. `ColumnMetadataImpl.extractFieldSpec` now parses the literal, compares
it with `FieldSpec.getDefaultNullValue(fieldType, dataType, null)` under `DataType.equals` (the predicate
`FieldSpec.equals`/`hashCode` use, byte[]-safe) and passes `null` when they match, so the spec holds the
shared static `FieldSpec.DEFAULT_*` constant and retains no literal. A custom default is kept verbatim (after
the STRING special-character recovery) and interned, so the segments of a table share it; a combination
without a type default (a METRIC BOOLEAN with an explicit default) keeps parsing the literal as before.
- Column names recur in every segment of a table, yet each segment parsed its own copy that was then retained
six times over: the column metadata map key, FieldSpec._name, the Schema map key and dimension/metric list
entry, and the loader's per-column maps. `SegmentMetadataImpl.addPhysicalColumns` now interns the parsed name,
and `extractFieldSpec` interns COLUMN_NAME, PARENT_COLUMN, DATETIME_FORMAT, DATETIME_GRANULARITY and complex
child names, so every String the metadata graph retains is one instance per distinct value per JVM. The JVM
string table holds interned strings weakly, so they live exactly as long as a loaded segment references them
and the table is bounded by the number of distinct column names; no code compares column names by identity.
Compatibility: metadata.properties is read and written exactly as before (writers keep emitting
defaultNullValue). The `/tables/{t}/segments/{s}/metadata` payload is unchanged because it bean-serializes
`FieldSpec.getDefaultNullValue()` (a value that is equal by construction) - pinned by a test for every data
type, including BYTES and UUID. FieldSpec.equals/hashCode ignore `_stringDefaultNullValue` and
getDefaultNullValueString() derives from the value, so Schema equality against the table schema, default-column
comparisons and reload decisions are unaffected. The only observable difference is that
`FieldSpec.toJsonObject()` of a segment-derived BYTES column no longer emits a redundant `defaultNullValue: ""`
(it compares the byte[] against the type default by identity, and the spec now holds the constant); no
endpoint serializes a segment-derived Schema that way. No public signature changes; mixed-version safe
(server-local, in-memory only).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
xiangfu0
force-pushed
the
xiangfu0/data-3221-2-intern-parse
branch
from
September 9, 2026 01:56
c566594 to
b61c923
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## xiangfu0/data-3221-1-lazy-index-sizes #19473 +/- ##
===========================================================================
- Coverage 67.75% 67.75% -0.01%
Complexity 1430 1430
===========================================================================
Files 3489 3489
Lines 224678 224690 +12
Branches 35470 35471 +1
===========================================================================
Hits 152232 152232
- Misses 60431 60435 +4
- Partials 12015 12023 +8
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two parse-time changes in
ColumnMetadataImplandSegmentMetadataImpl, both removing per-column copies of values that are identical across every segment of a table.defaultNullValueliteral that equals the type default is no longer handed to theFieldSpec, so the spec carries the shared staticFieldSpec.DEFAULT_*constant instead of retaining the literal and a freshly boxed value.FieldSpec.equals/hashCodecompare the value throughDataType.equalsand ignore the transient string form, so this is value-identical.The
/tables/{t}/segments/{s}/metadatapayload is bean-serialized from the getters, so it is unchanged; a test pins byte-identical JSON per data type.Tests
ColumnMetadataImplTest: type-default literals share the static constant across INT/LONG/FLOAT/DOUBLE/BOOLEAN/TIMESTAMP/STRING/JSON/BYTES/UUID/BIG_DECIMAL; custom literals round-trip; special characters in string defaults survive; the JSON is unchanged.SegmentMetadataImplTest: two loads of one segment share the name and spec instances, including OPEN_STRUCT children.Why
A server keeps one metadata object graph per (segment, column) for as long as the segment is loaded, so on wide tables the per-column footprint decides how many segments a server can hold. Measured end to end on a 1000-column segment, this series takes the heap retained at load from 4.08 MB to 0.175 MB per segment (4,080 to 174 bytes per column), with a fully compacting collector on both sides. No on-disk format change, the
/tables/{table}/segments/{segment}/metadataJSON stays byte-identical, and every public and SPI signature keeps working.Stack
Part 2 of 9, based on #19480. Review only this part's own commits; the earlier parts account for the rest of the diff.