feat: rework SegmentFileBuilderV10 to organize internal containers by projection to be friendly to partial loading/eviction

[clintropolis]

Description

Rework SegmentFileBuilderV10 so V10 containers are scoped by a new declared file group concept instead of size alone. Groups provide a layout hint: a new container rolls when the declared group changes, giving readers a clean 1-to-N mapping between groups and containers that will support per-group partial loading in the future. Projections are the primary caller today, but the mechanism is deliberately generic (shared data across columns, internal metadata, etc.).

changes:

• adds new method SegmentFileBuilder.startFileGroup(@Nullable String) which is a hint to the file builder about how to group the files a caller is adding
• SegmentFileBuilderV10 replaced the internal FileSmoosher with a purpose-built container writer that rolls on group changes or when a file won't fit, organizing containers by the new file group concept
• IndexMergerV10/IndexMergerBase use the new file group method to organize v10 segments by projection
• removed stuff from FileSmoosher that was only needed by v10 builder delegating to it
• AggregateProjectionSpec validates that the projection cannot be named __base

[FrankChen021]

Findings that could not be attached inline:

• processing/src/main/java/org/apache/druid/segment/file/SegmentFileBuilderV10.java:718 - P1 Delegated nested writes lose their file group. Nested addWithChannel calls are delegated while writerCurrentlyInUse is true, but mergeDelegatedFiles later replays those files with add(name, file) using whatever currentFileGroup is active at merge time. The original group active when the nested write was requested is not retained. If a nested file is created under one projection/container group and merged after the outer writer advances or clears the group, that nested file can be appended to the wrong container group, breaking the projection-to-container layout that partial loading depends on. Store the file group with each delegated temp file and restore/use it when merging.

[gianm]

Seems ok to me although I'm wondering about Frank's comment.

[gianm]

Might as well reserve all names starting with __, just in case we need another one later.

[clintropolis]

done, moved to a method on Projections

[clintropolis]

Seems ok to me although I'm wondering about Frank's comment.

Its like kind of real, but also maybe unlikely to ever happen in practice given standard uses of nested writers. The usage is like a column serializer is open and writing and needs to open a nested writer to write some of its parts to separate files which needs some temp files to do that get merged back in later. The problem described would occur only i think if someone closed the outer thingy but not the nested, then set a new group, then close the nested ones (which i'm not entirely sure how would occur with our current index merger paths). Nothing does this, because its pretty strange?

I went ahead and fixed the problem. Well, with the same limitations as the builder currently has here, that if the files aren't added to a group contiguously then they get stored in different containers, but it does at least ensure that containers have at most 1 group. So, if someone manages to do ^, it will put them into separate container associated with the group at the time of delegation instead of the current active group. 🤷

[FrankChen021]

The changes LGTM, no correctness issues found.