feat: add partial load rule types by clintropolis · Pull Request #19374 · apache/druid

clintropolis · 2026-04-24T18:44:41Z

Description

Adds a new family of retention rules, loadPartialByPeriod, loadPartialByInterval, loadPartialForever, laying the groundwork for partial loading of V10 segment projections on historicals. This PR includes the rule classes, matcher abstraction, and cascade semantics only; the coordinator plumbing, wire format, and historical-side partial load path are deferred to follow-up work.

changes:

adds PartialLoadRule abstract class to capture load rules that should partially load a segment on some tier
adds IntervalPartialLoadRule, ForeverPartialLoadRule, and PeriodPartialLoadRule` implementations to mirror non-partial load rules
adds PartialLoadMatcher interface to match and select what to partial load
adds ExactProjectionPartialLoadMatcher and WildcardProjectionPartialLoadMatcher to do partial loading of projections
adds CannotMatchBehavior enum to describe behavior of PartialLoadRule when PartialLoadMatcher is unable to match a segment
since partial loading is not available yet, partial rules function as regular load rules until follow-up work
tests

changes: * adds `PartialLoadRule` abstract class to capture load rules that should partially load a segment on some tier * adds `IntervalPartialLoadRule, `ForeverPartialLoadRule`, and `PeriodPartialLoadRule` implementations to mirror non-partial load rules * adds `PartialLoadMatcher` interface to match and select what to partial load * adds `ExactProjectionPartialLoadMatcher` and `WildcardProjectionPartialLoadMatcher` to do partial loading of projections * adds `CannotMatchBehavior` enum to describe behavior of `PartialLoadRule` when `PartialLoadMatcher` is unable to match a segment * since partial loading is not available yet, partial rules function as regular load rules until follow-up work * tests

FrankChen021 · 2026-04-25T06:34:15Z

+    this.matcher = matcher;
+    this.onCannotMatch = Configs.valueOrDefault(onCannotMatch, CannotMatchBehavior.FULL_LOAD);
+  }
+


P1 Partial rule fall-through is invisible to handoff checks

appliesTo returns false for a partial rule whose matcher cannot match and whose onCannotMatch is FALL_THROUGH. Handoff/readiness code that only asks whether the rule applies to the segment interval can therefore miss that this rule intentionally falls through to a later full-load rule, and can conclude that no load rule covers the segment. A realtime task for a segment without matching projections can then wait forever even though the cascade would load it through the following rule. The rule-selection path used by handoff needs to evaluate the same matcher/fall-through semantics as the coordinator cascade, not just interval applicability.

gianm · 2026-04-26T03:35:38Z

+ * version that introduces a new behavior falls back to the constructor's default ({@link #FULL_LOAD}) rather than
+ * failing to parse the rule.
+ */
+public enum CannotMatchBehavior


IMO it'd be nicer to define the enums like FALL_THROUGH("fallThrough") and have the "fallThrough" be how they serialize in JSON. SImilar to JoinAlgorithm and CloneQueryMode. Most Druid stuff uses thatStyle rather than THIS_ONE.

gianm · 2026-04-26T04:06:44Z

+ * configuration that drives the decision and the wire format of their corresponding {@link LoadSpec} wrapper, so the
+ * rule layer stays scheme-agnostic.
+ */
+@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type")


If we add a new one of these then the PartialLoadRule will fail to deserialize. Just like CannotMatchBehavior, maybe this should be lenient too?

The way to make things lenient is to add a dummy implementation as a defaultImpl, such as QueryCounterSnapshot for example. Servers will then at least know that they're dealing with an unrecognized PartialLoadMatcher.

gianm · 2026-04-26T04:08:32Z

+    return patterns;
+  }
+
+  @JsonProperty


@JsonInclude(NON_EMPTY) would be nice.

gianm · 2026-04-26T04:09:54Z

+        "type", WIRE_TYPE,
+        "delegate", baseLoadSpec,
+        "projections", resolved,
+        "ruleFingerprint", fingerprint


It isn't really a rule fingerprint, it's more of a projection names fingerprint. Maybe just call it fingerprint since projectionNamesFingerprint is a mouthful.

gianm · 2026-04-26T04:31:54Z

+ * Base for {@link PartialLoadMatcher} implementations that decide which of a segment's V10 projections to load.
+ * Subclasses supply the resolution policy via {@link #resolveProjectionNames(DataSegment)}; this base handles
+ * fingerprint computation and wraps the result into the {@code partialProjection} load-spec wire form consumed
+ * by the historical-side {@code PartialProjectionLoadSpec} (which does not exist yet, supplied in future work).


IMO best not to mention things that don't exist yet. Instead, the next PR should update this text. It will need to update it anyway to remove the "supplied in future work" comment.

gianm · 2026-04-26T04:33:03Z

+ */
+public abstract class ProjectionPartialLoadMatcher implements PartialLoadMatcher
+{
+  static final String WIRE_TYPE = "partialProjection";


LOAD_SPEC_TYPE? Sounds clearer.

gianm · 2026-04-26T04:36:48Z

+  {
+    final Hasher hasher = Hashing.sha256().newHasher();
+    for (String name : sortedDedupedNames) {
+      hasher.putString(name, StandardCharsets.UTF_8);


The javadoc for putString says that putUnencodedChars is preferred.

gianm · 2026-04-26T04:42:03Z

+  private static boolean matchesAny(String name, List<String> patterns)
+  {
+    for (String pattern : patterns) {
+      if (FilenameUtils.wildcardMatch(name, pattern)) {


How does this treat \*? I wonder if it escapes the * or if it looks for a literal \ followed by anything.

It's probaly not likely that a projection name will include * or ?, but it would be nice to support escaping anyway, in case they do.

ah, it did not handle escapes... had claude help generate a version that supports escaping which seems to not add too much complexity

gianm · 2026-04-26T04:57:25Z

+  @Override
+  public boolean appliesTo(Interval interval, DateTime referenceTimestamp)
+  {
+    return Rules.eligibleForLoad(period, interval, referenceTimestamp, includeFuture);


I wonder if anything weird will happen because appliesTo(segment.getInterval(), refTs) doesn't always match appliesTo(segment, refTs) like it does with the other rules. This overload is used in two places:

DataSourcesResource#isHandOffComplete, where it's used to short-circuit handoff checking for segments that no load rule applies to. I suppose the risk here is that we're in a situation where really no rule matches (because the segment doesn't have any desired projections for a partial-rule, and there's no other rule to fall back to), but the handoff checker thinks the partial-rule would match, and so handoff times out. This seems unlikely to happen— probably in a real cluster there would be another rule to fall back on— but is worth a comment somewhere acknowledging it's a risk. Possibly right here.

TieredBrokerHostSelector, where it's used to find the rule that will drive which Broker handles a request when druid.router.tierToBrokerMap is set. This is probably fine: in some configurations of load rules it might change which broker gets a query, but probably the behavior is defensible given that the routing isn't projection-aware.

i switched isHandOffComplete to use the metadata snapshot to use the version of appliesTo that takes a segment so that it will work properly.

I didn't make any changes for TieredBrokerHostSelector, since i'm not really sure how to improve that, i guess we can document it when we document the rest of this stuff.

+      List<String> projections
+  )
+  {
+    return DataSegment.builder()


FrankChen021 · 2026-04-27T13:36:16Z

+                                      SegmentId.of(dataSourceName, theInterval, version, partitionNumber)
+                                  );
+      // Segment isn't published in metadata; it will never be handed off.
+      if (segment == null) {


[P1] Do not treat a missing metadata snapshot segment as handed off

getRecentDataSourcesSnapshot() may return a snapshot up to the segment poll delay old, so a just-published realtime segment can be absent even though it still needs handoff. Returning true here tells the task the segment will never be handed off and can let it stop waiting before any historical has loaded it. This should return false or force a fresh metadata poll before concluding the segment is not published.

clintropolis added 2 commits April 24, 2026 11:41

switch exclude to patterns too, fix style

f77c277

FrankChen021 reviewed Apr 25, 2026

View reviewed changes

gianm reviewed Apr 26, 2026

View reviewed changes

clintropolis added 2 commits April 27, 2026 03:01

better

79ab545

improve a bunch of stuff

05256b0

github-advanced-security AI found potential problems Apr 27, 2026

View reviewed changes

Comment thread server/src/test/java/org/apache/druid/server/http/DataSourcesResourceTest.java

List<String> projections

)

{

return DataSegment.builder()

FrankChen021 reviewed Apr 27, 2026

View reviewed changes

clintropolis added 3 commits April 27, 2026 10:41

refresh snapshot if segment missing

e04eb70

fix style

83eef8b

oops

150daa6

Conversation

clintropolis commented Apr 24, 2026

Description

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants