Add latency tracking for MSE adaptive routing

[timothy-e]

#18553 added in-flight request tracking for MSE in the adaptive routing stats. This PR adds per-server latency. It looks very large, but much of it is tests, and some new files were introduced, which comes with a certain amount of boilerplate in Java.

1. The planner marks stages as needing timing. It assigns FragmentTypes and then the dispatcher uses those to turn those roles into timing collection in AdaptiveRoutingStageClassification.java. it always enables collection on stage 0, and otherwise only on trusted stages that either directly receive from pure leaves or are singleton-leaf stages themselves, while excluding stages fed by non-leaf senders so their timings do not include upstream cascade delay.

2. Stages track their elapsed time and return it as UPSTREAM_SERVER_RESPONSE_TIMES_MS stat (BaseMailboxReceiveOperator, BlockingMultiStreamConsumer, AdaptiveRoutingUpstreamTimings)

Each BaseMailboxReceiveOperator tracks per-sender wall-clock elapsed time (from when its BlockingMultiStreamConsumer.OfMseBlock is constructed until each sender's EOS arrives). The start time is captured at consumer construction time, not at query start, to avoid pipeline-breaker inflation: if a pipeline breaker on this stage blocks for 1000ms waiting on a slow sender, fast senders' EOS blocks sit unconsumed in their queues during that time, and measuring from query start would report ~1000ms for all of them.

These per-sender timings are encoded as "hostname|port=elapsedMs;..." in a new UPSTREAM_SERVER_RESPONSE_TIMES_MS stat key on BaseMailboxReceiveOperator.StatKey and propagated up through the query stats. A serialized string is used because it is more straightforward than adding a Map type to the StatKey system.

3. Broker reads leaf-server timings and resolves hostname|mailboxPort sender keys to full instance IDs and accumulates the maximum observed latency per instance across all trusted stages. A decrementedServers set prevents double-recording.

4. Query timeout: on query cancellation (e.g. reaching the timeout), we try to cancel with stats. We preserve completed timings and inject elapsed time for pending senders.

Testing

1. Performance impact: We ran some performance tests and saw no impact.

2. Manually with tc

Choose a server that we want to induce latency on. ssh to that server and:

sudo tc qdisc add dev ens5 root handle 1: prio
sudo tc qdisc add dev ens5 parent 1:3 handle 30: netem delay 1200ms

sudo tc filter add dev ens5 protocol ip parent 1:0 prio 3 u32 \
  match ip dport 8442 0xffff flowid 1:3

We induced latency only for MSE to validate that adaptive routing can be influenced by only MSE queries.

To reset the latency

sudo tc qdisc del dev ens5 root # cleanup

We saw the score spike for the targetted servers.
3. Chaos Agent
We kicked off a chaos agent run that does

1. 90s of 600ms MSE latency
2. 30s with no faults
3. 90s of 600ms SSE latency
4. 90s with no faults
5. 90s of 1200ms MSE latency
6. 30s with no faults
7. 90s of 1200ms SSE latency

(query timeout is 1s).

[codecov-commenter]

Codecov Report

❌ Patch coverage is 85.38206% with 44 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.45%. Comparing base (6ed151a) to head (f965c81).

Files with missing lines	Patch %	Lines
.../pinot/query/service/dispatch/QueryDispatcher.java	83.49%	9 Missing and 8 partials ⚠️
...e/dispatch/AdaptiveRoutingStageClassification.java	74.00%	7 Missing and 6 partials ⚠️
...rvice/dispatch/AdaptiveRoutingUpstreamTimings.java	83.78%	3 Missing and 3 partials ⚠️
...ava/org/apache/pinot/common/datatable/StatMap.java	20.00%	0 Missing and 4 partials ⚠️
...y/runtime/operator/BaseMailboxReceiveOperator.java	91.89%	1 Missing and 2 partials ⚠️
...uery/planner/physical/DispatchablePlanContext.java	83.33%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #18646      +/-   ##
============================================
+ Coverage     64.39%   64.45%   +0.06%     
  Complexity     1282     1282              
============================================
  Files          3362     3365       +3     
  Lines        207915   208185     +270     
  Branches      32463    32525      +62     
============================================
+ Hits         133883   134192     +309     
+ Misses        63258    63204      -54     
- Partials      10774    10789      +15     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-21	64.45% <85.38%> (+0.06%)	⬆️
temurin	64.45% <85.38%> (+0.06%)	⬆️
unittests	64.45% <85.38%> (+0.06%)	⬆️
unittests1	56.88% <85.04%> (+0.09%)	⬆️
unittests2	37.12% <10.96%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[timothy-e]

Hey @yashmayya, here's part two of adaptive routing for MSE. I've done some pretty extensive testing/baking with it on our QA/prod clusters over the last few weeks (although right prod is stats collection-only, not actually using adaptive routing). Sorry for the big change, I don't think there's a way to break it down smaller while keeping each part testable.