[Spark] Fire processing-time timers in timestamp order

[Eliaaazzz]

SparkTimerInternals.getNextProcessingTimer() picked the eligible processing-time timer with the latest timestamp, so when several timers for a key were due in the same micro-batch their @OnTimer callbacks ran latest-first: a callback scheduled for T2 observed state that the callback for an earlier T1 had not written yet. TimerUtils.getExpiredTimers collected the expired-timer sweep from an unordered set, so that path had no defined order either.

Timers now fire earliest-first, matching InMemoryTimerInternals and the other runners, and the timers fired by triggerExpiredTimers are sorted by timestamp; the deletion-only sweep stays unsorted.

Earliest-first would also have surfaced superseded settings of a re-set timer, which the old set-based store kept alongside the replacement. Timers are therefore stored in a map keyed by namespace, id and family, so a later setting replaces the prior one, per the TimerInternals contract. Timers restored from state that predates this change collapse to the setting with the latest target; a checkpoint written before it cannot be recovered by an upgraded application anyway (Spark's DStream checkpoints do not survive application upgrades), so the merge only normalizes bytes Spark already declares unrecoverable.

testProcessingTimersFireInTimestampOrder fails on master with the later timer drained first; testSettingATimerAgainClearsThePriorSetting and testAddTimersKeepsTheLatestSettingOfATimer pin the replacement semantics.

Fixes #39824.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[codecov]

Codecov Report

❌ Patch coverage is 77.77778% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.14%. Comparing base (6356a3c) to head (ea5211a).
⚠️ Report is 7 commits behind head on master.

Files with missing lines	Patch %	Lines
...am/runners/spark/stateful/SparkTimerInternals.java	75.00%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #39825      +/-   ##
============================================
+ Coverage     55.95%   58.14%   +2.19%     
- Complexity     2284    13139   +10855     
============================================
  Files          1116     2524    +1408     
  Lines        175741   265686   +89945     
  Branches       1475    10822    +9347     
============================================
+ Hits          98332   154495   +56163     
- Misses        74901   105353   +30452     
- Partials       2508     5838    +3330     

Flag	Coverage Δ
java	64.26% <77.77%> (-11.39%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[github-actions]

Assigning reviewers:

R: @chamikaramj added as fallback since no labels match configuration

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).