maintainer: prevent stale spans from reentering scheduler state

[wk989898]

What problem does this PR solve?

Issue Number: close #6072

What is changed and how it works?

This PR prevents a stale SpanReplication from being reintroduced into the scheduler’s Absent set after it has already been removed, replaced, or rebound to another node.

Previously, terminal dispatcher status handling performed the following operations separately:

1. Look up the SpanReplication by dispatcher ID.
2. Process the dispatcher’s Stopped or Removed status.
3. Check that no operator exists.
4. Call MarkSpanAbsent with the previously obtained span pointer.

A concurrent split could call ReplaceReplicaSet between these operations. The split removed the old span and created its replacement spans, but the terminal-status handler still held a pointer to the old span. Because MarkSpanAbsent did not verify that the span was still part of the controller’s desired state, it inserted the obsolete span into the scheduler’s Absent set.

This created a ghost Absent span: the scheduler repeatedly tried to create an Add operator for a dispatcher that no longer existed in the span controller, resulting in continuous add operator failed, span not found errors and a stalled checkpoint.

This PR makes the transition safe in two ways:

• MarkSpanAbsent now checks, while holding the span controller mutex, that the dispatcher ID is still registered in allTasks and that the registered value is the exact same SpanReplication instance. If the span has already been removed or replaced, the operation returns without changing scheduler or checkpoint-tracker state.

• The terminal-status fallback uses MarkSpanAbsentIfCurrent, which additionally verifies under the same lock that the span is still bound to the node that reported the terminal status. This prevents a delayed Stopped status from an old owner from marking a span Absent after it has already been moved to another node.

ReplaceReplicaSet and the new validation use the same span controller mutex. Therefore, removal/replacement and the Absent transition now have a deterministic order:

ReplaceReplicaSet first
-> old span is no longer the current task
-> MarkSpanAbsent is skipped

MarkSpanAbsent first
-> the span is still current and owned by the reporting node
-> the valid Absent transition completes

The terminal-status warning is now emitted only when the span is actually transitioned to Absent. Stale terminal statuses that lose the atomic validation are ignored without producing a misleading state-change log.

Check List

Tests

• Unit test
• Integration test

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Fix a race condition that could stall the changefeed checkpoint and downstream replication after dispatcher split or move operations.

Summary by CodeRabbit

• Bug Fixes
  • Improved handling of terminal dispatcher statuses when no operator is available.
  • Prevented stale or reassigned spans from being incorrectly marked absent.
  • Ensured rescheduling occurs only when the span is still assigned to the reporting node.
  • Added safeguards for removed spans and improved reliability during node removal and reassignment scenarios.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hicqu for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

Warning

Review limit reached

Next included review available in 42 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 76378602-9d17-4b97-b297-184d565bbcd9

📥 Commits

Reviewing files that changed from the base of the PR and between 39ca074 and 767d00a.

📒 Files selected for processing (1)

• tests/integration_tests/run_heavy_it_in_ci.sh

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f8724d47-6692-47fa-b2bb-505f8bb8a5bc

📥 Commits

Reviewing files that changed from the base of the PR and between b813e76 and 39ca074.

📒 Files selected for processing (5)

• maintainer/maintainer_controller.go
• maintainer/operator/operator_move_test.go
• maintainer/operator/operator_split_test.go
• maintainer/span/span_controller.go
• maintainer/span/span_controller_test.go

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The change prevents stale terminal statuses from marking reassigned or removed spans absent. It adds ownership-aware span-controller APIs and regression coverage for reassignment, removal, move, and split scenarios.

Changes

Span absence guard

Layer / File(s)	Summary
Validated absence transition maintainer/span/span_controller.go	MarkSpanAbsent now returns success status and rejects nil or stale spans. MarkSpanAbsentIfCurrent also validates the expected node.
Terminal status dispatch maintainer/maintainer_controller.go	Terminal statuses use the reporting node when marking a span absent. Rescheduling occurs only when the span remains current on that node.
Stale ownership and removal coverage maintainer/span/span_controller_test.go, maintainer/operator/operator_move_test.go, maintainer/operator/operator_split_test.go	Tests cover removed spans, owner rebinding, successful absence transitions, and replica registration for node-removal scenarios.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 39ca0

The PR prevents stale span state from being reintroduced during dispatcher split or move races, reducing the chance of stalled checkpoint progress; no actionable merge-blocking risk remains, so it is merge-ready after normal checks.

Sequence Diagram(s)

sequenceDiagram
  participant MaintainerController
  participant SpanController
  participant Scheduler
  MaintainerController->>SpanController: MarkSpanAbsentIfCurrent(span, reportingNode)
  SpanController->>SpanController: Validate current task and node ownership
  alt Span remains current on reportingNode
    SpanController->>Scheduler: Trigger rescheduling
  else Span was removed or reassigned
    SpanController-->>MaintainerController: Return false
  end

Loading

Suggested reviewers: hongyunyan

Poem

A rabbit checks each span with care,
No stale mark shall linger there.
If owners change or tasks depart,
The guard keeps scheduler state smart.
“Hop!” says Bun, “the tests now know—”
Old reports cannot overthrow.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes address issue #6072 by preventing stale span state during concurrent split, merge, and replica reassignment operations.
Out of Scope Changes check	✅ Passed	All code and test changes directly support stale-span prevention and regression coverage for the reported scheduler race.
Docstring Coverage	✅ Passed	Docstring check was indeterminate for this PR — some files could not be analyzed in time. Not blocking.
Title check	✅ Passed	The title clearly summarizes the main change: preventing stale spans from reentering scheduler state.
Description check	✅ Passed	The description includes the issue number, problem, implementation details, tests, and release note, but leaves the compatibility and documentation questions unanswered.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[wk989898]

/test all

[wk989898]

/test pull-cdc-mysql-integration-heavy

[wk989898]

/test pull-cdc-mysql-integration-heavy

[wk989898]

/test pull-cdc-mysql-integration-heavy