Skip to content

feat(supervisor): cancel a resumed run's in-flight checkpoint - #4502

Merged
nicktrn merged 7 commits into
mainfrom
feat/cancel-checkpoint-on-continue
Aug 5, 2026
Merged

feat(supervisor): cancel a resumed run's in-flight checkpoint#4502
nicktrn merged 7 commits into
mainfrom
feat/cancel-checkpoint-on-continue

Conversation

@nicktrn

@nicktrn nicktrn commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

A run controller must call the continue route to resume, so the supervisor already knows synchronously that any checkpoint still running for that run is pointless. It only acted on that for the compute path.

The continue route now cancels it for the Kubernetes path too, matching what completion already does since #4493. Called after the reply so the runner is never delayed, and skipped when there is no checkpoint client or when the compute path owns the run. The request is bounded by a 5s timeout so a hung call cannot leave the handler pending.

checkpoint_cancel_requests_total{result} records the outcome, using the same label names as the delete path where they overlap: sent, no_client, not_applicable, http_error.

No changeset: CheckpointClient is a server-only internal API, same as #4493.

refs TRI-12915

@nicktrn nicktrn self-assigned this Aug 4, 2026
@changeset-bot

changeset-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 3cf277b

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

The checkpoint client now provides cancelCheckpoints, which posts to the run checkpoint-cancellation endpoint with a five-second timeout and returns request success. The supervisor records cancellation outcomes, skips cancellation when required, and asynchronously cancels in-flight checkpoints after the continue route replies to the runner.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description explains the change and metrics but omits the required issue, checklist, testing, changelog, and screenshots sections. Add the required template sections, link the issue with “Closes #…”, document testing steps, and complete the checklist or explain skipped items.
✅ Passed checks (4 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly describes the main change: cancelling in-flight checkpoints when a resumed run continues.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/cancel-checkpoint-on-continue

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

devin-ai-integration[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@pkg-pr-new

pkg-pr-new Bot commented Aug 4, 2026

Copy link
Copy Markdown

Open in StackBlitz

@trigger.dev/build

npm i https://pkg.pr.new/@trigger.dev/build@3cf277b

trigger.dev

npm i https://pkg.pr.new/trigger.dev@3cf277b

@trigger.dev/core

npm i https://pkg.pr.new/@trigger.dev/core@3cf277b

@trigger.dev/python

npm i https://pkg.pr.new/@trigger.dev/python@3cf277b

@trigger.dev/react-hooks

npm i https://pkg.pr.new/@trigger.dev/react-hooks@3cf277b

@trigger.dev/redis-worker

npm i https://pkg.pr.new/@trigger.dev/redis-worker@3cf277b

@trigger.dev/rsc

npm i https://pkg.pr.new/@trigger.dev/rsc@3cf277b

@trigger.dev/schema-to-json

npm i https://pkg.pr.new/@trigger.dev/schema-to-json@3cf277b

@trigger.dev/sdk

npm i https://pkg.pr.new/@trigger.dev/sdk@3cf277b

commit: 3cf277b

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 new potential issues.

Open in Devin Review

Comment thread packages/core/src/v3/serverOnly/checkpointClient.ts
Comment thread apps/supervisor/src/workloadServer/index.ts

@1stvamp 1stvamp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Minor Q: other path uses DELETE_CHECKPOINTS_ON_COMPLETION, but this one doesn't, any reason or just missing?

@nicktrn

nicktrn commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator Author

Minor Q: other path uses DELETE_CHECKPOINTS_ON_COMPLETION, but this one doesn't, any reason or just missing?

Main reason: has nothing to do with deletion or completion. Devin got hung up on this too. Bad bot.

@nicktrn
nicktrn merged commit c01a4f1 into main Aug 5, 2026
59 checks passed
@nicktrn
nicktrn deleted the feat/cancel-checkpoint-on-continue branch August 5, 2026 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants