[v3-3-test] Fail task instances whose stored next_kwargs cannot be processed (#70685) - #71183
Merged
Merged
Conversation
) * Fail task instances whose stored next_kwargs cannot be processed handle_event_submit decoded a task instance's stored next_kwargs and assumed the result was a dict. Neither assumption held: the decode caught only four exception types, so anything the BaseSerialization fallback raised escaped, and the isinstance check sat under TYPE_CHECKING, so it never ran at runtime. Both escape as exceptions from a function whose callers walk every waiting task instance in one pass — the scheduler's timeout sweep and two API routes — so a single unusable payload aborted the whole batch. Decode through a helper that checks its result, and guard decode, event insertion and re-encode together. A task instance whose payload cannot be processed is re-queued to fail through the existing __fail__ path, so its normal retry and callback handling still runs, instead of being left parked for the next sweep to trip over again. Generated-by: Claude Opus 5 (1M context) following the guidelines at https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions * Report why a task instance could not be resumed, and to whom The Dag author sees only the task log, so the traceback has to travel in next_kwargs the way submit_failure already sends it; the process log where this runs is often not theirs to read. Decode and re-encode also fail for different reasons: blaming the stored kwargs for a payload the trigger just yielded points the author at database state that was never at fault. The sweep's summary counted an unresumable task as resolved. * Update airflow-core/src/airflow/models/trigger.py Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com> * Fix ruff D205/D213 on _fail_unresumable_task_instance docstring One-line summary on the second line + blank line before the description, so the docstring satisfies both D205 (blank between summary and description) and D213 (summary on second line). Static checks were failing on ruff for this. --------- Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com> Co-authored-by: Rahul Vats <rah.sharma11@gmail.com> (cherry picked from commit 2b7a0be)
boring-cyborg
Bot
added
area:Scheduler
amoghrajesh
approved these changes
Aug 5, 2026
potiuk
approved these changes
Aug 5, 2026
vatsrahul1001
added a commit
that referenced
this pull request
Aug 5, 2026
) (#71183) * Fail task instances whose stored next_kwargs cannot be processed handle_event_submit decoded a task instance's stored next_kwargs and assumed the result was a dict. Neither assumption held: the decode caught only four exception types, so anything the BaseSerialization fallback raised escaped, and the isinstance check sat under TYPE_CHECKING, so it never ran at runtime. Both escape as exceptions from a function whose callers walk every waiting task instance in one pass — the scheduler's timeout sweep and two API routes — so a single unusable payload aborted the whole batch. Decode through a helper that checks its result, and guard decode, event insertion and re-encode together. A task instance whose payload cannot be processed is re-queued to fail through the existing __fail__ path, so its normal retry and callback handling still runs, instead of being left parked for the next sweep to trip over again. Generated-by: Claude Opus 5 (1M context) following the guidelines at https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions * Report why a task instance could not be resumed, and to whom The Dag author sees only the task log, so the traceback has to travel in next_kwargs the way submit_failure already sends it; the process log where this runs is often not theirs to read. Decode and re-encode also fail for different reasons: blaming the stored kwargs for a payload the trigger just yielded points the author at database state that was never at fault. The sweep's summary counted an unresumable task as resolved. * Update airflow-core/src/airflow/models/trigger.py * Fix ruff D205/D213 on _fail_unresumable_task_instance docstring One-line summary on the second line + blank line before the description, so the docstring satisfies both D205 (blank between summary and description) and D213 (summary on second line). Static checks were failing on ruff for this. --------- (cherry picked from commit 2b7a0be) Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #70685 to
v3-3-testfor the 3.3.1 patch release — this was missed (the main PR merged but no backport was raised).A deferred task whose stored
next_kwargscan't be deserialized/processed was left stranded rather than failed. This routes it through the normal failure path (and the scheduler's awaiting-input timeout sweep survives such task instances instead of crashing).Cherry-picked with
-xfrom2b7a0be359. Applied cleanly — git auto-merged thetest_trigger.pyimports where this and the already-backported #69821 (#71163) both add imports. The fulltest_trigger.pysuite (plus the new scheduler test) passes on v3-3-test — 42 tests green.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the guidelines