Problem
The task can be requeued three times. A worker can then claim it for a fourth attempt and disappear again. On the next recovery sweep, requeue_stalled_tasks() archives its PGMQ message and sets step_tasks.permanently_stalled_at, but leaves step_tasks.status = 'started'. In a one-step flow, step_states.status and runs.status also remain started, with no message left to drive another transition. In a larger flow, dependent steps can remain blocked.
This is the current behavior in the recovery function and its max-requeue test. It means whenExhausted: 'fail', 'skip', or 'skip-cascade' is never applied to this failure mode, even though fail_task() already owns those transitions.
I see that the troubleshooting guide deliberately leaves the status started for investigation. The timestamp and requeue count would still identify the incident if the task became terminal; keeping it active also leaves the flow unable to finish or run a dependent cleanup step.
Suggested direction
Keep permanently_stalled_at as the diagnostic marker, but hand stamped tasks to fail_task() after recovery commits. For each candidate, lock and recheck the run, step, and task; raise attempts_count to at least the step's effective max_attempts; then call fail_task(run_id, step_slug, task_index, 'permanently stalled'). That lets the existing code decide whether to fail the run, skip the step, cascade a skip, start dependents, and cancel or archive siblings. The pass should also pick up already-stamped rows from earlier deployments and be safe to repeat.
I would keep this as a separate database call or scheduled job, rather than calling fail_task() inside requeue_stalled_tasks(). Recovery locks step_tasks first, while fail_task() locks the run and step before updating the task. Holding the recovery locks while entering fail_task() creates a lock-order inversion with live workers. The recovery call should commit before escalation begins.
We implemented and tested this approach in the Elixir port's 0.5.0 commit. The relevant pieces are the V06 SQL function, separate recovery calls, and tests for fail, skip, skip-cascade, map siblings, late completion, and previously stamped rows.
Would you be open to making a permanently stalled task terminal through the existing exhaustion policy while retaining the timestamp for investigation?
Problem
The task can be requeued three times. A worker can then claim it for a fourth attempt and disappear again. On the next recovery sweep,
requeue_stalled_tasks()archives its PGMQ message and setsstep_tasks.permanently_stalled_at, but leavesstep_tasks.status = 'started'. In a one-step flow,step_states.statusandruns.statusalso remainstarted, with no message left to drive another transition. In a larger flow, dependent steps can remain blocked.This is the current behavior in the recovery function and its max-requeue test. It means
whenExhausted: 'fail','skip', or'skip-cascade'is never applied to this failure mode, even thoughfail_task()already owns those transitions.I see that the troubleshooting guide deliberately leaves the status
startedfor investigation. The timestamp and requeue count would still identify the incident if the task became terminal; keeping it active also leaves the flow unable to finish or run a dependent cleanup step.Suggested direction
Keep
permanently_stalled_atas the diagnostic marker, but hand stamped tasks tofail_task()after recovery commits. For each candidate, lock and recheck the run, step, and task; raiseattempts_countto at least the step's effectivemax_attempts; then callfail_task(run_id, step_slug, task_index, 'permanently stalled'). That lets the existing code decide whether to fail the run, skip the step, cascade a skip, start dependents, and cancel or archive siblings. The pass should also pick up already-stamped rows from earlier deployments and be safe to repeat.I would keep this as a separate database call or scheduled job, rather than calling
fail_task()insiderequeue_stalled_tasks(). Recovery locksstep_tasksfirst, whilefail_task()locks the run and step before updating the task. Holding the recovery locks while enteringfail_task()creates a lock-order inversion with live workers. The recovery call should commit before escalation begins.We implemented and tested this approach in the Elixir port's 0.5.0 commit. The relevant pieces are the V06 SQL function, separate recovery calls, and tests for fail, skip, skip-cascade, map siblings, late completion, and previously stamped rows.
Would you be open to making a permanently stalled task terminal through the existing exhaustion policy while retaining the timestamp for investigation?