Skip to content

Scheduler crashes with DetachedInstanceError in adopt_or_reset_orphaned_tasksrepr(ti) lazy-loads state on a detached TaskInstance #71272

Description

@akhilpratap1991

Under which category would you file this issue?

Airflow Core

Apache Airflow version

3.2.2

What happened and how to reproduce it?

What happened

The scheduler enters an unrecoverable crash loop. On startup and on every
[scheduler] orphaned_tasks_check_interval, adopt_or_reset_orphaned_tasks()
builds a log message by calling repr(ti) on each orphaned TaskInstance.
TaskInstance.__repr__ reads self.state, which is a deferred column. When the
TaskInstance is detached from its SQLAlchemy session, that read triggers a
deferred DB load with no bound session and raises DetachedInstanceError. The
exception is unhandled in _run_scheduler_loop, so the scheduler process exits.

Because the scheduler dies before it finishes resetting the orphaned task, the
same orphan is still present on the next start and it crashes again — a
self-perpetuating loop. Nothing outside the scheduler clears the orphan, so it
never recovers without manual DB intervention. While it loops, no DAGs are
scheduled at all.

Traceback (paths are from the official apache/airflow image):

[error ] Exception when executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
  File ".../airflow/jobs/scheduler_job_runner.py", line 1546, in _execute
    ret = execute_callable()
  File ".../airflow/jobs/scheduler_job_runner.py", line 1612, in _run_scheduler_loop
    self.adopt_or_reset_orphaned_tasks()
  File ".../airflow/utils/session.py", line 100, in wrapper
    return func(*args, session=session, **kwargs)
  File ".../airflow/jobs/scheduler_job_runner.py", line 2814, in adopt_or_reset_orphaned_tasks
    for attempt in run_with_db_retries(logger=self.log):
  ...
  File ".../airflow/jobs/scheduler_job_runner.py", line 2864, in adopt_or_reset_orphaned_tasks
    reset_tis_message.append(repr(ti))
  File ".../airflow/models/taskinstance.py", line 1134, in __repr__
    return prefix + f"[{self.state}] ti_id={self.id}>"
  File ".../sqlalchemy/orm/attributes.py", line 569, in __get__
    return self.impl.get(state, dict_)
  ...
  File ".../sqlalchemy/orm/strategies.py", line 536, in _load_for_state
    raise orm_exc.DetachedInstanceError(
sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <TaskInstance at 0x...>
is not bound to a Session; deferred load operation of attribute 'state' cannot
proceed (Background on this error at: https://sqlalche.me/e/20/bhk3)

### How to reproduce

Using KubernetesExecutor:

1. Start a task so a `TaskInstance` is `running`/`queued` under a worker pod.
2. Cause it to become "orphaned" — e.g. the worker pod is lost, or the
   scheduler is killed while the task is in flight (SIGKILL / OOM / node drain)
   so the owning job is gone.
3. Start the scheduler. `adopt_or_reset_orphaned_tasks()` selects the orphaned
   `TaskInstance`(s); when it reaches `reset_tis_message.append(repr(ti))` the
   deferred `state` load fails with `DetachedInstanceError` and the scheduler
   exits. It then crash-loops on every restart because the orphan persists.

Manual workaround: set the stuck task instances to a terminal state directly in
the metadata DB (or via the UI "Mark Failed"/CLI from a non-scheduler pod), then
restart the scheduler.

### What you think should happen instead?

Adopting/resetting orphaned tasks must not be able to crash the scheduler, and
in particular building a diagnostic log line must never trigger a lazy DB load
on a possibly-detached instance. `adopt_or_reset_orphaned_tasks()` should either:

- construct the message from the task instance identity/key (already-loaded
  columns) instead of `repr(ti)`, or
- make `TaskInstance.__repr__` read only already-loaded attributes (e.g. from
  `__dict__`) so it degrades gracefully (`state=?`) rather than raising, or
- keep the instances bound/refreshed within the session while the message is
  built.

The task instances should still be reset as intended; only the fatal log call
needs to change.

### Operating System

Debian (official `apache/airflow:slim-3.2.2-python3.12` image), Python 3.12.

### Deployment

Other

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes (KubernetesExecutor). Reproducible with
the version pinned by the 3.2.2 constraints file.


### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

Happens repeatedly — every scheduler restart while an orphaned task instance
exists. Related (same code path / theme, but not the same `__repr__`/`state`
lazy-load crash): #30709, #39088, #19671, #58570, #13808.

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:Schedulerincluding HA (high availability) schedulerarea:corekind:bugThis is a clearly a bugneeds-triagelabel for new issues that we didn't triage yet

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions