What happened?
Description:
When a minion cannot resolve its master's hostname in DNS for a sustained period, forked per-job worker processes can hang forever in the DNS-resolution retry loop instead of failing and exiting. Because a new worker is spawned for each scheduled/periodic job during the outage and none of the stuck ones ever exit, worker processes accumulate without bound and exhaust system memory.
In one observed case, a minion configured with an unresolvable master (master: salt) accumulated 674 leaked worker processes consuming ~10.4 GB of RAM (~71% of host memory) over roughly two days, with process start times spread evenly across the entire outage window. The resulting memory pressure drove the kernel into continuous swap reclaim (kswapd0 pinned at ~80% CPU with hundreds of cumulative CPU-hours). The minion log contained millions of repetitions of DNS lookup or connection check of 'salt' failed.
Root cause
salt.minion.resolve_dns() retries dns_check() in a while True: loop every retry_dns seconds. The loop is only bounded if retry_dns_count is set to a number; otherwise it retries indefinitely. The shipped defaults are:
- retry_dns: 30 (retry loop active)
- retry_dns_count: None (unbounded)
A forked per-job worker inherits these defaults. If the job function it runs calls resolve_dns — for example status.ping_master, or the master-alive checks — and the master is unresolvable, the worker enters the unbounded retry loop and never returns.
The loop has an abort mechanism (_RESOLVE_DNS_ABORT, a module-level threading.Event set by the parent's SIGTERM handler, added in #69466), but it does not work for workers: a forked child holds its own dead copy of the event that the parent can never set. As a result, a wedged worker cannot be interrupted by SIGTERM and is only cleaned up by SIGKILL. Since the normal subprocess cleanup path (SubprocessList.cleanup()) only reaps processes that have already exited on their own, these hung workers are never reaped and simply pile up.
Notably, the failover code path already avoids this: eval_master() forces retry_dns = 0 for master_type: failover. The default str/multimaster path does not, so it remains exposed.
Steps to reproduce
- Configure a minion with a master hostname that does not resolve in DNS (e.g. master: salt with no corresponding DNS record), using the default retry_dns / retry_dns_count settings.
- Ensure the minion periodically runs a job that resolves the master DNS (e.g. a scheduled job calling status.ping_master, or master-alive checks).
- Leave the minion running for an extended period.
Expected behavior
A worker that cannot resolve the master should fail promptly (raising an error for that job) and exit, so the process is reaped and no processes accumulate.
Actual behavior
Worker processes hang indefinitely in the DNS retry loop, are not reaped, and accumulate over time until the host runs out of memory.
Type of salt install
Official deb
Major version
3006.x
What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)
amazonlinux-2023, debian-11, debian-12
salt --versions-report output
What happened?
Description:
When a minion cannot resolve its master's hostname in DNS for a sustained period, forked per-job worker processes can hang forever in the DNS-resolution retry loop instead of failing and exiting. Because a new worker is spawned for each scheduled/periodic job during the outage and none of the stuck ones ever exit, worker processes accumulate without bound and exhaust system memory.
In one observed case, a minion configured with an unresolvable master (master: salt) accumulated 674 leaked worker processes consuming ~10.4 GB of RAM (~71% of host memory) over roughly two days, with process start times spread evenly across the entire outage window. The resulting memory pressure drove the kernel into continuous swap reclaim (kswapd0 pinned at ~80% CPU with hundreds of cumulative CPU-hours). The minion log contained millions of repetitions of DNS lookup or connection check of 'salt' failed.
Root cause
salt.minion.resolve_dns() retries dns_check() in a while True: loop every retry_dns seconds. The loop is only bounded if retry_dns_count is set to a number; otherwise it retries indefinitely. The shipped defaults are:
A forked per-job worker inherits these defaults. If the job function it runs calls resolve_dns — for example status.ping_master, or the master-alive checks — and the master is unresolvable, the worker enters the unbounded retry loop and never returns.
The loop has an abort mechanism (_RESOLVE_DNS_ABORT, a module-level threading.Event set by the parent's SIGTERM handler, added in #69466), but it does not work for workers: a forked child holds its own dead copy of the event that the parent can never set. As a result, a wedged worker cannot be interrupted by SIGTERM and is only cleaned up by SIGKILL. Since the normal subprocess cleanup path (SubprocessList.cleanup()) only reaps processes that have already exited on their own, these hung workers are never reaped and simply pile up.
Notably, the failover code path already avoids this: eval_master() forces retry_dns = 0 for master_type: failover. The default str/multimaster path does not, so it remains exposed.
Steps to reproduce
Expected behavior
A worker that cannot resolve the master should fail promptly (raising an error for that job) and exit, so the process is reaped and no processes accumulate.
Actual behavior
Worker processes hang indefinitely in the DNS retry loop, are not reaped, and accumulate over time until the host runs out of memory.
Type of salt install
Official deb
Major version
3006.x
What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)
amazonlinux-2023, debian-11, debian-12
salt --versions-report output