Fail fast on unresolvable master DNS in forked minion workers

[twangboy]

What does this PR do?

A forked per-job worker inherits the minion's default retry_dns=30 / retry_dns_count=None settings. If the job it runs resolves the master DNS (e.g. status.ping_master or master-alive checks) while the master hostname is unresolvable, resolve_dns() enters its unbounded while-True retry loop and the worker never returns.

The loop's abort hook (_RESOLVE_DNS_ABORT, #69466) cannot rescue it: the event is a module-level threading.Event and a forked child holds its own dead copy that the parent's SIGTERM handler can never set. The wedged worker is therefore only killable via SIGKILL, and since SubprocessList cleanup only reaps already-exited processes, hung workers accumulate. A minion pointed at an unresolvable master leaked hundreds of workers and ~10GB of RAM over a couple of days.

Bound DNS retries in the worker preamble (Minion._target): when the operator has not set retry_dns_count, force it to 0 so resolve_dns raises SaltMasterUnresolvableError on the first attempt, the worker exits, and it is reaped normally. An explicit retry_dns_count is still honored. This only shortcuts DNS-resolution failure, never job execution time, so it cannot kill a legitimate long-running job. The main minion process is unchanged (still retries by default; bounded opt-in via retry_dns_count), mirroring the existing failover path that already forces retry_dns=0.

What issues does this PR fix or reference?

Fixes #69659

Merge requirements satisfied?

[NOTICE] Bug fixes or features added to Salt require tests.

• Docs
• Changelog - https://docs.saltproject.io/en/master/topics/development/changelog.html
• Tests written/updated

Commits signed with GPG?

Yes

[sujitdb]

LGTM , lets wait for the test to pass

[twangboy]

Putting this here to document:

The bug wasn't introduced by a single regression. It's the confluence of three separate design decisions that have been sitting there, compounding, since 2016 — no one broke this recently.

Timeline:

2011-04 → 2016-02:
Job workers moved from threads to forked (multiprocessing) processes, default True on POSIX by mid-2011, unified on Windows by Feb 2016. Forked workers inherit the entire parent opts dict via normal fork() semantics — this was never trimmed.

2012-11 / 2013-01:
retry_dns introduced, and resolve_dns() gained its while True: retry-forever shape almost immediately. It retried unboundedly from the very first version of this behavior — there was never a "fail once" era to regress from.

2016-03-17 (942b71b, ~Salt 2016.3.0)
status.ping_master started calling resolve_dns() from inside job-worker code. This is the commit that made the bug possible — before it, nothing running inside a forked worker called into the infinite retry loop.

2018-09 (shipped 2018.3.4)
retry_dns_count added, giving an opt-in way to bound the retries — but defaulted (and still defaults) to None/unbounded. It made the footgun avoidable, not safe-by-default.

2026-07-02 (today)
Our fix — forked workers now force retry_dns_count=0 unless the operator explicitly set one.

This has been a live, exploitable gap since Salt 2016.3.0 — about a decade. It didn't regress from previously-correct behavior; resolve_dns() has retried forever by default since the feature was born in 2012–2013, and once status.ping_master (2016) gave a forked worker a reason to call it, the leak became possible on any minion whose master DNS goes bad for long enough. retry_dns_count (2018) offered a manual escape hatch, but nothing made workers use it by default until today's fix.

One caveat from the investigation: the trace confirms status.ping_master as a trigger reachable from a worker, but didn't exhaustively check every module for other resolve_dns() call sites — there may be other, equally old trigger paths I haven't enumerated.