fix(model-install): preserve install tmpdir when a single multi-file part fails

[lorenzozanee]

Summary

A single part error in a multi-file install deletes the entire install tmpdir. A transient failure on one file (an HTTP 5xx, or the sidecar rename race from #9432) discards parts that already completed and .downloading partials holding resumable progress, so the next attempt restarts the whole install from zero.

The error path now preserves the tmpdir when it contains anything worth keeping (a completed part file or a non-empty partial): the install is marked paused and the install marker is persisted, so restarting failed files and resume can pick up where the download left off. When nothing on disk is worth keeping, the tmpdir is still removed as before.

Related Issues / Discussions

Closes #9481

QA Instructions

New regression test (fails on the previous HEAD, passes with this change):

pytest tests/app/services/model_install/test_model_install.py::test_multifile_install_part_error_preserves_tmpdir

Surrounding suites and lint:

pytest tests/app/services/model_install/test_model_install.py tests/app/services/download/test_download_queue.py
uvx ruff@0.11.2 check invokeai/app/services/model_install/model_install_default.py tests/app/services/model_install/test_model_install.py

Result: 52 passed (35 model install, 17 download queue), ruff clean.

Merge Plan

Standard merge.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• ❗Changes to a redux slice have a corresponding migration
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

[lstein]

Adversarial review

Reviewed at afb0c20b53. Existing suite is green on the branch (36 passed for tests/app/services/model_install/test_model_install.py). All findings below were reproduced with throwaway probe tests against this branch.

The premise is right and the bug is real — _download_error_callback really did rmtree gigabytes of resumable progress on a single transient failure. But the chosen mechanism, flipping the install job to a non-terminal PAUSED with no event, breaks three contracts at once, and they compound into a permanent wedge. Requesting changes on that basis.

---

Blockers

B1 — wait_for_job() never returns after a download failure

InstallStatus.PAUSED is not in in_terminal_state (model_install_common.py:268-270), and after the new branch runs nothing else advances the job.

Two production callers wait on exactly this:

• invokeai/app/invocations/flux_redux.py:159 — wait_for_job(job, timeout=600) after auto-installing SigLIP
• invokeai/app/invocations/ip_adapter.py:223 — wait_for_job(job, timeout=600) after auto-installing the CLIPVision encoder
• (also invokeai/backend/util/test_utils.py:54, timeout=10)

Before this PR a transient 5xx failed these fast with the download error. Now the invocation blocks for the full 10 minutes and then raises TimeoutError.

This is not confined to multi-file installs. Probe: a single-file install whose connection drops mid-stream leaves a non-empty .downloading, so _tmpdir_has_recoverable_data returns True and the job goes PAUSED:

STATUS: InstallStatus.PAUSED
TMPDIR EXISTS: True
    .invokeai_install.json 590
    only_part.safetensors.downloading 30880

Any ordinary network hiccup on a starter-model download now takes this path.

B2 — the wedge is permanent, and survives a restart

import_model() returns the existing non-terminal job for the same source and refuses to enqueue (model_install_default.py:515-517):

similar_jobs = [job for job in self._install_jobs if job.source == source and not job.in_terminal_state]
if similar_jobs:
    self._logger.warning(f"There is already an active install job for {source}. Not enqueuing.")
    return similar_jobs[0]

Probe confirms the second import_model() of the same source returns the same paused job object. So the retry can never start: every subsequent FLUX Redux / IP-Adapter generation re-enters wait_for_job on the same paused job and burns another 600 s.

Restarting the app does not clear it either — _restore_incomplete_installs() rebuilds the job from the marker as PAUSED and then continues (model_install_default.py:265-266), so the non-terminal job is back in _install_jobs and still blocks import_model. The only escape is a manual cancel or resume in the model manager UI.

B3 — nothing tells the client the install stopped

The new branch emits no event at all. Probe — the complete event list on the bus after the failure:

EVENTS: ['ModelInstallDownloadStartedEvent', 'ModelInstallDownloadProgressEvent']

And there is no other path by which the UI could find out:

• there is no model_install_paused event, and download_paused has no frontend handler
• download_error is log-only (setEventListeners.tsx:316)
• listModelInstalls has no pollingInterval and nothing invalidates the ModelInstalls tag on this path

So the queue row stays frozen at "downloading N%" indefinitely. Previously _set_error → _signal_job_errored → emit_model_install_error gave the row an error status and raised a toast.

B1 + B2 + B3 compound: the only actor who can recover the install is the user, and the user is never told there is anything to recover.

---

Medium

M1 — the new test does not cover the branch that matters

I deleted the entire .downloading half of _tmpdir_has_recoverable_data (leaving only the part.dest.is_file() check) and test_multifile_install_part_error_preserves_tmpdir still passed:

1 passed, 21 warnings in 4.04s

The completed aaa_good_part alone drives the recoverable branch, and assertion (d) holds trivially because the whole tmpdir survives. So the SlowAdapter — a 2-second real time.sleep in the suite — and the seeded partial buy no coverage, and the resumable-partial case from #9481 (the one the issue is actually about) is untested.

A mid-stream break on the sole file of a single-file install exercises the branch with nothing completed, needs no sleep, and is deterministic. Roughly:

class BrokenResp(Resp):
    def __init__(self, stream, status, headers, break_after):
        super().__init__(stream, status, headers)
        self._break_after, self._served = break_after, 0

    def read(self, chunk_size, **kwargs):
        if self._served >= self._break_after:
            raise OSError("connection reset by peer")
        n = min(chunk_size, self._break_after - self._served)
        self._served += n
        return super().read(n)

Then assert the sole part's .downloading survives with st_size > 0.

M2 — a user cancel racing a part error can resurrect the tmpdir

cancel_job() takes no self._lock, while _download_error_callback does. Interleaving:

1. part raises → _mfd_error → _download_error_callback → _tmpdir_has_recoverable_data observes the completed part file → True
2. thread switch; the user's cancel_job() runs _write_install_marker(CANCELLED), _delete_install_marker, _safe_rmtree — tmpdir gone
3. back in the error callback, _write_install_marker(..., PAUSED) runs path.parent.mkdir(parents=True, exist_ok=True) and recreates the directory, containing only a paused marker

_remove_dangling_install_dirs never deletes a paused marker, and _restore_incomplete_installs resurrects it as a phantom paused job on every startup, forever. The old code's _safe_rmtree on an already-removed dir was a harmless no-op; the marker write is not.

Related, same branch: install_job.status = InstallStatus.PAUSED is unconditional and will overwrite a user's CANCELLED. _download_cancelled_callback guards for exactly this (if not install_job.errored and not install_job.paused); this path has no such guard.

M3 — the "HTML page, not a model" diagnostic is bypassed

_set_error special-cases parts with a text/html content type to report "At least one file … is an HTML page, not a model. This can happen when an access token is required to download." That only runs in the else branch now. A repo where one file returns a 200 HTML login page (which completes, so dest.is_file() is True) and another 401s loses the actionable message and silently freezes instead.

---

Minor

• _tmpdir_has_recoverable_data tests completion via part.dest but partials via part.download_path. They coincide today, since _do_download sets job.download_path = job.dest whenever dest isn't a directory (and multifile parts always have a file dest), so this is correct — but part.download_path or part.dest for both would be self-consistent and robust to that invariant changing.
• Not a defect, just confirming I checked it: resume does not re-download the preserved completed parts. It re-issues a GET but short-circuits on the headers at download_default.py:545-549 without reading the body. The preservation does pay off.

---

Suggested direction

Keep the tmpdir, but don't change the job's terminal disposition. In the recoverable branch, still call _set_error(install_job, excp) and self._download_queue.cancel_job(download_job) — skip only the _safe_rmtree — and write the marker with status PAUSED:

if install_job._install_tmpdir is not None and self._tmpdir_has_recoverable_data(download_job):
    # Preserve completed parts and resumable partials for the next attempt, but keep the
    # install job terminal so waiters, retries and the UI all behave as before.
    self._write_install_marker(install_job, status=InstallStatus.PAUSED)
    self._set_error(install_job, excp)
    self._download_queue.cancel_job(download_job)
else:
    ...  # unchanged

That gets everything the PR is after, and none of the fallout:

• the job stays terminal, so wait_for_job returns, import_model allows a retry, and model_install_error still reaches the UI (B1, B2, B3)
• _remove_dangling_install_dirs leaves a paused marker alone, so the directory survives the startup sweep
• _find_reusable_tmpdir skips only COMPLETED/ERROR/CANCELLED markers — so the very next retry of the same source picks the preserved partials back up, which is exactly what #9481 asked for
• _restore_incomplete_installs still offers it as a resumable job after a restart

If you'd rather keep PAUSED as the job status, then B1/B2/B3 each need explicit work: a model_install_paused event plus a frontend handler and cache update, exclusion of auto-paused jobs from the import_model dedup, and an answer for wait_for_job's terminal-state loop.

Happy to re-review once these are addressed — the underlying fix is worth having.