Is there an existing issue for this problem?
Install method
Invoke's Launcher
Operating system
Windows
GPU vendor
Nvidia (CUDA)
GPU model
RTX 5090
GPU VRAM
32GB
Version number
6.14.0
Browser
Firefox 154.0.1 (64-bit)
System Information
- Case : Corsair 5000D Airflow - White
- Motherboard : MSI X670E - Gaming Plus WIFI
- CPU : AMD Ryzen 9 7950X 5GHz - All Cores
- GPU : MSI Surprim SOC RTX 5090 32GB, MHz (3000Mhz)
- RAM : 128GB Corsair Vengeance 6000MHz DDR5 (3600MHz because of Low Speed 4 Dimm limitation)
- Storage :
- 1x Gen 5 2TB Crucial T700 M.2 NVME SSD (Gen 4)
- 3x Gen 4 2TB Crucial P3P M.2 NVME SSD
- PSU : Corsair RM1200x Shift Fully Modular ATX 3.0 - 80 PLUS Gold - White
- Cooling : All fans are Noctuas
- OS : Windows 11 Pro
- Overclocked System : Yes
What happened
On affected Windows/NVIDIA RTX systems, InvokeAI can hit a lower-level CUDA/driver failure when pushing generation near the card’s practical workload boundary, especially during sustained high-resolution generation or upscaling.
In my testing, this is not usually triggered by a single small generation. The failure appears when the workload is heavy enough to saturate the GPU in both VRAM usage and memory bandwidth for a sustained period, particularly when generating or upscaling images at 2k and above.
The visible symptom is often not a clean InvokeAI exception. It may appear as one of the following:
Display driver nvlddmkm stopped responding
GPU device lost
black screen / driver reset
CUDA launch timeout
Windows TDR reset
This appears to happen around the boundary where the card is close to its practical VRAM/CUDA workload limit. In some cases, the card can handle the workload in principle, but the CUDA runtime/driver path becomes fragile and fails before InvokeAI can return a normal out-of-memory or recoverable error.
I am not claiming that CUDA Graphs are proven to be the root cause in InvokeAI. The request is for diagnostic/stability toggles that would let users test whether the failure is tied to graph-captured, fused, or aggressive PyTorch CUDA execution paths.
InvokeAI already exposes related settings such as:
attention_type: normal
pytorch_cuda_alloc_conf: "backend:native"
But there does not appear to be a config-level way to disable TorchInductor/Triton CUDA Graphs or force PyTorch SDPA into its conservative math-only backend.
What you expected to happen
I expected InvokeAI to either complete the generation/upscale successfully or fail cleanly with a recoverable CUDA/VRAM error.
I also expected there to be a supported diagnostic mode for Windows/NVIDIA CUDA stability issues, similar in spirit to how llama.cpp users can test CUDA Graphs-related instability with:
set GGML_CUDA_DISABLE_GRAPHS=1
Since InvokeAI uses Diffusers/PyTorch rather than llama.cpp/ggml, that exact variable does not apply. The closest equivalent would be an InvokeAI-supported way to force PyTorch away from graph-captured, fused, or aggressive CUDA runtime paths.
For example, it would be useful to expose settings like:
# Disable TorchInductor/Triton CUDA Graphs if torch.compile or Inductor paths are used.
torch_inductor_cudagraphs: false
# Force PyTorch scaled-dot-product attention to the conservative math backend.
torch_sdp_force_math: true
Or a broader stability mode:
cuda_conservative_mode: true
which could internally apply the safest available PyTorch CUDA settings.
How to reproduce the problem
A typical reproduction pattern on affected systems:
- Use InvokeAI on Windows with an NVIDIA RTX GPU.
- Generate normally at lower resolutions.
- Upscale, or generate at 2k and above.
- Push close to the card’s practical VRAM/workload boundary.
- Instead of a clean OOM or recoverable error, the run may trigger a driver/TDR-style failure:
- screen goes black,
- driver resets,
- InvokeAI loses the CUDA device,
- Windows logs an nvlddmkm/TDR event,
- or PyTorch reports a CUDA launch/device failure.
The failure seems more likely during high-resolution output or upscaling paths than during small generations.
The hypothesis is that one of the optimized CUDA paths becomes unstable near the hardware/driver boundary. Possible suspects include:
- fused attention kernels,
- Torch SDPA backend selection,
- TorchInductor/Triton CUDA Graphs if active,
- CUDA allocator behavior,
- NVIDIA driver/WDDM behavior.
Additional context
The reason I am suspecting this class of issue is that I recently reproduced a similar mysterious GPU crash while working with llama.cpp on the same system. In that case, the relevant workaround was to disable llama.cpp CUDA Graphs with:
set GGML_CUDA_DISABLE_GRAPHS=1
That experience made me revisit the crashes I had previously seen with InvokeAI. InvokeAI does not use llama.cpp/ggml for generation, so that exact environment variable does not apply. However, the failure pattern feels similar enough that I think it is worth exposing equivalent conservative PyTorch CUDA runtime toggles for debugging.
InvokeAI’s relevant runtime stack is roughly:
InvokeAI
→ Diffusers
→ PyTorch
→ PyTorch CUDA backend
→ NVIDIA driver / Windows WDDM TDR
My hypothesis is that this may be another case where pushing the GPU near the boundary of what the hardware can practically sustain exposes instability in an optimized CUDA execution path. This may or may not be fixable entirely in InvokeAI, especially if the underlying issue is in PyTorch, CUDA, NVIDIA drivers, Windows WDDM/TDR behavior, or this GPU generation’s driver/runtime support. But having conservative runtime toggles would make the issue much easier to isolate.
Important warning: on my system, repeated CUDA/kernel-path failures appear to make the NVIDIA driver/runtime progressively less stable during the same session. After one or more failures, lighter workloads that were previously stable may begin triggering the same failure. In practice, this means the first failure is the most useful diagnostic event; repeatedly reproducing the issue can make the system state noisier and less reliable.
I do not mean that the driver is permanently corrupted. The observed behavior is that the driver/runtime state appears degraded after failures until a reboot, driver reset, or clean recovery (I would heavily recommend a complete Nvidia driver cleanup with DDU in safe mode after the crash as your pc may not recover and you may have to hard power reset). Because of that, I would recommend capturing logs from at least one failure, but avoiding repeated stress attempts unless necessary.
My system settings are otherwise stable. The machine is not generally unstable under normal use, and lower-resolution InvokeAI generations are stable until the workload approaches the high-resolution/upscaling boundary described above. This appears to be specific to the CUDA/driver/runtime path under heavy InvokeAI generation or upscaling load, rather than a general system instability issue.
I am opening this as a bug report. However, I cannot run InvokeAI at this moment in time because the system is currently occupied with another long-running AI workload that is running 24/7, but I wanted to document the failure pattern and the requested diagnostic toggles while the llama.cpp CUDA Graphs workaround context is fresh in mind. I can add exact version info later if needed. Logs are unfortunately lost because the crash seems to roll the pc back a few mins strangely enough. This is a long standing bug and it is not necessarily invoke specific, but having the exposed settings to do the workaround would aid in overcoming this potential cuda bug that affects multiple cuda workloads once you do more than just gaming.
This would be useful both as a workaround and as a debugging tool. Users could test a configuration like:
attention_type: normal
pytorch_cuda_alloc_conf: "backend:native"
torch_sdp_force_math: true
torch_inductor_cudagraphs: false
If the crash disappears under those settings, maintainers would get a stronger signal that the problem is likely in the CUDA attention/runtime path rather than model files, workflow graph, scheduler, UI, general VRAM pressure, or unstable system settings.
Discord username
@sus_one - Advanced バカ Technology
Is there an existing issue for this problem?
Install method
Invoke's Launcher
Operating system
Windows
GPU vendor
Nvidia (CUDA)
GPU model
RTX 5090
GPU VRAM
32GB
Version number
6.14.0
Browser
Firefox 154.0.1 (64-bit)
System Information
What happened
On affected Windows/NVIDIA RTX systems, InvokeAI can hit a lower-level CUDA/driver failure when pushing generation near the card’s practical workload boundary, especially during sustained high-resolution generation or upscaling.
In my testing, this is not usually triggered by a single small generation. The failure appears when the workload is heavy enough to saturate the GPU in both VRAM usage and memory bandwidth for a sustained period, particularly when generating or upscaling images at 2k and above.
The visible symptom is often not a clean InvokeAI exception. It may appear as one of the following:
This appears to happen around the boundary where the card is close to its practical VRAM/CUDA workload limit. In some cases, the card can handle the workload in principle, but the CUDA runtime/driver path becomes fragile and fails before InvokeAI can return a normal out-of-memory or recoverable error.
I am not claiming that CUDA Graphs are proven to be the root cause in InvokeAI. The request is for diagnostic/stability toggles that would let users test whether the failure is tied to graph-captured, fused, or aggressive PyTorch CUDA execution paths.
InvokeAI already exposes related settings such as:
But there does not appear to be a config-level way to disable TorchInductor/Triton CUDA Graphs or force PyTorch SDPA into its conservative math-only backend.
What you expected to happen
I expected InvokeAI to either complete the generation/upscale successfully or fail cleanly with a recoverable CUDA/VRAM error.
I also expected there to be a supported diagnostic mode for Windows/NVIDIA CUDA stability issues, similar in spirit to how llama.cpp users can test CUDA Graphs-related instability with:
Since InvokeAI uses Diffusers/PyTorch rather than llama.cpp/ggml, that exact variable does not apply. The closest equivalent would be an InvokeAI-supported way to force PyTorch away from graph-captured, fused, or aggressive CUDA runtime paths.
For example, it would be useful to expose settings like:
Or a broader stability mode:
which could internally apply the safest available PyTorch CUDA settings.
How to reproduce the problem
A typical reproduction pattern on affected systems:
The failure seems more likely during high-resolution output or upscaling paths than during small generations.
The hypothesis is that one of the optimized CUDA paths becomes unstable near the hardware/driver boundary. Possible suspects include:
Additional context
The reason I am suspecting this class of issue is that I recently reproduced a similar mysterious GPU crash while working with llama.cpp on the same system. In that case, the relevant workaround was to disable llama.cpp CUDA Graphs with:
That experience made me revisit the crashes I had previously seen with InvokeAI. InvokeAI does not use llama.cpp/ggml for generation, so that exact environment variable does not apply. However, the failure pattern feels similar enough that I think it is worth exposing equivalent conservative PyTorch CUDA runtime toggles for debugging.
InvokeAI’s relevant runtime stack is roughly:
My hypothesis is that this may be another case where pushing the GPU near the boundary of what the hardware can practically sustain exposes instability in an optimized CUDA execution path. This may or may not be fixable entirely in InvokeAI, especially if the underlying issue is in PyTorch, CUDA, NVIDIA drivers, Windows WDDM/TDR behavior, or this GPU generation’s driver/runtime support. But having conservative runtime toggles would make the issue much easier to isolate.
Important warning: on my system, repeated CUDA/kernel-path failures appear to make the NVIDIA driver/runtime progressively less stable during the same session. After one or more failures, lighter workloads that were previously stable may begin triggering the same failure. In practice, this means the first failure is the most useful diagnostic event; repeatedly reproducing the issue can make the system state noisier and less reliable.
I do not mean that the driver is permanently corrupted. The observed behavior is that the driver/runtime state appears degraded after failures until a reboot, driver reset, or clean recovery (I would heavily recommend a complete Nvidia driver cleanup with DDU in safe mode after the crash as your pc may not recover and you may have to hard power reset). Because of that, I would recommend capturing logs from at least one failure, but avoiding repeated stress attempts unless necessary.
My system settings are otherwise stable. The machine is not generally unstable under normal use, and lower-resolution InvokeAI generations are stable until the workload approaches the high-resolution/upscaling boundary described above. This appears to be specific to the CUDA/driver/runtime path under heavy InvokeAI generation or upscaling load, rather than a general system instability issue.
I am opening this as a bug report. However, I cannot run InvokeAI at this moment in time because the system is currently occupied with another long-running AI workload that is running 24/7, but I wanted to document the failure pattern and the requested diagnostic toggles while the llama.cpp CUDA Graphs workaround context is fresh in mind. I can add exact version info later if needed. Logs are unfortunately lost because the crash seems to roll the pc back a few mins strangely enough. This is a long standing bug and it is not necessarily invoke specific, but having the exposed settings to do the workaround would aid in overcoming this potential cuda bug that affects multiple cuda workloads once you do more than just gaming.
This would be useful both as a workaround and as a debugging tool. Users could test a configuration like:
If the crash disappears under those settings, maintainers would get a stronger signal that the problem is likely in the CUDA attention/runtime path rather than model files, workflow graph, scheduler, UI, general VRAM pressure, or unstable system settings.
Discord username
@sus_one - Advanced バカ Technology