Add maca support - #2
Open
zhangjinnan wants to merge 2 commits into
Open
Conversation
xiaolongma88
reviewed
Aug 5, 2026
xiaolongma88
left a comment
Collaborator
There was a problem hiding this comment.
问题: using_tf32 会吞掉业务代码的 RuntimeError
engine/src/utils.py 中 using_tf32 上下文管理器把整个 yield 包进了 try/except (AttributeError, RuntimeError):
try:
was_cudnn = torch.backends.cudnn.allow_tf32
was_matmul = torch.backends.cuda.matmul.allow_tf32
torch.backends.cudnn.allow_tf32 = enabled
torch.backends.cuda.matmul.allow_tf32 = enabled
yield
torch.backends.cudnn.allow_tf32 = was_cudnn
torch.backends.cuda.matmul.allow_tf32 = was_matmul
except (AttributeError, RuntimeError):
yield # C500 等平台不支持 TF32, 静默跳过问题所在: 在 NVIDIA 等支持 TF32 的平台上, 属性访问本身不会抛异常, 但 with 块内业务代码抛出的 RuntimeError (例如 CUDA OOM、显存不足——PyTorch 里最常见的错误类型) 会被这个 except 捕获。生成器被 throw() 注入异常后再执行 yield, contextlib 会抛出 RuntimeError: generator didn't stop after throw(), 原始异常被完全掩盖。
而 engine/main.py:311 的 with using_tf32(enabled=True): 包着的正是量化核心计算, 一旦 OOM 或 CUDA error, 排查时只能看到 "generator didn't stop", 原始堆栈全部丢失, 极难定位。
建议: 只保护属性读写, 不要包住 yield, 恢复逻辑放进 finally:
@contextlib.contextmanager
def using_tf32(enabled: bool):
"""安全启用/禁用 TF32。C500 不支持 TF32 时静默跳过。"""
try:
was_cudnn = torch.backends.cudnn.allow_tf32
was_matmul = torch.backends.cuda.matmul.allow_tf32
except (AttributeError, RuntimeError):
was_cudnn = was_matmul = None # C500 等平台不支持 TF32
if was_cudnn is not None:
torch.backends.cudnn.allow_tf32 = enabled
torch.backends.cuda.matmul.allow_tf32 = enabled
try:
yield
finally:
if was_cudnn is not None:
torch.backends.cudnn.allow_tf32 = was_cudnn
torch.backends.cuda.matmul.allow_tf32 = was_matmul这样 C500 上依旧静默跳过, NVIDIA 上业务异常也能正常向外传播, 且无论块内是否抛异常 TF32 状态都能恢复。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.