Skip to content

Add maca support - #2

Open
zhangjinnan wants to merge 2 commits into
THeWakeSystems:mainfrom
zhangjinnan:add_maca_support
Open

Add maca support#2
zhangjinnan wants to merge 2 commits into
THeWakeSystems:mainfrom
zhangjinnan:add_maca_support

Conversation

@zhangjinnan

Copy link
Copy Markdown

No description provided.

@xiaolongma88 xiaolongma88 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

问题: using_tf32 会吞掉业务代码的 RuntimeError

engine/src/utils.pyusing_tf32 上下文管理器把整个 yield 包进了 try/except (AttributeError, RuntimeError):

try:
    was_cudnn = torch.backends.cudnn.allow_tf32
    was_matmul = torch.backends.cuda.matmul.allow_tf32
    torch.backends.cudnn.allow_tf32 = enabled
    torch.backends.cuda.matmul.allow_tf32 = enabled
    yield
    torch.backends.cudnn.allow_tf32 = was_cudnn
    torch.backends.cuda.matmul.allow_tf32 = was_matmul
except (AttributeError, RuntimeError):
    yield  # C500 等平台不支持 TF32, 静默跳过

问题所在: 在 NVIDIA 等支持 TF32 的平台上, 属性访问本身不会抛异常, 但 with 块内业务代码抛出的 RuntimeError (例如 CUDA OOM、显存不足——PyTorch 里最常见的错误类型) 会被这个 except 捕获。生成器被 throw() 注入异常后再执行 yield, contextlib 会抛出 RuntimeError: generator didn't stop after throw(), 原始异常被完全掩盖

engine/main.py:311with using_tf32(enabled=True): 包着的正是量化核心计算, 一旦 OOM 或 CUDA error, 排查时只能看到 "generator didn't stop", 原始堆栈全部丢失, 极难定位。

建议: 只保护属性读写, 不要包住 yield, 恢复逻辑放进 finally:

@contextlib.contextmanager
def using_tf32(enabled: bool):
    """安全启用/禁用 TF32。C500 不支持 TF32 时静默跳过。"""
    try:
        was_cudnn = torch.backends.cudnn.allow_tf32
        was_matmul = torch.backends.cuda.matmul.allow_tf32
    except (AttributeError, RuntimeError):
        was_cudnn = was_matmul = None  # C500 等平台不支持 TF32
    if was_cudnn is not None:
        torch.backends.cudnn.allow_tf32 = enabled
        torch.backends.cuda.matmul.allow_tf32 = enabled
    try:
        yield
    finally:
        if was_cudnn is not None:
            torch.backends.cudnn.allow_tf32 = was_cudnn
            torch.backends.cuda.matmul.allow_tf32 = was_matmul

这样 C500 上依旧静默跳过, NVIDIA 上业务异常也能正常向外传播, 且无论块内是否抛异常 TF32 状态都能恢复。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants