Skip to content

[JIT][Bugfix] Make blob codegen cache publication transactional - #4800

Open
Ruye-aa wants to merge 1 commit into
ROCm:mainfrom
Ruye-aa:codex/jit-transactional-codegen
Open

[JIT][Bugfix] Make blob codegen cache publication transactional#4800
Ruye-aa wants to merge 1 commit into
ROCm:mainfrom
Ruye-aa:codex/jit-transactional-codegen

Conversation

@Ruye-aa

@Ruye-aa Ruye-aa commented Aug 17, 2026

Copy link
Copy Markdown

Summary

  • run blob generators in a unique staging directory with checked subprocesses;
  • reject successful generator runs that produce no C++/HIP source;
  • publish generated sources only after compilation succeeds and restore the previous
    source cache if publication fails;
  • atomically replace the installed module so a failed rebuild keeps the previous
    healthy .so;
  • add CPU-only regression tests for interrupted generation and publication.

Problem

build_module() currently writes codegen output directly into the canonical blob/
directory and ignores the os.system() return code. It also removes the installed
module before a replacement has compiled. A failed or interrupted generator can
therefore leave partial sources that are compiled or reused, while also destroying the
last known-good binary.

This was observed on the CK MHA JIT path as a truncated fmha_fwd_api.cpp with no
matching dispatch branch, followed by persistent invalid argument for fmha_fwd
failures until the cache was deleted.

Implementation

The change moves generator execution into aiter/jit/utils/jit_cache.py:

  1. all commands for a module write to one unique staging directory;
  2. subprocess.run(..., check=True) propagates generator failure;
  3. the staging directory must contain at least one .cpp or .cu source;
  4. compilation consumes only the staged files;
  5. the staged source directory replaces blob/ after successful compilation, with
    rollback if directory publication fails;
  6. the compiled module is copied to a temporary sibling and published with
    os.replace().

Tests

python3 -m pytest -q op_tests/test_jit_cache_transaction.py
.....                                                                    [100%]
5 passed

python3 -m py_compile \
  aiter/jit/core.py \
  aiter/jit/utils/jit_cache.py \
  op_tests/test_jit_cache_transaction.py

git diff --check

The regression tests are CPU-only and cover:

  • a generator that writes a partial source and exits non-zero;
  • a zero-source generator result;
  • delayed publication of successful codegen output;
  • rollback when source-directory publication fails;
  • preservation of an existing .so when copying the replacement fails.

GPU compilation and CK MHA runtime validation are left to the upstream CI because the
local development host has no ROCm GPU.

Closes #4799

Signed-off-by: aiyang.ay <aiyang.ay@alibaba-inc.com>
@Ruye-aa
Ruye-aa requested a review from a team August 17, 2026 10:58
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:gfx1250-ffm-triton Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 4800 --add-label <label>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Interrupted JIT codegen can leave and persistently reuse an incomplete CK MHA cache

1 participant