Skip to content

Fix SonicMoE JIT lock warm-cache path - #47

Open
A-nnonymous wants to merge 1 commit into
PFCCLab:paddlefrom
A-nnonymous:fix-jit-lock-warm-cache
Open

Fix SonicMoE JIT lock warm-cache path#47
A-nnonymous wants to merge 1 commit into
PFCCLab:paddlefrom
A-nnonymous:fix-jit-lock-warm-cache

Conversation

@A-nnonymous

Copy link
Copy Markdown
Collaborator

Summary

This PR fixes a multi-rank SonicMoE JIT lock stall when a prebuilt extension already exists.

Previously, all ranks entered an exclusive FileLock before attempting to import an existing .so. If one rank stalled while holding the lock during import/dlopen/build-related work, other ranks kept waiting even though the extension artifact was already available.

The fix:

  • imports existing prebuilt extensions before acquiring the build lock;
  • keeps FileLock only around the producer/build path;
  • exits the lock before importing artifacts produced by another process;
  • adds SONIC_MOE_JIT_LOCK_TIMEOUT with a default of 600 seconds.

Validation

  • python -m py_compile sonicmoe/jit.py passed.
  • git diff --check passed.
  • The source file and PaddleFleet runtime symlink copy were verified identical locally.
  • The fix was previously validated in xiaoyao's allgather runtime where the SonicMoE lock spam disappeared and execution progressed past the JIT lock point.

🤖 Generated with Claude Code

Avoid serializing read-only prebuilt extension imports behind the build lock. Keep the lock for actual builds only, add a bounded lock timeout, and import artifacts outside the lock when another process finishes the build while waiters are blocked.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants