feat(slurm): harden distributed allocation runtime

[nabinchha]

📋 Summary

This completes the distributed-execution and failure-hardening slice of the Slurm allocation runtime while composing the allocation-local client worker merged through #911.

PRs #908, #909, #910, and #911 have merged. This branch is rebased directly onto the current feat/slurm-execution base at b98c043e after #910.

🔗 Related Issue

Closes #868

🔄 Changes

• resolve scheduler allocation hosts and launch one coordinated srun task per physical node for each deployment
• compose node-local lane workers with deterministic rank, GPU, rendezvous, stagger, and fail-fast cleanup behavior
• route remote lane-head readiness through client-host logical endpoints while preserving replica retry semantics
• enforce per-lane queue backpressure with bounded, fail-open metric sampling and typed 429/Retry-After behavior
• run client and per-node preflight checks before model launch and preserve one idempotent allocation cleanup owner
• split runtime step, endpoint, server, node-spec, and worker responsibilities into focused public modules
• cover multi-node execution, follower failure, cancellation, signals, readiness timeouts, partial startup, cleanup re-entry, backpressure, and no-orphan behavior

🧪 Testing

• make test passes (full repository suite not run)
• make test-slurm — 1,271 passed after rebasing onto merged feat: finalize Slurm shard winners #910
• make check-slurm
• focused affected runtime tests — 48 passed
• git diff --check
• Unit tests added/updated
• E2E tests added/updated (real-cluster proof remains in the joint acceptance lane)

✅ Checklist

• Follows commit message conventions
• Commits are signed off (DCO)
• Architecture docs updated (not applicable; implements the current reviewed architecture)

---

Description updated with AI

[greptile-apps]

Greptile Summary

This PR extends the Slurm allocation runtime from one-node orchestration to coordinated multi-node execution and hardens failure handling.

• Resolves allocation hosts and launches node-local workers with deterministic process and GPU placement.
• Adds server preflight, remote readiness probing, logical endpoint routing, and coordinated cleanup.
• Adds bounded queue admission with fail-open metric sampling and typed overload responses.
• Splits runtime construction, endpoint, server, node-spec, and worker responsibilities into focused modules.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
packages/data-designer-slurm/src/data_designer/slurm/runtime/controller.py	Generalizes allocation orchestration to verified multi-node layouts, node-level server steps, remote readiness checks, and client-host placement.
packages/data-designer-slurm/src/data_designer/slurm/runtime/backpressure.py	Adds queue-depth sampling and ASGI admission control; the previously reported ordinary-reader-failure lifecycle issue is fixed by converting failed samples into recoverable unavailable snapshots.
packages/data-designer-slurm/src/data_designer/slurm/runtime/node_worker.py	Adds node-local lane supervision with deterministic GPU assignment, staggered launch, signal handling, and fail-fast cleanup.
packages/data-designer-slurm/src/data_designer/slurm/runtime/server_steps.py	Builds coordinated per-node preflight and serving steps from validated deployment specifications.
packages/data-designer-slurm/src/data_designer/slurm/runtime/preflight.py	Resolves and verifies the scheduler allocation layout before distributed runtime steps are launched.

Sequence Diagram

sequenceDiagram
    participant C as AllocationController
    participant P as AllocationPreflight
    participant S as Slurm srun
    participant N as Node workers
    participant E as Logical endpoint
    participant G as Client worker
    C->>P: Verify allocation and resolve hosts
    P-->>C: AllocationLayout
    C->>S: Run client and server preflight steps
    S->>N: Validate node identity, GPUs, and ports
    N-->>C: Preflight complete
    C->>S: Launch one coordinated server task per node
    S->>N: Start assigned vLLM lanes
    C->>N: Probe remote backend readiness
    C->>S: Launch endpoint on client host
    S->>E: Route requests to remote lane heads
    C->>G: Start generation with logical endpoints
    G->>E: Send inference requests
    E->>N: Retry across replicas
    C->>S: Stop all managed steps during completion or failure

Loading

_{Reviews (6): Last reviewed commit: "fix Slurm queue sampler recovery" | Re-trigger Greptile}

[greptile-apps]

Sampler failure disables backpressure

If a vLLM or Prometheus metrics collector raises during sampling, the exception terminates this daemon thread while _thread remains non-None, so later requests cannot restart it. Once the cached snapshot becomes stale, admission permanently fails open and the configured queue limit stops producing 429 responses.

Suggested change

	def _sample_forever(self) -> None:
	while True:
	self.sample_once()
	time.sleep(self.settings.poll_interval_seconds)
	def _sample_forever(self) -> None:
	while True:
	try:
	self.sample_once()
	except Exception:
	pass
	time.sleep(self.settings.poll_interval_seconds)

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/data-designer-slurm/src/data_designer/slurm/runtime/backpressure.py
Line: 120-123

Comment:
**Sampler failure disables backpressure**

If a vLLM or Prometheus metrics collector raises during sampling, the exception terminates this daemon thread while `_thread` remains non-`None`, so later requests cannot restart it. Once the cached snapshot becomes stale, admission permanently fails open and the configured queue limit stops producing 429 responses.

```suggestion
    def _sample_forever(self) -> None:
        while True:
            try:
                self.sample_once()
            except Exception:
                pass
            time.sleep(self.settings.poll_interval_seconds)
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[nabinchha]

Fixed in 77933cc. QueueBackpressureController.sample_once now converts ordinary reader failures into a fresh unavailable snapshot, so admission fails open for that sample and the daemon continues polling; the next successful sample restores queue-limit rejection. BaseException is intentionally not caught, preserving process-control and shutdown signals. Added regression coverage for failure, fail-open behavior, recovery, and KeyboardInterrupt propagation. Validation: 5 focused tests and 1,235 full Slurm tests passed; check-slurm and focused strict complexity checks pass.