feat: reconcile persisted Slurm state

[nabinchha]

📋 Summary

Reconcile a persisted Slurm run from normalized active-queue and accounting observations in a fresh process. The status composition preserves bounded accounting lag and terminal evidence, validates readiness and generation records, and refuses to report a winner when scheduler evidence contradicts it.

🔗 Related Issue

Part of #869 (869#3).

🔀 Merge Order

PR #910 has merged into feat/slurm-execution. This PR is now the direct follow-up slice on that branch and is rebased onto the merged #910 commit (b98c043e). Child PR #915 remains based on this branch and must stay open.

🔄 Changes

• Add a public scheduler-observation collector over the normalized squeue/sacct client contract, including array-task and ordinary collection-job identities.
• Persist canonical per-attempt scheduler snapshots with restrictive permissions and atomic replacement.
• Add fresh-process run, shard, attempt, readiness, generation, and winner status composition through SlurmStateReconciler.
• Keep accounting lag nonterminal with a fixed reconciliation deadline and make reviewed terminal evidence immutable.
• Reject concurrent persisted-state changes and winner/scheduler contradictions instead of guessing status.
• Keep the reconciliation workflow composed across observation, storage, reading, derivation, and orchestration modules; no sibling imports of underscore-prefixed implementation names.
• Add package-install coverage for the new public APIs.

🧪 Testing

• Full Slurm suite after feat: finalize Slurm shard winners #910 merged: 1,265 passed
• make check-slurm
• Focused state, launcher, and package tests: 272 passed
• Isolated wheel build/install/import verification
• Unit tests added/updated
• E2E tests added/updated (not applicable to this state-only slice)

✅ Checklist

• Follows commit message conventions
• Commits are signed off (DCO)
• Architecture docs updated (not applicable; no architecture contract changed)

---

Description updated with AI

[greptile-apps]

Greptile Summary

This PR adds fresh-process reconciliation of persisted Slurm runs using normalized queue and accounting evidence.

• Adds collection and persistence of canonical scheduler observations with bounded accounting-lag handling.
• Adds attempt, shard, generation, and run status composition from persisted state.
• Adds concurrency and winner-consistency checks plus public package exports and installation coverage.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains from the provided follow-up-review context.

No blocking failure remains.

Important Files Changed

Filename	Overview
packages/data-designer-slurm/src/data_designer/slurm/state/observation.py	Adds normalized scheduler collection with accounting precedence, immutable terminal evidence, and bounded accounting-lag transitions.
packages/data-designer-slurm/src/data_designer/slurm/state/observer.py	Adds orchestration for loading persisted state, collecting and persisting observations, validating shard evidence, and composing run status.
packages/data-designer-slurm/src/data_designer/slurm/state/reader.py	Adds optional validated reads for readiness, scheduler observations, and generation-result pairs.
packages/data-designer-slurm/src/data_designer/slurm/state/status.py	Adds validated attempt, shard, generation, and run status models and deterministic aggregation helpers.
packages/data-designer-slurm/src/data_designer/slurm/state/storage.py	Adds restrictive atomic storage operations for per-attempt scheduler observations.
packages/data-designer-slurm/src/data_designer/slurm/launcher/client.py	Adopts the public scheduler-job identity type across queue, accounting, cancellation, and result-correlation APIs.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Load persisted run, plan, shards, and attempts] --> B[Load previous scheduler observations]
    B --> C[Query normalized squeue and sacct evidence]
    C --> D[Resolve current observation per scheduler identity]
    D --> E[Lock and reconcile each shard]
    E --> F[Persist scheduler observations]
    F --> G[Validate readiness, generation results, and winner]
    G --> H[Derive attempt and shard statuses]
    H --> I[Compose complete RunStatus]

Loading

_{Reviews (7): Last reviewed commit: "feat: reconcile persisted Slurm state" | Re-trigger Greptile}