Skip to content

Compile Slurm configuration into deterministic execution plans #875

Description

@nabinchha

Priority Level

High

Task Summary

Resolve strict authored Slurm input and cluster profiles into an immutable, deterministic execution plan before any scheduler submission occurs.

Technical Details & Implementation Plan

  • Load and validate authored run configuration and the selected cluster profile.
  • Resolve Data Designer invocation inputs, deployments, images, resources, mounts, shard ranges, client placement, and output intent into package-owned plan records.
  • Keep authored semantics separate from resolved scheduler/runtime facts.
  • Produce stable plan serialization and digests suitable for pinning in run, shard, and attempt records.
  • Validate cross-field resource, placement, path, and deployment constraints before submission.
  • Provide pure planning entry points that require no Slurm commands or runtime services.

Acceptance criteria

  • The same authored input, profile, and referenced artifacts produce identical plan bytes and digests.
  • Invalid deployment, placement, resource, mount, shard, and destination combinations fail before submission.
  • Plans contain all inputs required by serving, image, launcher/runtime, client-worker, state, and benchmark lanes.
  • Plans contain no credentials, mutable source references, or unresolved environment-dependent defaults.
  • Unit and golden tests cover single-node, multi-node, multi-deployment, sharded, and invalid configurations.

Out of scope

  • Slurm submission or command parsing.
  • Starting model services or the Data Designer client.
  • State persistence, retry, collection, or benchmark execution.

Investigation / Context

This is the configuration-resolution and deterministic-plan lane in #850. #866 and #868 consume planner-supplied placement and resources but do not own authored configuration or plan compilation.

Agent Plan / Findings

Keep planning pure and deterministic so every side effecting lane consumes a reviewed immutable plan rather than re-resolving authored input.

Dependencies

Depends on the public contract in #851, the optional package foundations in #852 and #853, and the shared records in #873.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

taskInternal development task

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions