[hz-xiaxz]

Why

Setting up a profile on a large shared Slurm cluster exposed six gaps in /setup-cluster / /using-slurm:

1. Per-user profiles could be committed. .gitignore named one site's profile explicitly, so any newly created <name>.toml (username, host, key path) showed up as an untracked file ready to commit. The skill only "reminded" the student.
2. The probe did not return what the skill promised. harness_slurm.sh probe-partitions printed partition/state/node-count rows only — no cores, memory or GPUs — one row per node state, for every partition on the cluster, including ones the user's account cannot submit to. The richer scripts/cluster_probe.py existed but no skill called it.
3. [limits] were seeded from the wrong source. The skill derived hard caps from partition MaxTime and node totals, but the limits a user actually hits are the per-user QOS caps (MaxTRESPU, MaxJobsPU, MaxWall). QOS was never read.
4. Mixed hardware and whole-node allocation were not representable. Slurm picks the node, but the harness has to reason about it before submitting: a partition mixing 64/96/128-core nodes has no single cores value, so on whole-node partitions the guardrail cannot know what a job will occupy (it counted a 1-task job as 1 cpu while it takes a full node), and there was no record of the node features needed to pin a node type with --constraint for fit or reproducible timing.
5. Memory per cpu was never read. Slurm does not reject --mem above cpus × MaxMemPerCPU; it raises the cpu count to fit and bills every core. Many sites set MaxMemPerCPU only per partition, so scontrol show config (the only place the probe looked) reports nothing. A single-threaded job asking --mem=3G on a 2764 MB/cpu partition ran on one core and was billed for two.
6. Nothing checked what the scheduler actually allocated. The billed doubling above went unnoticed for a whole campaign because no step compared sacct with the request, although that allocation is fully predictable from the request and the partition policy.

What

• .gitignore: ignore every *.toml / *.md in skills/using-slurm/profiles/ and allow-list the public examples (localhost, qdeshell, scnet, *-setup.md). /setup-cluster §4 now verifies with git check-ignore instead of reminding.
• cluster_probe.py
  • reads node features (sinfo %f) and groups each partition into node_types; cores/memory become the largest node type
  • reads scontrol show partition -o and the user's associations; drops partitions the user cannot submit to (account/QOS rules, or a QOS granting 0 cpus)
  • reads per-user QOS caps (sacctmgr show qos) → user_caps, partition QOS overriding the default QOS
  • detects whole-node partitions from OverSubscribe=EXCLUSIVE or a sbatch --test-only dry run of a 1-task job (nothing is submitted)
  • prefers a partition reserved for the user's own account as the default; adds a preemptible class
  • emits a proposed [limits] block (--limits-for <partition> to change the source partition)
  • exits non-zero instead of drafting an empty profile when sinfo times out (the ssh timeout text was previously parsed as partition rows)
• harness_slurm.sh probe-partitions delegates to cluster_probe.py --emit card, so /using-slurm and /cluster-jobs get the same per-user table.
• cluster_guardrail.py: on a whole_node partition or with --exclusive, grade nodes × cores-per-node; --constraint / -C selects the matching node type.
• Memory per cpu: cluster_probe.py records DefMemPerCPU / MaxMemPerCPU per partition (scontrol show partition, falling back to scontrol show config) as def_mem_per_cpu_mb / max_mem_per_cpu_mb in [[partitions]] and the card; cluster_guardrail.py grades a --mem / --mem-per-cpu request at the cpu count Slurm will allocate (soft memory verdict) and warns when the cap is unknown; /using-slurm gains a Sizing memory step (read the cap, size from a measured MaxRSS, check using N processors in --test-only) before the job is written.
• Predicted allocation: cluster_guardrail.py expect predicts what sacct will report per task (partition, task count, cpus incl. memory- or whole-node-raised ones, nodes, memory incl. DefMemPerCPU × cpus, time limit, billing). harness_slurm.sh submit prints it and saves results/.slurm/<jobid>.expect.json; harness_slurm.sh alloc-check <jobid> compares sacct with it (exit 1 on any difference) and reports peak RSS against the request. /using-slurm requires a mismatch to be explained (scontrol show job/partition) and its source fixed before anything else is submitted.
• Docs: schema additions (whole_node, qos, user_caps, [[partitions.node_types]], preemptible) in cluster-profiles.md; /setup-cluster §2–§4 rewritten around the new probe and QOS-seeded limits.

Testing

• make test — 283 passed, 9 skipped. Cluster suites: 21 new tests in the first commits, plus memory-per-cpu tests (partition vs cluster-wide cap, --mem / --mem-per-cpu / --mem=0 / unparseable), on a synthetic cluster fixture: shared / own-group / other-group / preemptible / GPU / request-only partitions).
• Ran the probe against a live ~1400-node Slurm cluster: it hid the 10 partitions the account cannot use, identified the three partitions that allocate whole nodes, and seeded limits from the QOS caps.
• On a second cluster with per-partition MaxMemPerCPU=2764: the probe records it, and the guardrail flags a 1-cpu --mem=3G script as allocating 2 cpus (matching sacct AllocCPUS=2 for jobs that ran with it) while the --mem=2G version is clean. The expect prediction for that job (2 cpus, 2 billing units, 3072M, 1 day) matched sacct on all six array tasks via alloc-check; the 2G prediction against the same job reports the cpu, billing and memory differences.

🤖 Generated with Claude Code