From 6aaa0629af115efa102f530aeb5da776e085b753 Mon Sep 17 00:00:00 2001 From: lukasWuttke <54042461+LukasWodka@users.noreply.github.com> Date: Wed, 2 Sep 2026 12:04:45 +0200 Subject: [PATCH 1/6] fix(doctor): node-fit checks FREE memory, not allocatable (backend#2870) (#628) * fix(doctor): node-fit checks FREE memory, not allocatable, so an over-committed control plane is caught (backend#2870) `checkNodeFit` fit the training envelope against each node' Allocatable and never against what is already requested on it. So a node whose control plane claims memory passed -- allocatable said "big enough" -- while the pod went Pending / Insufficient memory. This is the layer the ticket says owns nothing: the installer sizes from allocatable, cli doctor checked against allocatable, and the one number that would expose it (free memory) was computed and thrown away. Sum the requests already on each Ready node and fit cpu+memory against FREE (allocatable - requested, floored at 0). A node large enough by allocatable but not beside its own control plane now FAILS with a message that names the over-commit, distinct from "no node is big enough at all". Fail-closed: if the pod list cannot be read, free is UNKNOWN -- say so and fall back to allocatable rather than passing it off as free. Terminal pods hold nothing and are skipped (mirrors requestedMemory). Scope: this is DoD item 3 of backend#2870 (the cli-doctor layer). The installer-side refusal (1), the footprint schedulability test tied to the #2460 constant (2), the jobs-manager permanent-vs-transient message (4) and the CPU envelope audit (5) remain. Test: a 16Gi node whose control plane requests 12Gi (4Gi free) refuses an 8Gi envelope; the same node without the neighbour passes (the subtraction is load-bearing); a terminal neighbour is ignored. Mutation-proven: dropping the memory subtraction flips the over-commit case to a false OK. Co-Authored-By: Claude Opus 4.8 * chore(release): bump VERSION to 0.10.22 (backend#2870) v0.10.21 is already released and this PR changes published files under internal/*; the version-bump-gate requires the develop VERSION to lead the released tag so the next train hop cuts an unreleased version. Co-Authored-By: Claude Opus 4.8 * fix(doctor): exclude running jobs from the free sum, warn on unknown free, name the short dimension (backend#2870) Three findings on the first cut (Saqlain + Bugbot): - HIGH: the free sum counted every non-terminal pod INCLUDING a running training job that already holds the envelope, so on a single-node install doctor returned StatusFail and exited 2 during HEALTHY training -- a false negative on the state it exists to bless. Skip batch-Job pods (the job-name label the batch/v1 controller stamps); the envelope must fit beside the STEADY-STATE control plane, not a transient workload. - HIGH: when the pod list could not be read the OK path returned StatusOK with no caveat, asserting free schedulability it never verified. Now WARN with an allocatable-only caveat. - MEDIUM: the over-commit message always blamed FREE memory; it now names the actual short dimension (cpu / memory / both). Tests: a running job labelled job-name is excluded (healthy training stays ok); an unreadable pod list warns; a cpu over-commit names cpu. All mutation-adjacent and green; golden updated for the new copy. Co-Authored-By: Claude Opus 4.8 * fix(doctor): surface the unknown-free warn in the rollup, and do not blame disk shortfalls on memory (backend#2870) Two follow-on findings from the free-memory fix: - HIGH: the unknown-free StatusWarn detail started with "a Ready node can schedule...", which summarizeDoctor does not recognise as a can't-check, so the rollup fell through to "Ready to run training" at exit 0 -- the over-commit stayed invisible on the default path. Give the warn a distinct "could not read the pod list" prefix and teach the rollup to treat it as a can't-check (Unknown), beside the RESOURCE_REQUESTS / nodes-unlistable ones. - MEDIUM: the over-commit arm fired on !cpuMemFits && allocOnlyFit, which a DISK-only shortfall also satisfies (allocOnlyFit is cpu+mem only), so a disk short was reported as a FREE-memory over-commit. Guard the arm with (overCPU || overMem); a disk short now falls through to the generic fail that names ephemeral-storage. Tests: disk-only shortfall is not blamed on memory; the unknown-free warn rolls up to "couldnt check free compute", not Ready. golden updated. Co-Authored-By: Claude Opus 4.8 * fix(doctor): unknown free compute must outrank the soft GPU warn (backend#2870) Bugbot High. `summarizeDoctor` classifies a Node-capacity Warn as a can't-check by PREFIX, and the `gpuRequested && !fullFits` case sits ABOVE the `!freeKnown` branch with a detail carrying no such prefix. So with a GPU requested and the pod list unreadable, the soft GPU Warn won, the can't-check was never emitted, and the rollup fell through to "Ready to run training" at exit 0 -- a clean bill over a cluster whose free compute doctor had not looked at. Not an exotic path: the chart stamps `nvidia.com/gpu` on CPU-only installs, so `gpuRequested` is commonly true where no node exposes a GPU, which IS this case. The GPU fallback stays soft and stays reported; it just no longer suppresses the stronger statement. Both facts ride in one Warn, can't-check FIRST because the rollup matches on the prefix. AND THE PREFIX IS NOW DEFINED ONCE. It was written out three times -- producer, classifier, and the classifier's test -- which is a check holding its own copy of the rule it verifies. `doctor.CantVerifyFreeCompute` is the single definition; internal/cli already imports internal/doctor, so the classifier reads it rather than retyping it, and the test builds its fixture from it. A producer that rewords the phrase now moves all three together instead of leaving a test green against a string nothing emits. Three guards, two of them new: - unknown free AND gpu requested -> asserts the detail STARTS with the constant (Contains would pass on a detail that merely mentions it somewhere and still greens the run), and that the GPU fact survives. - gpu requested with free KNOWN -> asserts NO can't-check prefix, so the fix cannot be read as "always warn about free". - the classifier's can't-check table gains the COMBINED detail, which is what a GPU-requesting install with an unreadable pod list actually produces. Mutation-proved: deleting the new `!freeKnown` branch reddens the first test with the pre-fix detail verbatim. Verified: `go build ./...`, `go vet ./...` clean, `go test ./...` all packages ok. `zz-all-strings.golden` regenerated for the two new user-facing strings -- additions only, no deletions and no orphaned literal. Co-Authored-By: Claude Opus 4.8 * fix(doctor): the over-commit Fail must not advise sizing runs to the machine (backend#2870) Bugbot Medium, and the advice was not merely vague -- it was the opposite of the fix. `computeRemedy` ends every variant with "size runs to this machine with `resources set max`", and `set max` measures the machine's TOTAL, which is the figure the #2870 Fail just rejected. A user who follows the top line asks for MORE and the training stays stuck; the accurate wording was only behind `--verbose`. The two Node-capacity Fails need opposite advice and were sharing one rollup arm: - "no Ready node can fit" -> the machine is too small; sizing runs to it, or giving it more, is right. - "a Ready node IS large enough, but not beside what is already on it" -> the machine is fine; asking for more is what breaks it. So the over-commit Fail gets its own arm, keyed on a new `doctor.OverCommitted` prefix that the producer now composes its detail FROM -- the same one-definition treatment `CantVerifyFreeCompute` got in the previous commit, and for the same reason: this classification is by prefix, so a reworded producer would silently fall back to the generic arm and restore the bug. Plain terms, no Kubernetes vocabulary, like its two neighbours -- `renderDoctorDetails` is documented as the only place that appears, and the granular Remedy one `--verbose` away already names the knob. Two tests, both directions, because a one-sided fix here is easy to get wrong: - over-commit -> must NOT contain "resources set max", must warn against it. Its fixture detail is BUILT from `doctor.OverCommitted` so arm and producer cannot drift. - generic too-small Fail -> must STILL offer the sizing fix, so this cannot strip correct advice from the case it is correct for. Mutation-proved: deleting the new arm reddens the first test with the generic remedy verbatim ("... or size runs to this machine with `tracebloc resources set max`."). Verified: `go build`, `go vet` clean, `go test ./...` all packages ok. `zz-all-strings.golden` regenerated -- three new strings, one replaced by its constant-composed form. Co-Authored-By: Claude Opus 4.8 * fix(doctor): the resize nudge must not advise past what is FREE (backend#2870) Bugbot Medium, confirmed by @saqlainsyed007 against the code: the fit moved to FREE and the `resources set max` nudge stayed on allocatable-minus-overhead. `bestCPU`/`bestMem` are the largest ALLOCATABLE across Ready nodes; the verdict beside them is computed from per-node free. On a node that is big enough and already partly claimed those disagree, and the nudge won -- so an operator who followed "this machine could give a run up to cpu=...,memory=...Gi" recreated the exact over-commit this check now fails on, and the next `doctor` run told them not to size to their own machine. Advice that contradicts the verdict printed beside it is worse than no advice, so the ceiling is now bounded by free: the largest FREE cpu/memory on any one Ready node is tracked alongside the allocatable-derived best, and the machine handed to `MaxRunCores`/`MaxRunGiB` is the smaller of the two. When free is UNKNOWN the nudge is suppressed entirely rather than falling back to allocatable. There is nothing to bound it with, and that arm is already a Warn saying free could not be verified -- emitting a confident ceiling from the number we just said we could not trust is the same defect one step along. Test: a 32/64Gi node with 24 cores + 40Gi held by a non-job pod. The run (2/8Gi) still fits the free 8/24Gi so the result stays OK, and the detail must not advertise the allocatable ceiling `cpu=31,memory=61Gi`. Mutation-proved: forcing the bound off reddens exactly that test and nothing else; restored, the suite is green. go build, go vet, gofmt: clean. `go test ./...`: all packages ok. Co-Authored-By: Claude Opus 5 * fix(doctor): the nudge must name the command that applies its own numbers (backend#2870) Bugbot Medium on db56239, and it is the second half of my own last fix. Bounding the ceiling by free corrected the FIGURE and left the ATTRIBUTION behind: `tracebloc resources set max` sizes from allocatable via `LargestReadyNode`, so on a claimed node it applies more than the number printed beside it. An operator who read "up to cpu=7,memory=21Gi" and ran the command it named got the allocatable ceiling instead -- reapplying the over-commit this check had just started refusing. Same defect as the finding before it, one step along: the verdict and the advice beside it disagreed. `set max` is now named only when it would genuinely land on the printed numbers -- when nothing meaningful is claimed and the free-bounded ceiling equals the allocatable one. Otherwise the nudge names the explicit form, `tracebloc resources set --cores N --memory NGi`, which applies exactly what it printed. The drift signal survives on a busy node instead of being suppressed, and it stays truthful. The existing test asserted the FIGURE only, which is why a wrong command could sit beside a right number and stay green. It now also asserts that a claimed node does not name `set max` and does name the explicit form. Mutation-proved with a mutation that COMPILES -- my first attempt left `allocM` unused, and a build failure is not a caught mutation, it is an invalid one. Relaxing the equality to `<=` (always true for a free-bounded machine) makes the nudge say `set max` again and reddens exactly that assertion. `internal/cli`'s zz-all-strings golden is regenerated: the two format strings this changes, and nothing else (`git diff --numstat` = 2 added, 1 removed). go build, go vet, gofmt: clean. `go test ./...`: 18 packages ok. Co-Authored-By: Claude Opus 5 * fix(doctor): the over-commit Fail must outrank the stuck-Pending it causes (backend#2870) The previous commit fixed the FIGURE the nudge prints and left the ROLLUP still recommending the thing. summarizeDoctor consulted the Pod-health stuck-Pending arm before the over-commit arm, and that arm ends in computeRemedy -- "size runs to this machine with `resources set max`" -- which sizes from allocatable, the number the over-commit Fail just rejected. Following it raises the ask and leaves the job stuck. The two states CO-OCCUR BY CONSTRUCTION, which is what makes this a correctness bug rather than a preference about ordering: the producer's own Detail ends "so the pod schedules Pending" (internal/doctor/doctor.go:778), so an over-committed node is EXPECTED to also have pods stuck Pending. The over-commit arm was therefore close to unreachable in the field. Moved above the stuck-Pending WARN and no further: a hard Pod-health Fail is a different problem with a different fix (reinstall) and is not caused by this one, so it still wins. Both directions are asserted. The existing over-commit test could not have caught this -- it builds its fixture from allOK, so Pod health is OK and the over-commit arm wins either way. The new test constructs the state where both fire, which is the state the producer says is the normal one. Verified: go test ./... -> 18 packages ok, 0 failed; gofmt clean; go vet clean. Mutation-proved -- putting the stuck-Pending arm back on top reddens the new case with the real `set max` remedy in the message, and reddens ONLY the new case, which is the coverage gap Bugbot found. Co-Authored-By: Claude Opus 5 (1M context) --------- Co-authored-by: Claude Opus 4.8 --- VERSION | 2 +- internal/cli/doctor.go | 35 ++- internal/cli/doctor_test.go | 108 ++++++++ .../cli/testdata/golden/zz-all-strings.golden | 10 +- internal/doctor/doctor.go | 222 ++++++++++++++++- internal/doctor/doctor_test.go | 230 ++++++++++++++++++ 6 files changed, 599 insertions(+), 8 deletions(-) diff --git a/VERSION b/VERSION index e831019..d3dd9cb 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.10.21 +0.10.22 diff --git a/internal/cli/doctor.go b/internal/cli/doctor.go index f152b11..26c9707 100644 --- a/internal/cli/doctor.go +++ b/internal/cli/doctor.go @@ -450,6 +450,38 @@ func summarizeDoctor(results []doctor.Result, tok tokenState) (connected, ready ready = healthLine{doctor.StatusFail, "Not ready — part of your secure environment isn't running.", fmt.Sprintf("Reinstall with `%s`, or email support@tracebloc.io with `%s doctor --diagnose`.", installer.Cmd, launcher())} + case by["Node capacity"].Status == doctor.StatusFail && + strings.HasPrefix(by["Node capacity"].Detail, doctor.OverCommitted): + // THE OPPOSITE REMEDY FROM THE GENERIC CAPACITY ARM BELOW, which is why + // this case exists (Bugbot Medium, #628). `computeRemedy` ends every + // variant with "size runs to this machine with `resources set max`" -- and + // `set max` sizes from the machine's TOTAL, which is the figure this Fail + // just rejected. The machine is big enough; what is missing is room beside + // what is already on it. So the generic advice would raise the ask and + // leave the training stuck, which is worse than no advice: the user + // follows it and the symptom persists. + // + // IT SITS ABOVE THE STUCK-PENDING ARM, and that ordering is the whole + // point rather than a preference (Bugbot Medium, #628 second pass). The + // two states CO-OCCUR BY CONSTRUCTION: the producer's own Detail ends + // "so the pod schedules Pending" (`doctor.go:778`), so an over-committed + // node is *expected* to also have Pod health warning about pods stuck + // Pending. Below that arm this case was therefore almost unreachable in + // the field -- the stuck-Pending arm matched first and printed + // `computeRemedy`, putting `set max` back in front of the operator in the + // exact state this Fail exists to refuse. The first fix corrected the + // figure and left the ROLLUP still recommending the thing. + // + // Only a hard `Pod health` Fail outranks it: pods not running at all is a + // different problem with a different fix (reinstall), and it is not + // caused by this one. + // + // PLAIN TERMS, no Kubernetes vocabulary, like its two neighbours -- + // `renderDoctorDetails` is documented as the only place that appears, and + // the granular Remedy one `--verbose` away already names the knob. + ready = healthLine{doctor.StatusFail, + "Not ready — this machine is big enough, but the platform's own services have already claimed the room.", + fmt.Sprintf("Ask for less per training run, or give the machine more memory/CPU. Do NOT size runs to the machine here — that measures the machine's total, not what is free, so it would ask for MORE and leave the training stuck. `%s doctor --verbose` shows the exact numbers and the knob to turn.", launcher())} case by["Pod health"].Status == doctor.StatusWarn && !strings.HasPrefix(by["Pod health"].Detail, "could not list pods"): // Pods stuck Pending past the grace window (unschedulable / image can't // pull) mean training can't actually schedule — so this is NOT ready, even @@ -512,7 +544,8 @@ func summarizeDoctor(results []doctor.Result, tok tokenState) (connected, ready "Ready to run training — couldn't check your workloads (run with --verbose)", ""} case by["Node capacity"].Status == doctor.StatusWarn && (strings.HasPrefix(by["Node capacity"].Detail, "couldn't read RESOURCE_REQUESTS") || - strings.HasPrefix(by["Node capacity"].Detail, "could not list nodes")): + strings.HasPrefix(by["Node capacity"].Detail, "could not list nodes") || + strings.HasPrefix(by["Node capacity"].Detail, doctor.CantVerifyFreeCompute)): // checkNodeFit's Warn covers two different situations: a can't-check // (RESOURCE_REQUESTS unreadable, nodes unlistable) and the soft GPU // fallback. For a can't-check we simply don't know whether a node can fit diff --git a/internal/cli/doctor_test.go b/internal/cli/doctor_test.go index d51b0c9..e731443 100644 --- a/internal/cli/doctor_test.go +++ b/internal/cli/doctor_test.go @@ -389,6 +389,19 @@ func TestSummarizeDoctor(t *testing.T) { for _, detail := range []string{ "could not list nodes: nodes is forbidden", "couldn't read RESOURCE_REQUESTS from jobs-manager — skipping node-fit", + // backend#2870: unreadable pod list -> free unverifiable -> can't-check, + // not a green pass (Bugbot High: this used to roll up to "Ready"). + // + // BUILT FROM THE PRODUCER'S CONSTANT, not retyped. This string was a + // third copy of the prefix the classifier matches on -- so a producer + // that reworded it would leave this test passing against a phrase + // nothing emits any more. + doctor.CantVerifyFreeCompute + ", so free compute could not be verified — checked against allocatable only; an over-committed control plane would be invisible here", + // Bugbot High on #628: the same can't-check ARRIVING WITH the soft GPU + // warn. The GPU case fires first in checkNodeFit, so this combined + // detail is what a GPU-requesting install with an unreadable pod list + // actually produces -- and it must roll up the same way. + doctor.CantVerifyFreeCompute + ", so free compute could not be verified — checked against allocatable only; an over-committed control plane would be invisible here. Also, no single Ready node satisfies cpu+memory AND nvidia.com/gpu, so GPU jobs would rely on the CPU fallback (needs cpu=2, memory=8Gi)", } { _, r := summarizeDoctor(withDetail(allOK, "Node capacity", doctor.StatusWarn, detail), tokenOK) if r.status != doctor.StatusUnknown { @@ -400,6 +413,101 @@ func TestSummarizeDoctor(t *testing.T) { } }) + t.Run("over-commit Fail must NOT advise sizing runs to the machine", func(t *testing.T) { + // Bugbot Medium on #628. The two Node-capacity Fails need OPPOSITE advice: + // "no node is big enough" is fixed by giving the machine more (or sizing + // runs to it); "big enough, but not beside what is already running" is + // made WORSE by that, because `resources set max` measures the machine's + // total, which is the figure this Fail rejected. A user who follows it asks + // for more and stays stuck. + // + // DETAIL BUILT FROM THE PRODUCER'S CONSTANT so the arm and the producer + // cannot drift apart -- the classification is by prefix, so a reworded + // producer would silently fall through to the generic arm again. + _, r := summarizeDoctor(withDetail(allOK, "Node capacity", doctor.StatusFail, + doctor.OverCommitted+" for a training job (cpu=2, memory=8Gi) but not beside what is already running on it — the envelope over-asks the node's FREE memory, so the pod schedules Pending"), tokenOK) + if r.status != doctor.StatusFail { + t.Fatalf("over-commit is still Not ready, got %v", r.status) + } + if strings.Contains(r.remedy, "resources set max") { + t.Errorf("the top-line remedy tells the user to size runs to the machine, which raises the ask this Fail rejected: %q", r.remedy) + } + if !strings.Contains(r.remedy, "Do NOT") { + t.Errorf("the remedy should warn against sizing to the machine, got %q", r.remedy) + } + }) + + t.Run("over-commit outranks stuck-Pending, which it CAUSES", func(t *testing.T) { + // Bugbot Medium on #628, second pass -- and the case the test above could + // not reach. That one starts from `allOK`, so Pod health is OK and the + // over-commit arm is the first Fail either way. The bug lived in the state + // where BOTH fire. + // + // THEY CO-OCCUR BY CONSTRUCTION, which is what makes this ordering a + // correctness question and not a preference: the producer's Detail ends + // "so the pod schedules Pending" (internal/doctor/doctor.go:778), so an + // over-committed node is EXPECTED to also have pods stuck Pending. With + // the stuck-Pending arm first, the rollup printed `computeRemedy` -- which + // ends in `resources set max` -- in the one state the Node-capacity Fail + // exists to refuse. The figure was fixed on the previous commit and the + // rollup went on recommending the thing. + // + // Both details are built from the producer's own constant/text rather than + // retyped, so a reworded producer reddens this instead of silently falling + // through to the generic arm. + results := withDetail(allOK, "Node capacity", doctor.StatusFail, + doctor.OverCommitted+" for a training job (cpu=2, memory=8Gi) but not beside what is already running on it — the envelope over-asks the node's FREE memory, so the pod schedules Pending") + results = withDetail(results, "Pod health", doctor.StatusWarn, + "1 pod stuck Pending past the grace window") + + _, r := summarizeDoctor(results, tokenOK) + if r.status != doctor.StatusFail { + t.Fatalf("want Fail, got %v", r.status) + } + if strings.Contains(r.remedy, "resources set max") { + t.Errorf("the stuck-Pending arm shadowed the over-commit arm and put `set max` back in front of the operator, in the exact state the Fail refuses: %q", r.remedy) + } + if !strings.Contains(r.remedy, "Do NOT") { + t.Errorf("want the over-commit remedy, got the generic one: %q", r.remedy) + } + if !strings.Contains(r.text, "already claimed the room") { + t.Errorf("want the over-commit top line, got %q", r.text) + } + }) + + t.Run("a hard Pod-health Fail still outranks over-commit", func(t *testing.T) { + // The other side of the reorder: over-commit was moved above the + // stuck-Pending WARN, not above the Pod-health FAIL. Pods not running at + // all is a different problem with a different fix (reinstall), and it is + // not caused by over-commitment -- so it must still win. Without this, + // "move it up" could keep sliding until it shadowed a harder failure. + results := withDetail(allOK, "Node capacity", doctor.StatusFail, + doctor.OverCommitted+" for a training job (cpu=2, memory=8Gi) but not beside what is already running on it") + results = withDetail(results, "Pod health", doctor.StatusFail, + "2 pods CrashLoopBackOff") + + _, r := summarizeDoctor(results, tokenOK) + if r.status != doctor.StatusFail { + t.Fatalf("want Fail, got %v", r.status) + } + if !strings.Contains(r.text, "isn't running") { + t.Errorf("a hard Pod-health Fail must still win the rollup, got %q", r.text) + } + }) + + t.Run("generic capacity Fail still gets the sizing advice", func(t *testing.T) { + // The other side: the fix must not strip the correct advice from the Fail + // it IS correct for -- a machine that is genuinely too small. + _, r := summarizeDoctor(withDetail(allOK, "Node capacity", doctor.StatusFail, + "no Ready node can fit a training job (needs cpu=2, memory=8Gi)"), tokenOK) + if r.status != doctor.StatusFail { + t.Fatalf("want Fail, got %v", r.status) + } + if !strings.Contains(r.remedy, "resources set max") { + t.Errorf("a too-small machine should still be offered the sizing fix: %q", r.remedy) + } + }) + t.Run("node capacity GPU-soft warn → still ready", func(t *testing.T) { _, r := summarizeDoctor(withDetail(allOK, "Node capacity", doctor.StatusWarn, "no single Ready node satisfies cpu+memory AND nvidia.com/gpu — GPU jobs rely on the CPU fallback (needs cpu=2, memory=8Gi)"), tokenOK) diff --git a/internal/cli/testdata/golden/zz-all-strings.golden b/internal/cli/testdata/golden/zz-all-strings.golden index ff24c69..f687ec5 100644 --- a/internal/cli/testdata/golden/zz-all-strings.golden +++ b/internal/cli/testdata/golden/zz-all-strings.golden @@ -57,6 +57,7 @@ screen. %s/%d are runtime placeholders. "%s (unreadable: %v)" "%s Bound, mounted at %s" "%s contains a NUL byte — the file is corrupt or not really a CSV. The cluster rejects it after the upload; re-export the file and re-run." +"%s for a training job (%s) but not beside what is already running on it — the envelope over-asks the node's FREE %s, so the pod schedules Pending" "%s has a header but no data rows (0 ingestable records). Add at least one data row and re-run." "%s has duplicate column name(s): %s. Each column must be unique — the cluster rejects duplicates, and the schema would map onto the wrong column. Rename them and re-run." "%s is empty — add a header and at least one data row, then re-run" @@ -77,6 +78,7 @@ screen. %s/%d are runtime placeholders. "%s ×%d" "%s — %s" "%s — Kubernetes believes %.2f× the memory this machine has, because the k3d node containers are uncapped and each reports the whole VM" +"%s, so free compute could not be verified — checked against allocatable only; an over-committed control plane would be invisible here. Also, no single Ready node satisfies cpu+memory AND %s, so GPU jobs would rely on the CPU fallback (needs %s)" "%s, … and %d more" "%s/%s" "%s: %v" @@ -87,6 +89,7 @@ screen. %s/%d are runtime placeholders. "%w in namespace %q. If your client runs in another namespace, pass --namespace; if this cluster has no tracebloc client yet, run the installer: %s. Diagnose with `tracebloc doctor`." "%w on the cluster your kubeconfig points at — if this machine should have one, run the installer to provision it; otherwise point at the right cluster with --context/--namespace" "%w. Run `tracebloc login` to start a new one" +"'tracebloc resources set --cores %d --memory %dGi'" "(%d CPU · %d GiB" "(+%d more)" "(Pod phase: %s)" @@ -126,6 +129,7 @@ screen. %s/%d are runtime placeholders. "Already signed out." "Applies to your next training run; a run already going keeps its size." "Applying the resource change…" +"Ask for less per training run, or give the machine more memory/CPU. Do NOT size runs to the machine here — that measures the machine's total, not what is free, so it would ask for MORE and leave the training stuck. `%s doctor --verbose` shows the exact numbers and the knob to turn." "Ask one of these admins (or ask them to grant you access)" "Bookkeeping cleanup incomplete — the old table is gone, but its run-journal/salt rows may remain: %s" "Bookkeeping cleanup incomplete — the table is gone, but its run-journal/salt rows may remain: %s" @@ -207,6 +211,8 @@ screen. %s/%d are runtime placeholders. "Email it to support@tracebloc.io." "Email support@tracebloc.io with the output of `%s doctor --diagnose`." "Ensure your kubeconfig user can list nodes." +"Ensure your kubeconfig user can list pods cluster-wide, then re-run doctor to verify free capacity." +"Ensure your kubeconfig user can list pods cluster-wide, then re-run doctor to verify free capacity. If GPU training is expected, also ensure one node has both the compute and the GPU capacity, with its device plugin." "Enter" "Everything looks good — you're ready to run training." "Fix the failing checks above, then re-run `tracebloc client status --seal` to confirm the seal." @@ -252,6 +258,7 @@ screen. %s/%d are runtime placeholders. "Left alone" "Let each training run use up to %s?" "Local dataset" +"Lower RESOURCE_REQUESTS on jobs-manager to leave room for the platform's own pods, or move the control plane / add a node. The installer sizes the envelope from allocatable, not free, so a machine that is 'big enough' can still be over-committed (backend#2870)." "Machine credential — needed by the installer to connect this client" "Memory" "Memory for one run in GiB (2–%d)" @@ -273,6 +280,7 @@ screen. %s/%d are runtime placeholders. "Not ready — part of your secure environment can't start yet." "Not ready — part of your secure environment isn't running." "Not ready — the training images can't be pulled." +"Not ready — this machine is big enough, but the platform's own services have already claimed the room." "Not ready — your active client points at namespace %q, which isn't on this cluster, so data commands will keep failing until you repoint." "Not signed in yet." "Not signed in — run `%s login`." @@ -808,4 +816,4 @@ screen. %s/%d are runtime placeholders. "· %d classes" "— largest node offers ephemeral-storage=%s" "— sign-in codes are valid for %s" -"— this machine could give a run up to cpu=%d,memory=%dGi ('tracebloc resources set max')" +"— this machine could give a run up to cpu=%d,memory=%dGi (%s)" diff --git a/internal/doctor/doctor.go b/internal/doctor/doctor.go index 949fe0f..18b3ffd 100644 --- a/internal/doctor/doctor.go +++ b/internal/doctor/doctor.go @@ -623,7 +623,53 @@ func checkNodeFit(ctx context.Context, cs kubernetes.Interface, env map[string]s // A pod gets ALL its requested resources from ONE node, so evaluate each // node as a whole — never OR cpu/mem and GPU across different nodes, which // would pass even when no single node can run the job (Bugbot on PR #91). - var cpuMemFits, fullFits bool + // FREE, not allocatable (backend#2870). A node whose control plane already + // requests memory has less room than its allocatable advertises, and fitting + // the training envelope against allocatable is the blind spot that let every + // install go Pending unnoticed: the installer writes `allocatable − overhead`, + // and if the real control plane exceeds `overhead` the pod cannot schedule even + // though allocatable "fits". Sum the requests already on each node and evaluate + // the job against what is FREE. If the pod list cannot be read, free is UNKNOWN: + // say so and fall back to allocatable, never silently pass it off as free. + reqCPU := map[string]int64{} // node -> already-requested millicores + reqMem := map[string]int64{} // node -> already-requested bytes + freeKnown := true + if pods, perr := cs.CoreV1().Pods("").List(ctx, metav1.ListOptions{}); perr == nil { + for i := range pods.Items { + p := pods.Items[i] + // A pod with no node holds no node's memory yet; a terminal pod holds + // none at all -- counting either would understate free (mirrors + // requestedMemory's Succeeded/Failed skip). + if p.Spec.NodeName == "" || + p.Status.Phase == corev1.PodSucceeded || p.Status.Phase == corev1.PodFailed { + continue + } + // SKIP batch-Job pods (they carry the `job-name` label the batch/v1 + // controller stamps -- see internal/submit/watch.go). A running + // training or ingestion job holds the envelope itself, so counting it + // would make doctor report "no room for a training job" on the exact + // healthy state it exists to bless -- a false negative that exits 2 + // during training (Bugbot High). The question is whether the envelope + // fits beside the STEADY-STATE control plane (Deployments/DaemonSets), + // not beside a transient workload; those pods have no `job-name`. + if _, isJob := p.Labels["job-name"]; isJob { + continue + } + for j := range p.Spec.Containers { + r := p.Spec.Containers[j].Resources.Requests + if q, ok := r[corev1.ResourceCPU]; ok { + reqCPU[p.Spec.NodeName] += q.MilliValue() + } + if q, ok := r[corev1.ResourceMemory]; ok { + reqMem[p.Spec.NodeName] += q.Value() + } + } + } + } else { + freeKnown = false + } + + var cpuMemFits, fullFits, allocOnlyFit, overCPU, overMem bool // Largest Ready node for the drift nudge — CPU-major with memory as the // tie-break, EXACTLY like resources.nodeLarger, so the advertised ceiling // always matches what `resources set max` will actually apply (Bugbot). @@ -633,6 +679,13 @@ func checkNodeFit(ctx context.Context, cs kubernetes.Interface, env map[string]s // to pick a node, so it must not perturb the drift nudge's tie-break. var bestDisk resource.Quantity var sawDisk bool + // The most cpu/memory actually FREE on any one Ready node. Tracked next to + // the allocatable-derived best above because the drift nudge must not + // advertise a ceiling the fit would refuse: the fit moved to free, the nudge + // did not, so on a large-but-claimed node it advised sizing UP to a figure + // that recreates the over-commit this very check now fails (Bugbot Medium, + // confirmed by @saqlainsyed007). + var bestFreeCPUm, bestFreeMemB int64 for i := range nodes.Items { n := nodes.Items[i] if !nodeReady(n) { @@ -650,7 +703,37 @@ func checkNodeFit(ctx context.Context, cs kubernetes.Interface, env map[string]s bestDisk = d } } - nodeCPUMem := alloc.Cpu().Cmp(cpuReq) >= 0 && alloc.Memory().Cmp(memReq) >= 0 + // Allocatable minus what is already requested on THIS node (never below 0). + // When free is unknown, this is allocatable and the caveat is reported below. + freeCPUm := alloc.Cpu().MilliValue() + freeMemB := alloc.Memory().Value() + if freeKnown { + if freeCPUm -= reqCPU[n.Name]; freeCPUm < 0 { + freeCPUm = 0 + } + if freeMemB -= reqMem[n.Name]; freeMemB < 0 { + freeMemB = 0 + } + } + if freeCPUm > bestFreeCPUm || + (freeCPUm == bestFreeCPUm && freeMemB > bestFreeMemB) { + bestFreeCPUm, bestFreeMemB = freeCPUm, freeMemB + } + nodeCPUMem := freeCPUm >= cpuReq.MilliValue() && freeMemB >= memReq.Value() + // Whether the node is big enough IGNORING neighbours -- the old, blind + // verdict. Kept only to tell "no node is big enough at all" apart from + // "a node is big enough but not beside its control plane" in the message. + if alloc.Cpu().Cmp(cpuReq) >= 0 && alloc.Memory().Cmp(memReq) >= 0 { + allocOnlyFit = true + // Which dimension the FREE fit falls short on, so the over-commit + // message names cpu vs memory instead of always blaming memory (Bugbot). + if freeCPUm < cpuReq.MilliValue() { + overCPU = true + } + if freeMemB < memReq.Value() { + overMem = true + } + } // Disk joins cpu+memory as a WHOLE-NODE condition. A pod gets every // resource it requests from ONE node, so this must be AND-ed into the // same per-node verdict and never OR-ed across nodes (Bugbot on PR #91 @@ -678,14 +761,59 @@ func checkNodeFit(ctx context.Context, cs kubernetes.Interface, env map[string]s } switch { + case !cpuMemFits && allocOnlyFit && freeKnown && (overCPU || overMem): + // The #2870 case: a node IS big enough, but not beside the requests its own + // control plane already holds. Allocatable said yes; free says no. This is + // the shape that goes Pending/Insufficient memory after a clean install. + short := "memory" + switch { + case overCPU && overMem: + short = "cpu and memory" + case overCPU: + short = "cpu" + } + return Result{ + Name: name, + Status: StatusFail, + Detail: fmt.Sprintf("%s for a training job (%s) but not beside what is already running on it — the envelope over-asks the node's FREE %s, so the pod schedules Pending", OverCommitted, req, short), + Remedy: "Lower RESOURCE_REQUESTS on jobs-manager to leave room for the platform's own pods, or move the control plane / add a node. The installer sizes the envelope from allocatable, not free, so a machine that is 'big enough' can still be over-committed (backend#2870).", + } case !cpuMemFits: + detail := fmt.Sprintf("no Ready node can fit a training job (needs %s)", req) + if !freeKnown { + detail += " — checked against allocatable only; the pod list could not be read, so an over-committed control plane would be invisible here" + } return Result{ Name: name, Status: StatusFail, - Detail: fmt.Sprintf("no Ready node can fit a training job (needs %s)", req), + Detail: detail, Remedy: "Add/resize a node to meet the job's requests, or lower RESOURCE_REQUESTS on jobs-manager.", } case gpuRequested && !fullFits: + // UNKNOWN FREE OUTRANKS THE SOFT GPU WARN (Bugbot High, #628). + // + // This case sits ABOVE the `!freeKnown` branch in the default arm, and its + // detail carries no `CantVerifyFreeCompute` prefix -- so when the pod list + // could not be read AND a GPU is requested, this Warn won, the can't-check + // was never emitted, and `summarizeDoctor` fell through to "Ready to run + // training" at exit 0. Doctor printed a clean bill over a cluster whose + // free compute it had not looked at. + // + // It is not an exotic path: the chart stamps `nvidia.com/gpu` on CPU-only + // installs too, so `gpuRequested` is commonly true where no node exposes a + // GPU -- which is exactly this case. + // + // The GPU fallback stays SOFT and stays reported; it just no longer + // suppresses the stronger statement. Both facts go in one Warn, with the + // can't-check FIRST because the rollup matches on the prefix. + if !freeKnown { + return Result{ + Name: name, + Status: StatusWarn, + Detail: fmt.Sprintf("%s, so free compute could not be verified — checked against allocatable only; an over-committed control plane would be invisible here. Also, no single Ready node satisfies cpu+memory AND %s, so GPU jobs would rely on the CPU fallback (needs %s)", CantVerifyFreeCompute, gpuName, req), + Remedy: "Ensure your kubeconfig user can list pods cluster-wide, then re-run doctor to verify free capacity. If GPU training is expected, also ensure one node has both the compute and the GPU capacity, with its device plugin.", + } + } return Result{ Name: name, Status: StatusWarn, @@ -705,17 +833,101 @@ func checkNodeFit(ctx context.Context, cs kubernetes.Interface, env map[string]s // stale when a machine GROWS. When the configured budget uses no more // than half of what this machine could give one run (largest node − // platform overhead), say so. + // + // BOUNDED BY FREE, NOT ALLOCATABLE. The ceiling is derived from the + // largest node, but the verdict above is derived from what is FREE on + // it. On a node that is big enough and already partly claimed those two + // disagree, and the nudge won: an operator who ran `resources set max` + // recreated the exact over-commit this check fails on, and the next + // `doctor` told them not to size to their own machine. Advice that + // contradicts the verdict printed beside it is worse than no advice. + // + // So the machine handed to `MaxRunCores`/`MaxRunGiB` is the smaller of + // allocatable and free, and when free is UNKNOWN the nudge is suppressed + // entirely -- there is nothing to bound it with, and this arm is already + // a Warn that says free could not be verified. m := resources.Machine{CPU: bestCPU, Mem: bestMem} + if freeKnown { + freeCPU := *resource.NewMilliQuantity(bestFreeCPUm, resource.DecimalSI) + freeMem := *resource.NewQuantity(bestFreeMemB, resource.BinarySI) + if freeCPU.Cmp(m.CPU) < 0 { + m.CPU = freeCPU + } + if freeMem.Cmp(m.Mem) < 0 { + m.Mem = freeMem + } + } maxCores, maxGiB := resources.MaxRunCores(m), resources.MaxRunGiB(m) - if maxCores >= 1 && maxGiB >= 2 && + if freeKnown && maxCores >= 1 && maxGiB >= 2 && cpuReq.MilliValue()*2 <= int64(maxCores)*1000 && memReq.Value()*2 <= int64(maxGiB)<<30 { - detail += fmt.Sprintf(" — this machine could give a run up to cpu=%d,memory=%dGi ('tracebloc resources set max')", maxCores, maxGiB) + // NAME THE COMMAND THAT APPLIES THESE NUMBERS (Bugbot Medium). + // Bounding the ceiling by free fixed the figure and left the + // attribution behind: `resources set max` sizes from ALLOCATABLE via + // `LargestReadyNode`, so on a claimed node it applies more than the + // figure printed beside it -- and reapplies the over-commit this + // check just started refusing. The printed numbers and the suggested + // command have to be the same thing. + // + // So `set max` is named only when it would genuinely land on these + // numbers -- i.e. nothing meaningful is claimed and the free-bounded + // ceiling equals the allocatable one. Otherwise the explicit form is + // named, which applies exactly what is printed. + allocM := resources.Machine{CPU: bestCPU, Mem: bestMem} + how := fmt.Sprintf("'tracebloc resources set --cores %d --memory %dGi'", maxCores, maxGiB) + if maxCores == resources.MaxRunCores(allocM) && maxGiB == resources.MaxRunGiB(allocM) { + how = "'tracebloc resources set max'" + } + detail += fmt.Sprintf(" — this machine could give a run up to cpu=%d,memory=%dGi (%s)", maxCores, maxGiB, how) + } + // UNKNOWN free is not a clean pass (Bugbot High). When the pod list could + // not be read, this fit was against allocatable, not free -- so it cannot + // assert schedulability, and an over-committed control plane would be + // invisible. Warn and say so rather than greening node capacity. + if !freeKnown { + return Result{ + Name: name, + // DISTINCT can't-check PREFIX so the rollup (summarizeDoctor in + // cli/doctor.go) classifies this as "couldn't check free compute" + // rather than greening it: it matches Node-capacity can't-checks by + // prefix, and "a Ready node can schedule..." would fall through to + // "Ready to run training" at exit 0 (Bugbot High). Keep the + // "allocatable only" phrase the caveat and its test rely on. + Status: StatusWarn, + Detail: CantVerifyFreeCompute + ", so free compute could not be verified — checked against allocatable only; an over-committed control plane would be invisible here (the node fits the envelope on allocatable: " + req + ")", + Remedy: "Ensure your kubeconfig user can list pods cluster-wide, then re-run doctor to verify free capacity.", + } } return Result{Name: name, Status: StatusOK, Detail: detail} } } +// OverCommitted is the prefix of the #2870 Fail: a node IS big enough, but not +// beside what its own control plane already holds. ONE definition, same reason +// as CantVerifyFreeCompute below. +// +// The rollup needs to tell this Fail apart from the generic "no node is big +// enough" one because the REMEDIES ARE OPPOSITE. `computeRemedy` ends every +// variant with "size runs to this machine with `tracebloc resources set max`", +// and `set max` sizes from ALLOCATABLE -- the exact figure this Fail rejected. +// Following it raises the envelope and the pod stays Pending (Bugbot Medium, +// #628). The generic arm is right for a too-small machine and wrong here. +const OverCommitted = "a Ready node is large enough" + +// CantVerifyFreeCompute is the prefix every "we could not check free compute" +// Node-capacity Warn must start with, and the ONE definition of it. +// +// `summarizeDoctor` in internal/cli/doctor.go classifies a Node-capacity Warn as +// a can't-check by PREFIX, so the producer's wording is load-bearing: a Warn that +// does not start with this string falls through to "Ready to run training" at +// exit 0. That string was written out three times -- here, in the classifier, and +// in the classifier's test -- which is a rule the checks held their own copy of. +// +// It is exported rather than duplicated because the two live in different +// packages and internal/cli already imports this one. A third caller that needs +// the same classification must reference this, not retype it. +const CantVerifyFreeCompute = "could not read the pod list" + // checkImagePull verifies that any registry pull secret the jobs-manager // references exists and is a well-formed dockerconfigjson — so private-image // pulls don't ImagePullBackOff. (Bad-but-well-formed credentials can't be diff --git a/internal/doctor/doctor_test.go b/internal/doctor/doctor_test.go index 4ae6205..3248e63 100644 --- a/internal/doctor/doctor_test.go +++ b/internal/doctor/doctor_test.go @@ -678,6 +678,57 @@ func TestCheckNodeFit(t *testing.T) { t.Fatalf("=> %v (%q), want ok without nudge", r.Status, r.Detail) } }) + t.Run("large but CLAIMED node -> nudge must not advise past free", func(t *testing.T) { + // Bugbot Medium, confirmed by @saqlainsyed007. The fit moved to FREE; + // the nudge stayed on allocatable. On a node that is big enough and + // already partly claimed the two disagree, and the nudge won -- so an + // operator who ran `resources set max` recreated the exact over-commit + // this check now fails on, and the next `doctor` told them not to size + // to their own machine. + // + // 32/64Gi node, 24 cores + 40Gi already held by a non-job pod: the run + // (2/8Gi) still fits in the 8/24Gi that is free, so this stays OK -- but + // the advertised ceiling may not be the allocatable-derived 31/61Gi. + claim := &corev1.Pod{ + ObjectMeta: metav1.ObjectMeta{Name: "control-plane", Namespace: ns}, + Spec: corev1.PodSpec{ + NodeName: "n1", + Containers: []corev1.Container{{ + Name: "c", + Resources: corev1.ResourceRequirements{ + Requests: corev1.ResourceList{ + corev1.ResourceCPU: resource.MustParse("24"), + corev1.ResourceMemory: resource.MustParse("40Gi"), + }, + }, + }}, + }, + Status: corev1.PodStatus{Phase: corev1.PodRunning}, + } + cs := fake.NewClientset(node("n1", "32", "64Gi"), claim) + r := checkNodeFit(bg(), cs, cpuOnly) + if r.Status != StatusOK { + t.Fatalf("=> %v (%q), want ok (2/8Gi fits in the free 8/24Gi)", r.Status, r.Detail) + } + if strings.Contains(r.Detail, "cpu=31,memory=61Gi") { + t.Fatalf("nudge advertises the ALLOCATABLE ceiling 31/61Gi while only "+ + "8 cores / 24Gi are free -- following it recreates the over-commit "+ + "this check fails on: %q", r.Detail) + } + // AND THE COMMAND MUST APPLY THE PRINTED NUMBERS (Bugbot Medium). + // Bounding the figure by free is only half the fix: `set max` sizes from + // allocatable via LargestReadyNode, so attributing a free-bounded figure + // to it still sends the operator to a command that over-commits. On a + // claimed node the explicit form is the honest one. + if strings.Contains(r.Detail, "resources set max") { + t.Fatalf("nudge names 'set max', which sizes from ALLOCATABLE and so "+ + "applies more than the figure printed beside it: %q", r.Detail) + } + if r.Detail != "" && !strings.Contains(r.Detail, "resources set --cores") { + t.Fatalf("nudge does not name the explicit command that applies exactly "+ + "what it printed: %q", r.Detail) + } + }) t.Run("heterogeneous nodes: nudge quotes the CPU-major node, matching set max (Bugbot)", func(t *testing.T) { // resources.LargestReadyNode (what `set max` applies) is CPU-major: // it picks cpuBig (32/64Gi -> max 31/61Gi), not memBig (8/128Gi -> @@ -693,6 +744,185 @@ func TestCheckNodeFit(t *testing.T) { }) } +// cpPod is a control-plane pod scheduled on a node, requesting memory — the +// neighbour whose requests the fit must subtract (backend#2870). +func cpPod(name, nodeName, mem string) *corev1.Pod { + return podOn(name, nodeName, "", mem, nil) +} + +// podOn is a running pod on a node requesting cpu/mem (empty = unset), with +// optional labels (e.g. the batch/v1 job-name label that marks a training pod). +func podOn(name, nodeName, cpu, mem string, labels map[string]string) *corev1.Pod { + reqs := corev1.ResourceList{} + if cpu != "" { + reqs[corev1.ResourceCPU] = resource.MustParse(cpu) + } + if mem != "" { + reqs[corev1.ResourceMemory] = resource.MustParse(mem) + } + return &corev1.Pod{ + ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: "kube-system", Labels: labels}, + Spec: corev1.PodSpec{ + NodeName: nodeName, + Containers: []corev1.Container{{Name: "c", Resources: corev1.ResourceRequirements{Requests: reqs}}}, + }, + Status: corev1.PodStatus{Phase: corev1.PodRunning}, + } +} + +// backend#2870: node-fit must be against FREE memory (allocatable − what is +// already requested on the node), not allocatable. A node big enough by +// allocatable but over-committed by the control plane it hosts must FAIL. +func TestCheckNodeFitFreeMemory(t *testing.T) { + req := map[string]string{"RESOURCE_REQUESTS": "cpu=2,memory=8Gi"} + + t.Run("allocatable fits but FREE does not -> fail", func(t *testing.T) { + // 16Gi node, control plane already claims 12Gi -> 4Gi free < 8Gi envelope. + // Allocatable (16Gi) "fits"; free (4Gi) does not. + cs := fake.NewClientset(node("n1", "4", "16Gi"), cpPod("cp", "n1", "12Gi")) + r := checkNodeFit(bg(), cs, req) + if r.Status != StatusFail { + t.Fatalf("=> %v (%q), want fail (over-committed)", r.Status, r.Detail) + } + if !strings.Contains(r.Detail, "FREE") || !strings.Contains(r.Detail, "over-asks") { + t.Fatalf("detail should name the free-memory over-commit: %q", r.Detail) + } + }) + + // The subtraction is load-bearing: the SAME node without the neighbour passes, + // so it is the pod request — not the node size — that flips the verdict. + t.Run("same node without the neighbour -> ok", func(t *testing.T) { + cs := fake.NewClientset(node("n1", "4", "16Gi")) + if r := checkNodeFit(bg(), cs, req); r.Status != StatusOK { + t.Fatalf("=> %v (%q), want ok", r.Status, r.Detail) + } + }) + + // A terminal pod holds no memory; free stays 16Gi and the job fits. + t.Run("terminal neighbour does not consume free", func(t *testing.T) { + done := cpPod("old", "n1", "12Gi") + done.Status.Phase = corev1.PodSucceeded + cs := fake.NewClientset(node("n1", "4", "16Gi"), done) + if r := checkNodeFit(bg(), cs, req); r.Status != StatusOK { + t.Fatalf("=> %v (%q), want ok (terminal pod ignored)", r.Status, r.Detail) + } + }) + + // A RUNNING training job (batch/v1 job-name label) holds the envelope itself. + // It must NOT count against free, or doctor fails on the exact healthy state it + // blesses (Bugbot High). Same 12Gi neighbour, but labelled a Job -> still ok. + t.Run("a running training job is excluded, not counted -> ok", func(t *testing.T) { + job := podOn("train-sim", "n1", "", "12Gi", map[string]string{"job-name": "exp-42"}) + cs := fake.NewClientset(node("n1", "4", "16Gi"), job) + if r := checkNodeFit(bg(), cs, req); r.Status != StatusOK { + t.Fatalf("=> %v (%q), want ok (training job excluded)", r.Status, r.Detail) + } + }) + + // The over-commit message names the SHORT dimension, not always memory (Bugbot). + // Control plane claims cpu (3 of 4), leaving 1 free < the 2-cpu envelope; memory + // is fine. The message must say cpu, not memory. + t.Run("over-commit on cpu names cpu, not memory", func(t *testing.T) { + cp := podOn("cp", "n1", "3", "", nil) // 3 cpu, no memory + cs := fake.NewClientset(node("n1", "4", "16Gi"), cp) + r := checkNodeFit(bg(), cs, req) // needs cpu=2, memory=8Gi + if r.Status != StatusFail { + t.Fatalf("=> %v (%q), want fail (cpu over-commit)", r.Status, r.Detail) + } + if !strings.Contains(r.Detail, "FREE cpu") { + t.Fatalf("detail should name FREE cpu, not memory: %q", r.Detail) + } + }) + + // A DISK-only shortfall must not take the over-commit arm (Bugbot Medium): + // free cpu+memory are fine, only ephemeral-storage is short, so the message + // must name disk via the generic fail, not blame FREE memory. + t.Run("disk-only shortfall is not blamed on memory", func(t *testing.T) { + diskReq := map[string]string{"RESOURCE_REQUESTS": "cpu=2,memory=8Gi,ephemeral-storage=50Gi"} + n := nodeWithDisk("n1", "4", "16Gi", "20Gi") // cpu+mem fit free; disk 20Gi < 50Gi + cs := fake.NewClientset(n) + r := checkNodeFit(bg(), cs, diskReq) + if r.Status != StatusFail { + t.Fatalf("=> %v (%q), want fail (disk)", r.Status, r.Detail) + } + if strings.Contains(r.Detail, "over-asks") || strings.Contains(r.Detail, "FREE memory") { + t.Fatalf("disk shortfall must not be reported as a memory over-commit: %q", r.Detail) + } + if !strings.Contains(r.Detail, "ephemeral-storage") { + t.Fatalf("detail should name ephemeral-storage: %q", r.Detail) + } + }) + + // UNKNOWN free is not a clean pass (Bugbot High): when the pod list can't be + // read, the fit falls back to allocatable and must WARN with a caveat, never + // green node capacity as if free were verified. + t.Run("unknown free (pod list fails) -> warn with caveat", func(t *testing.T) { + cs := fake.NewClientset(node("n1", "4", "16Gi")) + cs.PrependReactor("list", "pods", func(k8stesting.Action) (bool, runtime.Object, error) { + return true, nil, errors.New("pods is forbidden") + }) + r := checkNodeFit(bg(), cs, req) + if r.Status != StatusWarn { + t.Fatalf("=> %v (%q), want warn (free unknown)", r.Status, r.Detail) + } + if !strings.Contains(r.Detail, "allocatable only") { + t.Fatalf("detail should caveat allocatable-only: %q", r.Detail) + } + }) + + // Bugbot High on #628: the SOFT GPU warn used to outrank this one. + // + // `gpuRequested && !fullFits` sits above the `!freeKnown` branch and its + // detail carried no can't-check prefix, so with a GPU requested and the pod + // list unreadable the GPU Warn won, the caveat was never emitted, and + // `summarizeDoctor` -- which classifies by PREFIX -- fell through to "Ready + // to run training" at exit 0 over a cluster whose free compute doctor had + // not looked at. The chart stamps `nvidia.com/gpu` on CPU-only installs, so + // this is the common shape rather than an exotic one. + t.Run("unknown free AND gpu requested -> can't-check wins, GPU still reported", func(t *testing.T) { + gpu := map[string]string{ + "RESOURCE_REQUESTS": "cpu=2,memory=8Gi", + "GPU_REQUESTS": "nvidia.com/gpu=1", + } + cs := fake.NewClientset(node("n1", "4", "16Gi")) // cpu/mem fit, NO gpu + cs.PrependReactor("list", "pods", func(k8stesting.Action) (bool, runtime.Object, error) { + return true, nil, errors.New("pods is forbidden") + }) + r := checkNodeFit(bg(), cs, gpu) + if r.Status != StatusWarn { + t.Fatalf("=> %v (%q), want warn", r.Status, r.Detail) + } + // PREFIX, not Contains: that is what the rollup matches on, so a detail + // merely mentioning the phrase somewhere would still green the run. + if !strings.HasPrefix(r.Detail, CantVerifyFreeCompute) { + t.Fatalf("detail must START with %q so summarizeDoctor classifies it as a can't-check, got %q", + CantVerifyFreeCompute, r.Detail) + } + // The soft GPU fact is not lost, it is just no longer the whole story. + if !strings.Contains(r.Detail, "nvidia.com/gpu") { + t.Fatalf("the GPU fallback should still be reported: %q", r.Detail) + } + }) + + // The other side, so the fix above cannot be read as "always warn about free": + // with the pod list READABLE, the GPU warn keeps its own wording and must not + // claim a can't-check. + t.Run("gpu requested, free KNOWN -> plain GPU warn, no can't-check prefix", func(t *testing.T) { + gpu := map[string]string{ + "RESOURCE_REQUESTS": "cpu=2,memory=8Gi", + "GPU_REQUESTS": "nvidia.com/gpu=1", + } + cs := fake.NewClientset(node("n1", "4", "16Gi")) + r := checkNodeFit(bg(), cs, gpu) + if r.Status != StatusWarn { + t.Fatalf("=> %v (%q), want warn", r.Status, r.Detail) + } + if strings.HasPrefix(r.Detail, CantVerifyFreeCompute) { + t.Fatalf("free WAS readable; this must not report a can't-check: %q", r.Detail) + } + }) +} + func dockerSecret(name string, data []byte) *corev1.Secret { return &corev1.Secret{ ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: ns}, From 8c79ddaccc2a2e70d3302e2a2c2c9d7ef2cc63dc Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 3 Sep 2026 06:16:12 +0200 Subject: [PATCH 2/6] chore(deps): bump the k8s group with 2 updates (#623) Bumps the k8s group with 2 updates: [k8s.io/apimachinery](https://github.com/kubernetes/apimachinery) and [k8s.io/client-go](https://github.com/kubernetes/client-go). Updates `k8s.io/apimachinery` from 0.36.4 to 0.37.0 - [Commits](https://github.com/kubernetes/apimachinery/compare/v0.36.4...v0.37.0) Updates `k8s.io/client-go` from 0.36.4 to 0.37.0 - [Changelog](https://github.com/kubernetes/client-go/blob/master/CHANGELOG.md) - [Commits](https://github.com/kubernetes/client-go/compare/v0.36.4...v0.37.0) --- updated-dependencies: - dependency-name: k8s.io/apimachinery dependency-version: 0.37.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: k8s - dependency-name: k8s.io/client-go dependency-version: 0.37.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: k8s ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- go.mod | 41 ++++++++++++++--------- go.sum | 103 ++++++++++++++++++++++++++++++++------------------------- 2 files changed, 83 insertions(+), 61 deletions(-) diff --git a/go.mod b/go.mod index 5698508..45652b1 100644 --- a/go.mod +++ b/go.mod @@ -47,27 +47,36 @@ require ( golang.org/x/term v0.45.0 golang.org/x/text v0.41.0 gopkg.in/yaml.v3 v3.0.1 - k8s.io/api v0.36.4 - k8s.io/apimachinery v0.36.4 - k8s.io/client-go v0.36.4 + k8s.io/api v0.37.0 + k8s.io/apimachinery v0.37.0 + k8s.io/client-go v0.37.0 ) require ( github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect github.com/emicklei/go-restful/v3 v3.13.0 // indirect - github.com/fxamacker/cbor/v2 v2.9.0 // indirect + github.com/fxamacker/cbor/v2 v2.9.1 // indirect github.com/go-logr/logr v1.4.3 // indirect - github.com/go-openapi/jsonpointer v0.21.0 // indirect - github.com/go-openapi/jsonreference v0.20.2 // indirect - github.com/go-openapi/swag v0.23.0 // indirect + github.com/go-openapi/jsonpointer v1.0.0 // indirect + github.com/go-openapi/jsonreference v1.0.0 // indirect + github.com/go-openapi/swag v0.27.1 // indirect + github.com/go-openapi/swag/cmdutils v0.27.1 // indirect + github.com/go-openapi/swag/conv v0.27.1 // indirect + github.com/go-openapi/swag/fileutils v0.27.1 // indirect + github.com/go-openapi/swag/jsonutils v0.27.1 // indirect + github.com/go-openapi/swag/loading v0.27.1 // indirect + github.com/go-openapi/swag/mangling v0.27.1 // indirect + github.com/go-openapi/swag/netutils v0.27.1 // indirect + github.com/go-openapi/swag/pools v0.27.1 // indirect + github.com/go-openapi/swag/stringutils v0.27.1 // indirect + github.com/go-openapi/swag/typeutils v0.27.1 // indirect + github.com/go-openapi/swag/yamlutils v0.27.1 // indirect github.com/google/gnostic-models v0.7.0 // indirect github.com/google/uuid v1.6.0 // indirect github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674 // indirect github.com/inconshreveable/mousetrap v1.1.0 // indirect - github.com/josharian/intern v1.0.0 // indirect github.com/json-iterator/go v1.1.12 // indirect github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect - github.com/mailru/easyjson v0.7.7 // indirect github.com/mattn/go-colorable v0.1.14 // indirect github.com/mattn/go-isatty v0.0.22 // indirect github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b // indirect @@ -78,20 +87,20 @@ require ( github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect github.com/rivo/uniseg v0.4.7 // indirect github.com/x448/float16 v0.8.4 // indirect - go.yaml.in/yaml/v2 v2.4.3 // indirect + go.yaml.in/yaml/v2 v2.4.4 // indirect go.yaml.in/yaml/v3 v3.0.4 // indirect golang.org/x/net v0.57.0 // indirect - golang.org/x/oauth2 v0.34.0 // indirect - golang.org/x/time v0.14.0 // indirect + golang.org/x/oauth2 v0.36.0 // indirect + golang.org/x/time v0.15.0 // indirect google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af // indirect gopkg.in/evanphx/json-patch.v4 v4.13.0 // indirect gopkg.in/inf.v0 v0.9.1 // indirect k8s.io/klog/v2 v2.140.0 // indirect - k8s.io/kube-openapi v0.0.0-20260317180543-43fb72c5454a // indirect - k8s.io/streaming v0.36.4 // indirect - k8s.io/utils v0.0.0-20260210185600-b8788abfbbc2 // indirect + k8s.io/kube-openapi v0.0.0-20260721132016-d427ff9ee9ad // indirect + k8s.io/streaming v0.37.0 // indirect + k8s.io/utils v0.0.0-20260626114624-be93311217bd // indirect sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 // indirect sigs.k8s.io/randfill v1.0.0 // indirect - sigs.k8s.io/structured-merge-diff/v6 v6.3.3 // indirect + sigs.k8s.io/structured-merge-diff/v6 v6.4.2 // indirect sigs.k8s.io/yaml v1.6.0 // indirect ) diff --git a/go.sum b/go.sum index 337cfaa..5c01e85 100644 --- a/go.sum +++ b/go.sum @@ -7,7 +7,6 @@ github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkY github.com/chengxilo/virtualterm v1.0.4 h1:Z6IpERbRVlfB8WkOmtbHiDbBANU7cimRIof7mk9/PwM= github.com/chengxilo/virtualterm v1.0.4/go.mod h1:DyxxBZz/x1iqJjFxTFcr6/x+jSpqN0iwWCOK1q10rlY= github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g= -github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= github.com/creack/pty v1.1.17 h1:QeVUsEDNrLBW4tMgZHvxy18sKtr6VI492kBhUfhDJNI= github.com/creack/pty v1.1.17/go.mod h1:MOBLtS5ELjhRRrroQr9kyvTxUAFNvYEK993ew/Vr4O4= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= @@ -20,18 +19,44 @@ github.com/emicklei/go-restful/v3 v3.13.0 h1:C4Bl2xDndpU6nJ4bc1jXd+uTmYPVUwkD6bF github.com/emicklei/go-restful/v3 v3.13.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc= github.com/fatih/color v1.19.0 h1:Zp3PiM21/9Ld6FzSKyL5c/BULoe/ONr9KlbYVOfG8+w= github.com/fatih/color v1.19.0/go.mod h1:zNk67I0ZUT1bEGsSGyCZYZNrHuTkJJB+r6Q9VuMi0LE= -github.com/fxamacker/cbor/v2 v2.9.0 h1:NpKPmjDBgUfBms6tr6JZkTHtfFGcMKsw3eGcmD/sapM= -github.com/fxamacker/cbor/v2 v2.9.0/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ= +github.com/fxamacker/cbor/v2 v2.9.1 h1:2rWm8B193Ll4VdjsJY28jxs70IdDsHRWgQYAI80+rMQ= +github.com/fxamacker/cbor/v2 v2.9.1/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ= github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= -github.com/go-openapi/jsonpointer v0.19.6/go.mod h1:osyAmYz/mB/C3I+WsTTSgw1ONzaLJoLCyoi6/zppojs= -github.com/go-openapi/jsonpointer v0.21.0 h1:YgdVicSA9vH5RiHs9TZW5oyafXZFc6+2Vc1rr/O9oNQ= -github.com/go-openapi/jsonpointer v0.21.0/go.mod h1:IUyH9l/+uyhIYQ/PXVA41Rexl+kOkAPDdXEYns6fzUY= -github.com/go-openapi/jsonreference v0.20.2 h1:3sVjiK66+uXK/6oQ8xgcRKcFgQ5KXa2KvnJRumpMGbE= -github.com/go-openapi/jsonreference v0.20.2/go.mod h1:Bl1zwGIM8/wsvqjsOQLJ/SH+En5Ap4rVB5KVcIDZG2k= -github.com/go-openapi/swag v0.22.3/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14= -github.com/go-openapi/swag v0.23.0 h1:vsEVJDUo2hPJ2tu0/Xc+4noaxyEffXNIs3cOULZ+GrE= -github.com/go-openapi/swag v0.23.0/go.mod h1:esZ8ITTYEsH1V2trKHjAN8Ai7xHb8RV+YSZ577vPjgQ= +github.com/go-openapi/jsonpointer v1.0.0 h1:kR9tHqY0CtZaOPVFm622dPVNhrvYpwr4uCxgL3h1H8s= +github.com/go-openapi/jsonpointer v1.0.0/go.mod h1:Z3rw7dWu1p9IgitXCFamSlA5lmDiklEB6vkaxcNZW5Y= +github.com/go-openapi/jsonreference v1.0.0 h1:jlmTr6torcd1YgDQvSfNmRtKzYDO4FGBkrAdlAVWnpY= +github.com/go-openapi/jsonreference v1.0.0/go.mod h1:jtwdyGbJk0Xhe5Y+rwtglQP6Sb1WZST4rT32LWB+sv0= +github.com/go-openapi/swag v0.27.1 h1:VotvOLWW8q/EAxB0YdsBBGC8XYyeL1YwBj2ungAGPNg= +github.com/go-openapi/swag v0.27.1/go.mod h1:GTkJPwHfhJp6MWr4/rCh64HVI3Ofu+tcsbfjfHmTxpE= +github.com/go-openapi/swag/cmdutils v0.27.1 h1:I7sYqaWVl5mq0NEmNQkAmFDyNin9ufvMX/p2zwtQaOE= +github.com/go-openapi/swag/cmdutils v0.27.1/go.mod h1:Sm1MVFMkF6guJJ+pQqHnQA3N0j9qALV3NxzDSv6bETM= +github.com/go-openapi/swag/conv v0.27.1 h1:8wi9ZG+olmY1wXphl93EWniPtbSPkXM/feH7FgjsvrU= +github.com/go-openapi/swag/conv v0.27.1/go.mod h1:QbqMivkpKhC3g1B1GGGOJ6ANewI3S62dbzYu3Duowqs= +github.com/go-openapi/swag/fileutils v0.27.1 h1:QQqBSoi5mW4XpU85nS0mLcA+zAE6vLzrb0QkmLKf9oM= +github.com/go-openapi/swag/fileutils v0.27.1/go.mod h1:VvJFZLTZS0AI854gEQz5tk7dBESdLjiNUMSZ/th2ry8= +github.com/go-openapi/swag/jsonutils v0.27.1 h1:SVgK3i4USzCU5mibOOS/l4ea2h9UQXy7J7RNLTjuXjU= +github.com/go-openapi/swag/jsonutils v0.27.1/go.mod h1:tdlEpZqdcQ17uj6J4YdK9vd8It5qWMwjWXOs0tjpRlk= +github.com/go-openapi/swag/jsonutils/fixtures_test v0.27.1 h1:mJu3COL9WEaZVp/Kf2PRMi7tPszPEJfSr/OO75ynCs8= +github.com/go-openapi/swag/jsonutils/fixtures_test v0.27.1/go.mod h1:mofwUWx70wvskwESqRJ//k/9kURmCgyJl5m5Ppoh5kY= +github.com/go-openapi/swag/loading v0.27.1 h1:/DxUgDXKbBX4bcn7r9uEXfJyzN5XpiJmZplzQTjrRCY= +github.com/go-openapi/swag/loading v0.27.1/go.mod h1:jvGh3iA2+zyUUycB5fgJWzeHnhrpvGnJJM0RVE9ZShE= +github.com/go-openapi/swag/mangling v0.27.1 h1:yC9D0HyUE8gbP+BfmGx9+AA89ikwZTMjESK3OnnoaqA= +github.com/go-openapi/swag/mangling v0.27.1/go.mod h1:jtBE2+V+3pILxOR7Vgce+Cwp6A2PgZbvVqfNntbVs0w= +github.com/go-openapi/swag/netutils v0.27.1 h1:mICMFoS82F5TZ4Zy3cqmcQk+BFeCp3Uyq3Np7GI0/qU= +github.com/go-openapi/swag/netutils v0.27.1/go.mod h1:J+WYyFMLtvtCGqa6jLv+YNUmIKI3ZRQRrvfNDMoQoEQ= +github.com/go-openapi/swag/pools v0.27.1 h1:9LeadcMyb2GJCbXX5hVQDbZ2Lq9TL4dCs/nx1j5DO0E= +github.com/go-openapi/swag/pools v0.27.1/go.mod h1:kVQefhSK5RWuRe7BXsL8htgBPAMpN7HDGpGEknqugeE= +github.com/go-openapi/swag/stringutils v0.27.1 h1:ZXePZ0r2p1qSjo8tD3Un4vFj8+FqlCkczxDrJIhYUp8= +github.com/go-openapi/swag/stringutils v0.27.1/go.mod h1:lzRN95CxXmA03XcDWHLOb6nOMcxCqR5rGY0lOgsfRoM= +github.com/go-openapi/swag/typeutils v0.27.1 h1:KSTdFlfnse4r6dP9IrEnwMldjE+zs71UeEB3//PtVXc= +github.com/go-openapi/swag/typeutils v0.27.1/go.mod h1:Srm0xFNRZ1Y+vCxJclo5qzx8aj+1pAKda/YfFPrG0dQ= +github.com/go-openapi/swag/yamlutils v0.27.1 h1:ftxv6xvXb1E3zohUc+okZ9nSqNb9StQX/FXnKZ98sQA= +github.com/go-openapi/swag/yamlutils v0.27.1/go.mod h1:bnxFIB1qewGRiZHypXGZ3fNgf13/0HfRgnS/iZBDrOo= +github.com/go-openapi/testify/enable/yaml/v2 v2.6.0 h1:gGHwAJ0R/5jU8BEGDbfRNR3hL68dAVi84WuOApp29B0= +github.com/go-openapi/testify/enable/yaml/v2 v2.6.0/go.mod h1:tY+St1SGq4NFl0QIqdTY4aEdbChAHxhyB77XQi9iJCo= +github.com/go-openapi/testify/v2 v2.6.0 h1:5PKH2HE7YJ/LuRPQGvSxBRlFXNQhSetBLlGAgUEu3ug= +github.com/go-openapi/testify/v2 v2.6.0/go.mod h1:SgsVHtfooshd0tublTtJ50FPKhujf47YRqauXXOUxfw= github.com/google/gnostic-models v0.7.0 h1:qwTtogB15McXDaNqTZdzPJRHvaVJlAl+HVQnLmJEJxo= github.com/google/gnostic-models v0.7.0/go.mod h1:whL5G0m6dmc5cPxKc5bdKdEN3UjI7OUGxBlw57miDrQ= github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= @@ -45,21 +70,14 @@ github.com/hinshun/vt10x v0.0.0-20220119200601-820417d04eec h1:qv2VnGeEQHchGaZ/u github.com/hinshun/vt10x v0.0.0-20220119200601-820417d04eec/go.mod h1:Q48J4R4DvxnHolD5P8pOtXigYlRuPLGl6moFx3ulM68= github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= -github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY= -github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 h1:Z9n2FFNUXsshfwJMBgNA0RU6/i7WVaAegv3PtuIHPMs= github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51/go.mod h1:CzGEWj7cYgsdH8dAjBGEr58BoE7ScuLd+fwFZ44+/x8= -github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= -github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= -github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= -github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0= -github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= github.com/mattn/go-colorable v0.1.2/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE= github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE= github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8= @@ -100,22 +118,17 @@ github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk= github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= -github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= -github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= -github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY= -github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA= +github.com/stretchr/objx v0.5.3 h1:jmXUvGomnU1o3W/V5h2VEradbpJDwGrzugQQvL0POH4= +github.com/stretchr/objx v0.5.3/go.mod h1:rDQraq+vQZU7Fde9LOZLr8Tax6zZvy4kuNKF+QYS+U0= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= -github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= -github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= -github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM= github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg= github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= -go.yaml.in/yaml/v2 v2.4.3 h1:6gvOSjQoTB3vt1l+CU+tSyi/HOjfOjRLJ4YwYZGwRO0= -go.yaml.in/yaml/v2 v2.4.3/go.mod h1:zSxWcmIDjOzPXpjlTTbAsKokqkDNAVtZO0WOMiT90s8= +go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ= +go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ= go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc= go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= @@ -126,8 +139,8 @@ golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE= golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU= -golang.org/x/oauth2 v0.34.0 h1:hqK/t4AKgbqWkdkcAeI8XLmbK+4m4G5YeQRrmiotGlw= -golang.org/x/oauth2 v0.34.0/go.mod h1:lzm5WQJQwKZ3nwavOZ3IS5Aulzxi68dUSgRHujetwEA= +golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs= +golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -148,8 +161,8 @@ golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8= golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M= -golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI= -golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4= +golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U= +golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= @@ -166,25 +179,25 @@ gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= -k8s.io/api v0.36.4 h1:RxrvqCL6vgH5/+UnTeu1IIFqYmGfy0hnyrod1rn35Oo= -k8s.io/api v0.36.4/go.mod h1:S2B3orCFBDhrgyWbLeuKcT2QdHIpQesBkCYSlWtwUOw= -k8s.io/apimachinery v0.36.4 h1:PT2UzkupGuAx/+xT5XjiMJ1WGpY3fn9/hdAvjweRet4= -k8s.io/apimachinery v0.36.4/go.mod h1:p2I2dipt7JHG+quVwQ1d02d28O4GdDi77RByQ13MTpk= -k8s.io/client-go v0.36.4 h1:MDvfDNvMSt0Br94SK8neviVlwL9qifw9B26hJCpD1K0= -k8s.io/client-go v0.36.4/go.mod h1:pNK4WKELbwlEDvtbE8l22lEZL5THYF61H5EealokZmA= +k8s.io/api v0.37.0 h1:Z//Vj9N7RA/yS2sDmxyeo7h+RR4zbUrd2vrd3Z0TbB4= +k8s.io/api v0.37.0/go.mod h1:LKXgcJWMc+f4OLbP5SFR8rulEg07zZhpi/zMULiBImk= +k8s.io/apimachinery v0.37.0 h1:Np2AbDtf8x6RDHiD8T9LbKJ9gaegeVNa8yNm5FuGKm0= +k8s.io/apimachinery v0.37.0/go.mod h1:RN3nhprFSCxOi5Selxd7oMTXOe/c+ZbcE7Im+TS2zkE= +k8s.io/client-go v0.37.0 h1:nsN31fy8wBySuZ+QRnKmrjRSQLOG2rvoGN0tKd12zhQ= +k8s.io/client-go v0.37.0/go.mod h1:FcGqw+Ll/gNQiq+nPGY1Oyt9y7SgDh1d3MW3RFDEbn0= k8s.io/klog/v2 v2.140.0 h1:Tf+J3AH7xnUzZyVVXhTgGhEKnFqye14aadWv7bzXdzc= k8s.io/klog/v2 v2.140.0/go.mod h1:o+/RWfJ6PwpnFn7OyAG3QnO47BFsymfEfrz6XyYSSp0= -k8s.io/kube-openapi v0.0.0-20260317180543-43fb72c5454a h1:xCeOEAOoGYl2jnJoHkC3hkbPJgdATINPMAxaynU2Ovg= -k8s.io/kube-openapi v0.0.0-20260317180543-43fb72c5454a/go.mod h1:uGBT7iTA6c6MvqUvSXIaYZo9ukscABYi2btjhvgKGZ0= -k8s.io/streaming v0.36.4 h1:RS5YlhrdBN2pKGVjgygGntdu6SNdsduyjGWGe3cX0vo= -k8s.io/streaming v0.36.4/go.mod h1:tJ6S2bZa2HxIBauguBbCWSCYyd93Grfz1+z3tcOvlDE= -k8s.io/utils v0.0.0-20260210185600-b8788abfbbc2 h1:AZYQSJemyQB5eRxqcPky+/7EdBj0xi3g0ZcxxJ7vbWU= -k8s.io/utils v0.0.0-20260210185600-b8788abfbbc2/go.mod h1:xDxuJ0whA3d0I4mf/C4ppKHxXynQ+fxnkmQH0vTHnuk= +k8s.io/kube-openapi v0.0.0-20260721132016-d427ff9ee9ad h1:oXImqH8mQNk7PmvzKhmN3ddJoY6OnyM225MXwGHPm0A= +k8s.io/kube-openapi v0.0.0-20260721132016-d427ff9ee9ad/go.mod h1:0/mqHCVhlumdJ3BhCfnjSZQE037nAhNodh1/hK0T8/I= +k8s.io/streaming v0.37.0 h1:iPBUZLZiKt5bV+lxJurASMOV07VuBhNpiwJt2//AWrM= +k8s.io/streaming v0.37.0/go.mod h1:APlJR26ZWRcVy5bIEj0QRrKUXROtBHPcxl2NT7EAzPU= +k8s.io/utils v0.0.0-20260626114624-be93311217bd h1:Ea7fgQ5we8Y9T0OX5o0dAHzQOBRI07D/dEYRaB9ZZEs= +k8s.io/utils v0.0.0-20260626114624-be93311217bd/go.mod h1:xDxuJ0whA3d0I4mf/C4ppKHxXynQ+fxnkmQH0vTHnuk= sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 h1:IpInykpT6ceI+QxKBbEflcR5EXP7sU1kvOlxwZh5txg= sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730/go.mod h1:mdzfpAEoE6DHQEN0uh9ZbOCuHbLK5wOm7dK4ctXE9Tg= sigs.k8s.io/randfill v1.0.0 h1:JfjMILfT8A6RbawdsK2JXGBR5AQVfd+9TbzrlneTyrU= sigs.k8s.io/randfill v1.0.0/go.mod h1:XeLlZ/jmk4i1HRopwe7/aU3H5n1zNUcX6TM94b3QxOY= -sigs.k8s.io/structured-merge-diff/v6 v6.3.3 h1:u08YRbVUi59ri4YD6cg0UqNM4Dimn0sIl+wldcx5PYw= -sigs.k8s.io/structured-merge-diff/v6 v6.3.3/go.mod h1:M3W8sfWvn2HhQDIbGWj3S099YozAsymCo/wrT5ohRUE= +sigs.k8s.io/structured-merge-diff/v6 v6.4.2 h1:qdOxHwrl2Kaag1aQEarlYcOA9vSyGCp3CIki3aW8c4Q= +sigs.k8s.io/structured-merge-diff/v6 v6.4.2/go.mod h1:M3W8sfWvn2HhQDIbGWj3S099YozAsymCo/wrT5ohRUE= sigs.k8s.io/yaml v1.6.0 h1:G8fkbMSAFqgEFgh4b1wmtzDnioxFCUgTZhlbj5P9QYs= sigs.k8s.io/yaml v1.6.0/go.mod h1:796bPqUfzR/0jLAl6XjHl3Ck7MiyVv8dbTdyT3/pMf4= From 82b547743f60687e6d0aeb6cfa8df7f9d99d9d72 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 3 Sep 2026 06:20:05 +0200 Subject: [PATCH 3/6] chore(deps): bump softprops/action-gh-release from 3.0.2 to 3.0.3 (#626) Bumps [softprops/action-gh-release](https://github.com/softprops/action-gh-release) from 3.0.2 to 3.0.3. - [Release notes](https://github.com/softprops/action-gh-release/releases) - [Changelog](https://github.com/softprops/action-gh-release/blob/master/CHANGELOG.md) - [Commits](https://github.com/softprops/action-gh-release/compare/3d0d9888cb7fd7b750713d6e236d1fcb99157228...efb35369e0ad2afab669f228072c1b0d510eae64) --- updated-dependencies: - dependency-name: softprops/action-gh-release dependency-version: 3.0.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/release.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 5419e9b..39595ff 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -271,7 +271,7 @@ jobs: echo "tag=$REF" >> "$GITHUB_OUTPUT" - name: Create GitHub Release - uses: softprops/action-gh-release@3d0d9888cb7fd7b750713d6e236d1fcb99157228 # v3.0.2 + uses: softprops/action-gh-release@efb35369e0ad2afab669f228072c1b0d510eae64 # v3.0.3 with: tag_name: ${{ steps.tag.outputs.tag }} name: ${{ steps.tag.outputs.tag }} From 070a0377037beadd9183d8e338e0eed2ee1fd55a Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 3 Sep 2026 06:20:26 +0200 Subject: [PATCH 4/6] chore(deps): bump sigstore/cosign-installer from 3.9.1 to 4.1.2 (#625) Bumps [sigstore/cosign-installer](https://github.com/sigstore/cosign-installer) from 3.9.1 to 4.1.2. - [Release notes](https://github.com/sigstore/cosign-installer/releases) - [Commits](https://github.com/sigstore/cosign-installer/compare/398d4b0eeef1380460a10c8013a76f728fb906ac...6f9f17788090df1f26f669e9d70d6ae9567deba6) --- updated-dependencies: - dependency-name: sigstore/cosign-installer dependency-version: 4.1.2 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/release.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 39595ff..c241150 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -106,7 +106,7 @@ jobs: cache: true - name: Install cosign - uses: sigstore/cosign-installer@398d4b0eeef1380460a10c8013a76f728fb906ac # v3.9.1 + uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2 with: cosign-release: 'v2.4.1' From fa4a67b56e014888b1023e8f0f3abce500c613dd Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 3 Sep 2026 06:20:36 +0200 Subject: [PATCH 5/6] chore(deps): bump actions/download-artifact from 7.0.0 to 8.0.1 (#624) Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 7.0.0 to 8.0.1. - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](https://github.com/actions/download-artifact/compare/37930b1c2abaa49bbe596cd826c3c89aef350131...3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c) --- updated-dependencies: - dependency-name: actions/download-artifact dependency-version: 8.0.1 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/release.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index c241150..fa7e224 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -224,7 +224,7 @@ jobs: ref: ${{ github.ref }} - name: Download all matrix artifacts - uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0 + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: path: dist/ merge-multiple: true From a06b5429bb1f6fd341153c69502be8d024d43834 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 3 Sep 2026 06:20:46 +0200 Subject: [PATCH 6/6] chore(deps): bump dorny/paths-filter from 4.0.2 to 4.0.3 (#622) Bumps [dorny/paths-filter](https://github.com/dorny/paths-filter) from 4.0.2 to 4.0.3. - [Release notes](https://github.com/dorny/paths-filter/releases) - [Changelog](https://github.com/dorny/paths-filter/blob/master/CHANGELOG.md) - [Commits](https://github.com/dorny/paths-filter/compare/7b450fff21473bca461d4b92ce414b9d0420d706...ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d) --- updated-dependencies: - dependency-name: dorny/paths-filter dependency-version: 4.0.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/e2e.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index 5fb9839..3acd4c7 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -63,7 +63,7 @@ jobs: outputs: e2e: ${{ steps.filter.outputs.e2e }} steps: - - uses: dorny/paths-filter@7b450fff21473bca461d4b92ce414b9d0420d706 # v4.0.2 + - uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3 id: filter with: # Union of both suites' dependency surface. Kind suite: the