fix(cluster): refuse a mutating command on a cluster that is not the install's (backend#2863) - #608
Conversation
…install's (backend#2863) Every command resolved its target cluster from the ambient kubeconfig + current-context while binding only the NAMESPACE from the active client. On a machine whose current-context points elsewhere -- a laptop that also administers a managed cluster, the normal case for anyone running both -- a mutating command acted on that other cluster: `data ingest` staged a private dataset onto it, `data delete` dropped a table and removed files from its shared PVC, `resources set` rolled its jobs-manager, and `tracebloc delete` uninstalled a release of the same name. The namespace binding made it MORE likely, not less: it supplied a namespace that probably exists on the other cluster too, so discovery succeeded and nothing on screen looked wrong. - record the cluster anchor (kube-system UID -- the same value the backend client record keys on) at `client create`, clear it on offboard - resolveClusterTarget takes a `mutates bool`: the compiler makes every caller decide, so a new command cannot forget an opt-in helper - `delete` guards separately and EARLY -- it shells out to helm and never resolved a target -- so a refusal happens before the credential is revoked Deliberately asymmetric: a mismatch refuses, an unreadable identity refuses for data commands (we are about to write to a cluster we cannot name), and an unrecorded anchor warns and proceeds so existing installs are not locked out. `delete` inverts the middle one: an unreachable cluster is the main reason to offboard, so it must not block. Ten mutations proven to redden their own test, each with the anchor asserted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 29f7f4b. Configure here.
saqlainsyed007
left a comment
There was a problem hiding this comment.
Reviewed at b7cc1878 (high effort — a destructive-op safety guard, 694/-16 across 20 files). Exemplary work: right altitude, correct ordering on every path, and mutation-proven. Approving.
What it is (backend#2863). Every cluster-touching command resolved its target from the ambient kubeconfig's current-context while binding only the namespace from the active client — so on a laptop whose current-context points at some other cluster (the normal case for anyone who also administers a managed cluster), a mutating command acted on that cluster: data ingest staged a private dataset onto it, data delete dropped a table off it, resources set rolled its jobs-manager, and tracebloc delete fired helm uninstall at it — observed in the field against an EKS endpoint. The namespace binding made it more likely, not less, because the namespace probably existed there too. This records which cluster the secure environment runs on (the kube-system UID — the same anchor ProvisionedClient.ClusterID carries, so local and remote agree by construction) and refuses a mutating command that reaches a different one.
I verified the guard on every path, and the ordering is the crux:
resolveClusterTargettakes a requiredmutates bool— a positional the compiler forces every caller to decide, so a new command cannot forget it (an opt-in helper is one it could). Confirmed the check runs beforeDiscoverSharedPVCand before the target is returned, so a mutating command cannot have touched anything by the time it's refused. The three mutating call sites passtrue(data ingest,data delete,resources set); the read-onlydata listpassesfalseand is correctly ungated.tracebloc deletenever resolves a target (it hands raw flags tohelm uninstall), so it guards separately and early — before the credential is revoked and before any teardown — leaving a refusal fully re-runnable.
The asymmetry is deliberate and I checked both directions: for data commands, a remote identity we cannot read refuses (about to write to a cluster we can't name), while a missing local anchor or unreadable local config warns-and-proceeds (don't lock out pre-existing configs — the warning names client create as the fix). For delete it's inverted: an unreachable cluster proceeds silently (a dead cluster is the main reason to offboard), and only a positively-different identity refuses. The check is on identity, not context-name, so BYO clusters (EKS/AKS/OpenShift, no k3d context) aren't broken.
Tests are the house bar: behavioural cases construct the wrong-cluster scenario so each fails on the pre-fix code by the command proceeding, not by a message change; two source-derived anti-rot guards (TestEveryClusterCallSiteDeclaresMutationIntent reddens on any new ungated call site; TestActiveClientHasOneWritePath pins the single anchor-writing path, since the guard is silent when the anchor is missing); both fail closed on an unparseable file. Ten mutations each mapped to the specific test that reddens with the anchor asserted — and the PR even names a caught test-defect (withAnchor first wrote to a throwaway &Profile{} so the anchor never persisted and the tests passed for the wrong reason; now it reloads and asserts the write landed).
VERSION bump is the required version-bump-gate (green). CI green, MERGEABLE, no open threads. This is real attack-surface reduction on a mistake that already happened. LGTM.
Summary
Closes tracebloc/backend#2863.
Every cluster-touching command resolved its target from the ambient kubeconfig + current-context, while binding only the namespace from the active client. On a machine whose current-context points somewhere else — a laptop that also administers a managed cluster, which is the normal case for anyone who runs both — a mutating command acted on that other cluster:
data ingestdata deleteresources settracebloc deleteThe namespace binding made this more likely, not less: it supplied a namespace that probably exists on the other cluster too, so discovery succeeded and nothing on screen looked wrong. This was observed in the field — a
tb deletefiredhelm uninstallat an EKS endpoint.The hazard was already documented, at
internal/nodeboot/nodeboot.go, whose comment ends "preserving the default-context behavior". That clause was the bug.What changed
config.Profile.ActiveClientClusterID— records which cluster, not just which namespace. The value is the kube-system namespace UID: the same anchorapi.ProvisionedClient.ClusterIDcarries, read by the samecluster.ClusterID, so local and remote agree by construction. Recorded atclient create, cleared on offboard.resolveClusterTargettakes amutates bool. Not a convenience flag — the compiler making every caller decide. An opt-inguard()helper is one a new command forgets to call; a required parameter is one it cannot. The check runs before PVC discovery and before the target is returned, so a mutating command cannot have touched anything by the time it is told this is the wrong cluster.deleteguards separately, and early. It never resolved a target at all — it hands the raw flags tohelm uninstall— so the check sits before the credential is revoked and before any teardown step. A refusal leaves the machine exactly as it was, and the command stays re-runnable.The check is on identity, not context name. Pinning a context string would break bring-your-own-cluster installs (EKS/AKS/OpenShift have no k3d context) and would still pass if two kubeconfigs named the same context differently.
The failure modes are deliberately asymmetric
deleteThe
deleteinversion is the one that matters: blocking there would make a dead cluster unremovable. The no-anchor case warns rather than refusing so configs written before this field are not locked out of their own commands — and the warning namesclient createas the fix.Read-only commands (
data list,resources) are not gated. Being wrong about which cluster you are reading is a confusing answer, not a destructive act, and the target is already printed.Test plan
go build ./... && go vet ./... && go test ./... -count=1— green;gofmtclean.Behavioural tests construct the wrong-cluster case explicitly, so each fails on the pre-fix code by the command proceeding, not by a message change.
TestDelete_WrongCluster_RefusesBeforeAnyChangeasserts the credential was not revoked and no teardown step ran.Two anti-rot properties, both derived from source rather than restating a list:
TestEveryClusterCallSiteDeclaresMutationIntentparses everyresolveClusterTargetcall site and requires a recorded intent, a literaltrue/falseat the call site, and no stale rows. A new call site reddens until someone decides.TestActiveClientHasOneWritePathpins that onlysetActiveClientactivates a client — the guard is silent when the anchor is missing, so a second write path that forgot it would disable the guard with nothing in any log to say so.Both fail closed: an unparseable file is "cannot tell", not "agrees".
Ten mutations, each proven to redden its own test with the anchor asserted (an inert mutation and good coverage look identical in a log):
Mutating_WrongClusterMutating_WrongClusterMutating_UnreadableIDMutating_NoAnchordelete's guard removedDelete_WrongClusterdeleterefuses on an unreachable clusterDelete_Unreachabledeleteleaves the anchor behindDelete_ClearsTheAnchorEveryClusterCallSiteRecordsTheClusterAnchorActiveClientHasOneWritePathOne caught defect worth naming: the first version of
withAnchorwrote tocfg.Current()without selecting an env.Current()returns a throwaway&Profile{}in that case, so the anchor never persisted and the refusal tests passed for the wrong reason.withAnchornow reloads and asserts the write landed.Checklist
develop🤖 Generated with Claude Code
Note
High Risk
Changes safety gates for destructive cluster operations (data delete, ingest, offboard, resources); behavior shifts when kubeconfig context disagrees with the recorded anchor, though legacy configs without an anchor only get warnings.
Overview
Fixes backend#2863: mutating CLI commands could hit the cluster from the ambient kubeconfig current-context while only the active client’s namespace was pinned—so ingest, delete,
resources set, or offboard could run against another cluster (e.g. EKS) with no obvious signal.The PR records which cluster the install belongs on:
ActiveClientClusterID(kube-system UID, same as the backend) is saved inclient createviasetActiveClientand cleared on offboard.resolveClusterTargetnow requires amutatesflag; when true it runsguardActiveClientClusteron the already-built clientset (via newcluster.ClusterIDFrom) before PVC discovery or returning the target. Mutating callers (data ingest/delete,resources set,seal) passtrue; reads (data list,resources show) passfalse.tracebloc deleteadds a separate early identity check (it never usedresolveClusterTarget) so refusal happens before revoke or Helm teardown.Failure handling is intentional: cluster ID mismatch or unreadable ID on mutating paths → refuse; missing anchor → warn and proceed; delete still allows offboard when the cluster is unreachable. Tests cover wrong-cluster refusal, delete ordering, anchor lifecycle, and AST-based checks so new
resolveClusterTargetcall sites must declare mutation intent.Reviewed by Cursor Bugbot for commit b7cc187. Bugbot is set up for automated code reviews on this repo. Configure here.