Skip to content

fix(cluster): refuse a mutating command on a cluster that is not the install's (backend#2863) - #608

Merged
LukasWodka merged 2 commits into
developfrom
fix/2863-cluster-identity-guard
Aug 31, 2026
Merged

fix(cluster): refuse a mutating command on a cluster that is not the install's (backend#2863)#608
LukasWodka merged 2 commits into
developfrom
fix/2863-cluster-identity-guard

Conversation

@LukasWodka

@LukasWodka LukasWodka commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes tracebloc/backend#2863.

Every cluster-touching command resolved its target from the ambient kubeconfig + current-context, while binding only the namespace from the active client. On a machine whose current-context points somewhere else — a laptop that also administers a managed cluster, which is the normal case for anyone who runs both — a mutating command acted on that other cluster:

Command What it did to the wrong cluster
data ingest staged a private dataset onto it
data delete dropped a table, removed files from its shared PVC
resources set rolled its jobs-manager with a new envelope
tracebloc delete uninstalled a release of the same name

The namespace binding made this more likely, not less: it supplied a namespace that probably exists on the other cluster too, so discovery succeeded and nothing on screen looked wrong. This was observed in the field — a tb delete fired helm uninstall at an EKS endpoint.

The hazard was already documented, at internal/nodeboot/nodeboot.go, whose comment ends "preserving the default-context behavior". That clause was the bug.

What changed

  • config.Profile.ActiveClientClusterID — records which cluster, not just which namespace. The value is the kube-system namespace UID: the same anchor api.ProvisionedClient.ClusterID carries, read by the same cluster.ClusterID, so local and remote agree by construction. Recorded at client create, cleared on offboard.
  • resolveClusterTarget takes a mutates bool. Not a convenience flag — the compiler making every caller decide. An opt-in guard() helper is one a new command forgets to call; a required parameter is one it cannot. The check runs before PVC discovery and before the target is returned, so a mutating command cannot have touched anything by the time it is told this is the wrong cluster.
  • delete guards separately, and early. It never resolved a target at all — it hands the raw flags to helm uninstall — so the check sits before the credential is revoked and before any teardown step. A refusal leaves the machine exactly as it was, and the command stays re-runnable.

The check is on identity, not context name. Pinning a context string would break bring-your-own-cluster installs (EKS/AKS/OpenShift have no k3d context) and would still pass if two kubeconfigs named the same context differently.

The failure modes are deliberately asymmetric

Situation Data commands delete
Identified as a different cluster refuse refuse
Identity unreadable refuse — about to write to a cluster we cannot name; every caller needs API access anyway proceed — an unreachable cluster is the main reason to offboard
No anchor recorded warn + proceed warn + proceed

The delete inversion is the one that matters: blocking there would make a dead cluster unremovable. The no-anchor case warns rather than refusing so configs written before this field are not locked out of their own commands — and the warning names client create as the fix.

Read-only commands (data list, resources) are not gated. Being wrong about which cluster you are reading is a confusing answer, not a destructive act, and the target is already printed.

Test plan

go build ./... && go vet ./... && go test ./... -count=1 — green; gofmt clean.

Behavioural tests construct the wrong-cluster case explicitly, so each fails on the pre-fix code by the command proceeding, not by a message change. TestDelete_WrongCluster_RefusesBeforeAnyChange asserts the credential was not revoked and no teardown step ran.

Two anti-rot properties, both derived from source rather than restating a list:

  • TestEveryClusterCallSiteDeclaresMutationIntent parses every resolveClusterTarget call site and requires a recorded intent, a literal true/false at the call site, and no stale rows. A new call site reddens until someone decides.
  • TestActiveClientHasOneWritePath pins that only setActiveClient activates a client — the guard is silent when the anchor is missing, so a second write path that forgot it would disable the guard with nothing in any log to say so.

Both fail closed: an unparseable file is "cannot tell", not "agrees".

Ten mutations, each proven to redden its own test with the anchor asserted (an inert mutation and good coverage look identical in a log):

# Mutation Test that reddened
M1 guard never invoked for mutating commands Mutating_WrongCluster
M2 mismatch warns instead of refusing Mutating_WrongCluster
M3 unreadable identity proceeds Mutating_UnreadableID
M4 no-anchor refuses instead of warning Mutating_NoAnchor
M5 delete's guard removed Delete_WrongCluster
M6 delete refuses on an unreachable cluster Delete_Unreachable
M7 delete leaves the anchor behind Delete_ClearsTheAnchor
M8 a call site silently flips to read-only EveryClusterCallSite
M9 install records namespace but not cluster RecordsTheClusterAnchor
M10 a second activation path appears ActiveClientHasOneWritePath

One caught defect worth naming: the first version of withAnchor wrote to cfg.Current() without selecting an env. Current() returns a throwaway &Profile{} in that case, so the anchor never persisted and the refusal tests passed for the wrong reason. withAnchor now reloads and asserts the write landed.

Checklist

  • Targets develop
  • Tests added, and mutation-proven
  • No behaviour change for a correctly-pointed kubeconfig or an unanchored config
  • User-facing copy regenerated into the golden catalog

🤖 Generated with Claude Code


Note

High Risk
Changes safety gates for destructive cluster operations (data delete, ingest, offboard, resources); behavior shifts when kubeconfig context disagrees with the recorded anchor, though legacy configs without an anchor only get warnings.

Overview
Fixes backend#2863: mutating CLI commands could hit the cluster from the ambient kubeconfig current-context while only the active client’s namespace was pinned—so ingest, delete, resources set, or offboard could run against another cluster (e.g. EKS) with no obvious signal.

The PR records which cluster the install belongs on: ActiveClientClusterID (kube-system UID, same as the backend) is saved in client create via setActiveClient and cleared on offboard. resolveClusterTarget now requires a mutates flag; when true it runs guardActiveClientCluster on the already-built clientset (via new cluster.ClusterIDFrom) before PVC discovery or returning the target. Mutating callers (data ingest/delete, resources set, seal) pass true; reads (data list, resources show) pass false. tracebloc delete adds a separate early identity check (it never used resolveClusterTarget) so refusal happens before revoke or Helm teardown.

Failure handling is intentional: cluster ID mismatch or unreadable ID on mutating paths → refuse; missing anchor → warn and proceed; delete still allows offboard when the cluster is unreachable. Tests cover wrong-cluster refusal, delete ordering, anchor lifecycle, and AST-based checks so new resolveClusterTarget call sites must declare mutation intent.

Reviewed by Cursor Bugbot for commit b7cc187. Bugbot is set up for automated code reviews on this repo. Configure here.

…install's (backend#2863)

Every command resolved its target cluster from the ambient kubeconfig +
current-context while binding only the NAMESPACE from the active client. On a
machine whose current-context points elsewhere -- a laptop that also administers
a managed cluster, the normal case for anyone running both -- a mutating command
acted on that other cluster: `data ingest` staged a private dataset onto it,
`data delete` dropped a table and removed files from its shared PVC,
`resources set` rolled its jobs-manager, and `tracebloc delete` uninstalled a
release of the same name.

The namespace binding made it MORE likely, not less: it supplied a namespace
that probably exists on the other cluster too, so discovery succeeded and
nothing on screen looked wrong.

- record the cluster anchor (kube-system UID -- the same value the backend
  client record keys on) at `client create`, clear it on offboard
- resolveClusterTarget takes a `mutates bool`: the compiler makes every caller
  decide, so a new command cannot forget an opt-in helper
- `delete` guards separately and EARLY -- it shells out to helm and never
  resolved a target -- so a refusal happens before the credential is revoked

Deliberately asymmetric: a mismatch refuses, an unreadable identity refuses for
data commands (we are about to write to a cluster we cannot name), and an
unrecorded anchor warns and proceeds so existing installs are not locked out.
`delete` inverts the middle one: an unreachable cluster is the main reason to
offboard, so it must not block.

Ten mutations proven to redden their own test, each with the anchor asserted.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodka LukasWodka self-assigned this Aug 31, 2026
@LukasWodka

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 29f7f4b. Configure here.

@shujaatTracebloc
shujaatTracebloc requested review from saadqbal and removed request for saqlainsyed007 August 31, 2026 09:42
@aptracebloc
aptracebloc requested review from saqlainsyed007 and removed request for saadqbal August 31, 2026 09:56
@shujaatTracebloc
shujaatTracebloc requested review from saadqbal and removed request for saqlainsyed007 August 31, 2026 09:57
@aptracebloc
aptracebloc requested review from saqlainsyed007 and removed request for saadqbal August 31, 2026 10:15

@saqlainsyed007 saqlainsyed007 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at b7cc1878 (high effort — a destructive-op safety guard, 694/-16 across 20 files). Exemplary work: right altitude, correct ordering on every path, and mutation-proven. Approving.

What it is (backend#2863). Every cluster-touching command resolved its target from the ambient kubeconfig's current-context while binding only the namespace from the active client — so on a laptop whose current-context points at some other cluster (the normal case for anyone who also administers a managed cluster), a mutating command acted on that cluster: data ingest staged a private dataset onto it, data delete dropped a table off it, resources set rolled its jobs-manager, and tracebloc delete fired helm uninstall at it — observed in the field against an EKS endpoint. The namespace binding made it more likely, not less, because the namespace probably existed there too. This records which cluster the secure environment runs on (the kube-system UID — the same anchor ProvisionedClient.ClusterID carries, so local and remote agree by construction) and refuses a mutating command that reaches a different one.

I verified the guard on every path, and the ordering is the crux:

  • resolveClusterTarget takes a required mutates bool — a positional the compiler forces every caller to decide, so a new command cannot forget it (an opt-in helper is one it could). Confirmed the check runs before DiscoverSharedPVC and before the target is returned, so a mutating command cannot have touched anything by the time it's refused. The three mutating call sites pass true (data ingest, data delete, resources set); the read-only data list passes false and is correctly ungated.
  • tracebloc delete never resolves a target (it hands raw flags to helm uninstall), so it guards separately and early — before the credential is revoked and before any teardown — leaving a refusal fully re-runnable.

The asymmetry is deliberate and I checked both directions: for data commands, a remote identity we cannot read refuses (about to write to a cluster we can't name), while a missing local anchor or unreadable local config warns-and-proceeds (don't lock out pre-existing configs — the warning names client create as the fix). For delete it's inverted: an unreachable cluster proceeds silently (a dead cluster is the main reason to offboard), and only a positively-different identity refuses. The check is on identity, not context-name, so BYO clusters (EKS/AKS/OpenShift, no k3d context) aren't broken.

Tests are the house bar: behavioural cases construct the wrong-cluster scenario so each fails on the pre-fix code by the command proceeding, not by a message change; two source-derived anti-rot guards (TestEveryClusterCallSiteDeclaresMutationIntent reddens on any new ungated call site; TestActiveClientHasOneWritePath pins the single anchor-writing path, since the guard is silent when the anchor is missing); both fail closed on an unparseable file. Ten mutations each mapped to the specific test that reddens with the anchor asserted — and the PR even names a caught test-defect (withAnchor first wrote to a throwaway &Profile{} so the anchor never persisted and the tests passed for the wrong reason; now it reloads and asserts the write landed).

VERSION bump is the required version-bump-gate (green). CI green, MERGEABLE, no open threads. This is real attack-surface reduction on a mistake that already happened. LGTM.

@LukasWodka
LukasWodka merged commit 7ea346e into develop Aug 31, 2026
31 checks passed
@LukasWodka
LukasWodka deleted the fix/2863-cluster-identity-guard branch August 31, 2026 10:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants