fix(chart): reconcile the telemetry Collector in place once its token Secret exists

[LukasWodka]

Summary

The <release>-telemetry-status record for the skipped-no-token state promised that the skip "resolves itself once jobs-manager writes the Secret". Nothing performed that: the lookup deciding the Collector's three-state rule runs only inside a helm upgrade, and the hourly auto-upgrade tick only upgraded onto a newer published chart (already at latest; nothing to do / deployed version is ahead of repo … skipping). Measured on a fresh installer edge: the token Secret was present within minutes, three ticks passed, no DaemonSet — the Collector rendered a day later only because a new chart happened to ship. Tracked internally.

This PR does two things together:

1. Mechanism — same-version reconcile in the tick. When the tick has no newer chart to install, it now asks one question and acts on one answer: if the stored release manifest records the Collector as skipped-no-token (which, by the decider's construction, means telemetryCollector.enabled was unset), the stored manifest carries no Collector DaemonSet, the repo serves the deployed chart version, and a server-side dry-run of that same version now resolves enabled (the token Secret has arrived), the tick re-runs helm upgrade pinned to --version "$CURRENT" with the exact flags and values discipline of the normal upgrade (--reset-then-reuse-values --atomic --cleanup-on-fail --timeout). It logs one line naming why. Any other state behaves exactly as before.
2. Honest record. The reason text now says what actually triggers rendering: the next auto-upgrade tick (CronJob schedule read from autoUpgrade.schedule, repo from autoUpgrade.repoUrl — never restated), or, with autoUpgrade.enabled: false, "run helm upgrade on this release". The skipped-incomplete-values branch loses its "resolves itself" as well. No branch of the record promises a self-resolution any more.

The condition, in one place

telemetry_reconcile_verdict() in the rendered auto-upgrade.sh is the only decider — tri-state, and 2 never acts:

• 0 reconcile: stored state skipped-no-token and no DaemonSet named $TELEMETRY_COLLECTOR_NAME in the stored manifest and the repo serves $CURRENT and helm upgrade … --version "$CURRENT" --reset-then-reuse-values --dry-run=server --hide-secret resolves enabled.
• 1 nothing to do: any other stored state (enabled, disabled-by-operator, skipped-incomplete-values, no record at all — a release rendered by a chart predating it), or a dry-run that still says skipped-no-token. Quiet, as today.
• 2 cannot tell: helm get manifest unreadable/empty, dry-run failed or rendered no record, the record contradicts the manifest (says skipped, DaemonSet present), the TELEMETRY_* names missing from env, or the deployed version is not served by the repo. Logged as reconcile: NOT re-rendering, cannot tell …, no upgrade.

Derived, not restated. No Secret name, Class A rule or DaemonSet gate is copied into the script. The decision is read from the chart's own decider twice — the stored state out of helm get manifest, the would-be state out of --dry-run=server (helm 3.13+; the image is pinned at 3.16.4), which runs the templates' real lookups (release-scoped, legacy and pre-override Secret names alike) against the cluster. The three names the awk matches on (TELEMETRY_STATUS_NAME, TELEMETRY_STATUS_ANNOTATION, TELEMETRY_COLLECTOR_NAME) arrive in the CronJob env from two new helpers (tracebloc.telemetryStatusName, tracebloc.telemetryStateAnnotation) plus the existing tracebloc.telemetryCollectorName — the same helpers the status template and the DaemonSet render with, pinned by tests/telemetry_status_wiring_test.yaml (including under fullnameOverride). The image has no kubectl, and a bare curl against the API is exactly what the house style forbids, so helm is the only client used.

Deployed ahead of the index (current > latest)

This is the state the ticket caught, and it deserves a plain statement: an edge running a chart version the repo does not serve (a dev chart installed from a local path) cannot be re-rendered by the tick, because there is nothing it can fetch — helm upgrade needs a chart, and the deployed one exists only inside the release. The verdict names it every tick (deployed chart X is not served by <repo> (newest there: Y): a same-version re-render is impossible, the Collector renders on the next published chart), and the status record says the same. The reconcile fixes the fleet case — an edge on the published chart whose Secret arrived after install, which until now logged already at latest; nothing to do forever.

Type

• Bug fix
• Chart change (Chart.yaml version and appVersion 1.9.116 → 1.9.117)

Test plan

Everything CI runs for the chart tier, locally at CI severity (outputs kept):

• make lint (bash -n on every script + shellcheck -S error) — green; shellcheck -S warning -x on both new gates — clean.
• make drift — drift: all 50 guards green (48 existing + the 2 new ones; auto-upgrade-inflight-vs-wedge.sh unchanged and green) (includes the two new guards).
• make helm-lint helm-vocab — 5 lints green, chart-env-vocabulary: all 177 checks passed.
• make helm-template with kubeconform — aks/bm/eks/oc all Invalid: 0, Errors: 0.
• make helm-unittest — Test Suites: 42 passed, 42 total / Tests: 750 passed, 750 total (was 743; +7 here).
• make bats — bats scripts/tests/*.bats 2067 ok, 0 not ok, rc=0.
• Rendered script parses under the image's own busybox sh -n (alpine/helm:3.16.4).

New checks

• scripts/tests/auto-upgrade-telemetry-reconcile.sh (registered in make drift) — drives the rendered script with a stub helm on PATH, 23 cases: the full tick end to end (Secret arrived → reconcile pinned to $CURRENT with all flags; Secret still absent / enabled / disabled-by-operator / incomplete-values / no record → no upgrade; contradictory record / unreadable manifest / failed dry-run / dry-run without record / names missing → no upgrade and the log says cannot tell; ahead-of-index → named, no upgrade; newer chart → normal upgrade, dry-run never consulted) and the extracted telemetry_reconcile_verdict function directly for the rc 0/1/2 table. Fixtures are real helm template renders of this chart, not typed YAML; the names come from the rendered CronJob's env.
• scripts/tests/auto-upgrade-telemetry-reconcile-mutations.sh (registered in make drift) — the permanent mutation proof, each asserting the specific finding: (a) drop the DaemonSet-absent check → contradictory case reconciles, gate red; (b) accept a skipped-no-token dry-run → Secret-still-absent case reconciles, gate red; (c) unhook the reconcile from the at-latest branch → Secret-arrived case does not reconcile, gate red; (d) restore "this resolves itself once it does" in the status template → telemetry_collector_test.yaml reddens on "the no-token skip names the trigger that actually re-renders it, not a self-resolution". All four seen red, tree as shipped seen green first.
• helm-unittest: 4 status-record cases in telemetry_collector_test.yaml (via kubernetesProvider, which is what lets the skipped-no-token state be reached offline — a kube-system Namespace and no Secret), 1 script-shape case in auto_upgrade_test.yaml, and the 2-case wiring suite.

Deliberately not changed

• telemetryCollector.enabled stays tri-state with no default; the collector is not defaulted on (option 4 of the internal ticket). The refusal-when-enabled-and-no-token path is untouched.
• The existing auto-upgrade-inflight-vs-wedge.sh gate still passes unchanged: it runs the script without the TELEMETRY_* env, which now takes the fail-closed 2 branch and logs, rather than erroring under set -u.
• No installer script touched, so manifest.sha256 is unchanged.
• Log text already at latest; nothing to do became already at latest; nothing to upgrade (and … skipping → … skipping the upgrade) so the line is not contradicted by a reconcile that follows it; nothing in the repo asserted the old strings.

Checklist

• Version bumped (Chart.yaml version + appVersion)
• Tests added / updated, mutation-proved
• shellcheck -S warning -x + bash -n clean on every shell touched
• No customer names, internal paths or internal tracker refs in this title/body (tracked internally)

---

Note

Medium Risk
Changes fleet auto-upgrade to run conditional same-version helm upgrades and depends on server-side dry-run lookups; mistakes could trigger extra upgrades or leave collectors disabled, but tri-state fail-closed logic and extensive new tests mitigate that.

Overview
Fixes a gap where the telemetry Collector stayed off after the token Secret appeared because collector state only updates on helm upgrade, while the hourly auto-upgrade job previously exited when there was no newer chart.

When the release is already at latest (or ahead of the repo index), the tick now runs telemetry_reconcile_verdict: if the stored manifest shows skipped-no-token, no Collector DaemonSet, the repo serves the deployed version, and a server-side dry-run of that same version would resolve enabled, it performs a same-version helm upgrade with the usual flags. Ambiguous cases fail closed with logged “cannot tell” and no upgrade.

The telemetry-status ConfigMap stops claiming skips “resolve themselves”; it names the real trigger (next auto-upgrade tick with schedule/repo from values, manual upgrade if auto-upgrade is off, or wait for a published chart on dev builds). Shared helpers tracebloc.telemetryStatusName and tracebloc.telemetryStateAnnotation wire the CronJob env to the status template. Chart 1.9.118; drift guards, helm-unittest cases, and mutation proofs exercise the behavior (record-honesty mutation runs in the Helm unit tests job where helm-unittest is installed).

^{Reviewed by Cursor Bugbot for commit 47b68ac. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

bugbot run

[LukasWodka]

bugbot run

[LukasWodka]

bugbot run

[LukasWodka]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 47b68ac. Configure here.}