release-train: staging -> main - #1052
Merged
Merged
Conversation
…ining pods follow the tracebloc registry default (ghcr.io) (#1041) What: both JOB_IMAGE_HOST render sites in jobs-manager-deployment.yaml (the api container and pods-monitor) now render `{{ include "tracebloc.tbRegistry" . }}/` instead of `<global.imageRegistry|docker.io>/`. At the chart default the jobs-manager stamps `ghcr.io/` onto every training image it spawns (tracebloc/client-<task>-<arch>:<CLIENT_ENV>); every one of those images is published on GHCR at the same digest as its Docker Hub copy (the GHCR migration), so this changes where the training pods pull from, not which bytes run. No new egress: ghcr.io is already required for the control-plane images (1.9.113) and the ingestor. Why: the control plane moved to ghcr.io in 1.9.113 while the training host stayed on a docker.io literal, so one install pulled the two halves of the product from two registries and a rollback of one did not move the other. ONE precedence chain, the same helper the control-plane images, the image-refresh CronJob and NOTES.txt already read: `global.imageRegistry` (a mirror re-homes everything) wins, else `images.traceblocRegistry`, else the chart default "ghcr.io". The helper is nil-guarded and `| default`-chained, so an empty or absent knob renders `ghcr.io/`, never a bare "/". Rollback is the SAME knob as 1.9.113 and moves the control plane and the training host together: helm upgrade <release> tracebloc/client -n <ns> \ --reset-then-reuse-values --set images.traceblocRegistry=docker.io Mirrors are unaffected: `global.imageRegistry` still wins for JOB_IMAGE_HOST exactly as before. The client-runtime's own fallback for an UNSET JOB_IMAGE_HOST changes separately in that project; the chart always sets the variable, so only installs predating the key ever see that default. Tests (helm-unittest 0.5.2, one template per test): jobs_manager_test: default -> "ghcr.io/" on both containers global_image_registry_test: mirror still wins on both containers; no mirror -> the tracebloc registry default, not docker.io tracebloc_registry_test: the "does NOT move ... JOB_IMAGE_HOST" case is split: the ingestor repository still does not follow the knob; JOB_IMAGE_HOST now does (docker.io -> "docker.io/", quay.example -> "quay.example/"), an EMPTY knob renders the default (dig-empty trap), and mirror + knob -> mirror wins. NOTES now says "control-plane + training images". Contract text updated to match: _helpers.tpl doc block, values.yaml comment, values.schema.json description, docs/INSTALL.md (egress table + registry paragraph), a 1.9.114 MIGRATION.md entry, and scripts/list-images.sh's derivation note (the task NAMES still come from Docker Hub's repository list, which GHCR has no anonymous equivalent of; the host prefix is the rendered JOB_IMAGE_HOST -- dual-published, so the names agree). Chart 1.9.113 -> 1.9.114 (template change). Suite counts: helm unittest 733/733 (41 suites; was 729, +5 new, -1 replaced); helm lint --strict clean on aks/bm/eks/oc; helm template x4 renders "ghcr.io/" at both sites with every control-plane image: line byte-identical to the base; make drift 48/48 guards; chart-env-vocabulary 177/177. Mutations (helm unittest client; anchor asserted applied, restored byte-for-byte, 733/733 green after each): M1 first JOB_IMAGE_HOST site back to the mirror-or-docker.io dig -> 4 failed / 733 (default on both containers x2 suites, quay.example knob, EMPTY-knob trap) M2 both sites hardcoded to "ghcr.io/" (helper ignored) -> 4 failed / 733 (mirror re-home, docker.io rollback knob, quay.example knob, mirror-wins-over-knob) bats: every scripts/tests/*.bats that reads values/templates/helm template (23 files, 1057 tests) green on the final tree.
…verable mirror (#1034) * ci(mirror): fail-closed publish guard + publisher for the public deliverable mirror The public face of this repo becomes a separate, deliverable-only mirror (charts, installer, README/LICENSE/docs, the Pages chart index, releases). Nothing reaches it except through the new Mirror publish workflow, and nothing leaves that workflow except what scripts/publish-guard.sh staged from the explicit allowlist (.publish-include) and cleared through four guards: forbidden paths and forbidden strings (.publish-forbidden), and gitleaks. Every guard fails closed; "could not tell" never publishes. scripts/publish-mirror.sh does the push: it refuses an unset mirror and a mirror equal to this repository, pushes plainly (never force), and never overwrites an existing release. The workflow runs after Release Helm Chart completes and on dispatch with dry-run defaulting to true; the mirror name (MIRROR_REPO) has no default. Customer identifiers for the string scan are supplied privately at publish time, not committed to this public file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(mirror): split the string guard into refuse and report tiers A guard that can never pass is a guard nobody arms: on the real tree the single [strings] tier refused with 825 hits, all of them internal ticket identifiers and non-production hostnames whose fate is still a decision, not a defect. The scan now has two tiers in .publish-forbidden: - [strings-refuse] mailboxes (support@ spared), arn:aws:, ECR account ids, plus the private tenant needles from --extra-forbidden. A hit refuses, as before. - [strings-report] ticket/RFC identifiers and non-production hosts. Hits are counted and printed (per-needle totals, ten most-hit files) but refuse only under --strict. The list itself is refused (exit 2) on an unknown section header, a needle in both tiers, or an empty [strings-refuse]. The header regex is loose on purpose so a misspelt header is refused by name rather than read as a needle of the section before it. The workflow gains a `strict` dispatch input (default false) and honours the PUBLISH_STRICT repository variable for every run, including workflow_run, which has no inputs; flipping either arms the report tier. Tests: 39 -> 47. Report-tier hit alone -> exit 0 with count and table; the same under --strict -> exit 1 naming [strings-report (strict)]; refuse-tier hit -> exit 1 naming [strings-refuse]; duplicate needle, unknown section and empty refuse tier -> exit 2 with the offender named. The committed list is exercised needle by needle with inputs written in the test, and the real tree is now asserted clean (exit 0). Each new check was mutation-proved. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(mirror): run the tooling from this commit, treat the tag as data, keep prereleases off the default branch Review findings on the mirror-publish workflow, each with its test: - Untrusted checkout. The job checked out the release tag and then ran scripts/publish-guard.sh and publish-mirror.sh from it while the job went on to mint an App token. Now the one actions/checkout has no ref (the workflow's own commit), and the release tag is fetched separately into a detached worktree under RUNNER_TEMP as data — staged and scanned, never executed. Before the fetch the plan step requires the release object's tag_name to equal the run's tag and takes the expected commit from workflow_run.head_sha (a dispatch asks the API); the fetched tag must resolve to exactly that commit or the run refuses. The guard reads the allowlist and forbidden list from the tooling checkout, whatever the tag carries. - Prerelease overwrote the mirror's default branch. workflow_run always publishes, and the tree push never looked at PRERELEASE. The plan step now derives publish_tree=false for a prerelease and says why; the default branch and gh-pages pushes are gated on it; the release is still created, marked prerelease, pinned to the mirror's current default-branch head — an empty mirror is refused rather than given an RC as its first content. - Captured output hid refusals. `target` and `tree` ran through `$(...)`, so under set -e their ::error:: lines never reached the log. Both now run directly and write their results (repo=/name=, result=/sha=) through a new --output FILE option, which the workflow points at $GITHUB_OUTPUT. - The gitleaks download carries --tlsv1.2 like every other privileged fetch. scripts/tests/mirror-publish-workflow.bats executes the plan, src, target and keep step bodies read out of the workflow itself (gh shimmed, the tag fetch against a real bare repo) and pins the shape — no checkout ref, every tree push gated, the release step not, no captured publisher, one pinned tag fetch — with mutations asserted to change the document before they are judged. publish-mirror.bats covers --output for target and tree, including that a refusal writes nothing and annotates stdout. Each new check was mutation-proved against the real workflow and script. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(mirror): pin the fixture origin's HEAD so the tag-fetch case runs on a fresh runner The workflow test's bare origin relied on init.defaultBranch: unset on the runner, its HEAD pointed at a `master` nobody pushed, the clone had an unborn HEAD, and `rev-parse HEAD` handed the src step the literal word HEAD as the expected commit — the case failed with "cut at HEAD" instead of proving the pinned fetch. The bare HEAD is now set to main explicitly and the commit is read from the seed repository; reproduced locally with GIT_CONFIG_KEY_0=init.defaultBranch GIT_CONFIG_VALUE_0=master before and after. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(guard): strip [allow] tokens as whole words, name private needles by number only Two review findings on scripts/publish-guard.sh (kept byte-identical with the cli repository's copy): - The [allow] pass removed a token with an unanchored, case-sensitive sed replace before re-testing the needle. A mailbox that merely ENDS in the public support address (devsupport@…) lost the token, the mailbox rule no longer matched, and the internal address could ship; a differently-cased public address was refused. The strip now matches the token as a whole word — not the tail of a longer local part, not the head of a longer domain, a sentence-ending dot still a boundary — and case-insensitively, as the scan itself matches. - Refuse-tier hits printed the needle, and the workflow tees that log into the run summary. The needles from --extra-forbidden are the identifiers kept out of the committed list because this repository is public, so one deliverable hit would have published them in the log. Private needles are now scanned in their own pass and named `private needle #N` in every line the guard prints or writes (stdout, the report file, grep-error text); the committed needles are still named by pattern. publish-guard.bats: devsupport@ is refused, Support@Tracebloc.io. at a sentence end passes, and the tenant case asserts the redacted name and that the pattern appears neither in the output nor in publish-guard-report.txt. Each check was mutation-proved by restoring the old strip and the old label in the script and watching only its named cases redden. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(mirror): only the newest stable release replaces the mirror's default branch A re-run of the chart workflow for an earlier tag, or a dispatch naming one, used to push that tag's tree to the mirror's default branch and chart index, rolling the public README, installer and charts back — and, when that release was already mirrored, stopping there with the rollback in place. The plan step now asks GitHub for the source repository's releases/latest and sets publish_tree=false for any stable tag that is not it; the release is still mirrored, pinned to the mirror's current default-branch head. An unreadable releases/latest refuses the run rather than guessing. Tests (mirror-publish-workflow.bats, executing the real step bodies): the newest stable release publishes the tree and asks releases/latest of the source repo; an older stable tag sets publish_tree=false with a notice and is not marked prerelease; an unreadable releases/latest is refused; a mutation that removes the comparison lets the older tag publish the tree (so the test is live). The shape check now derives the step that decides publish_tree and refuses one that never reads releases/latest, mutation-proved. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci(mirror): refuse a non-boolean isPrerelease; keep the guard summary under errexit Two review findings on the workflow body. isPrerelease was compared only against an exact `true`, so a release object missing the field (or carrying anything else) read as stable and left the default-branch and gh-pages pushes armed. The plan step now accepts exactly `true` or `false` and refuses everything else. Actions runs every `run:` body with -e, which `set -uo pipefail` does not turn off, so a refusing guard exited the step at the tee pipeline before the step summary was written or `exit "$rc"` ran. Both guard steps now catch the guard's status with `|| rc=$?`, write the summary, and exit with that status. The steps gained ids (guard-tree, guard-pages) so their bodies are testable. Tests (mirror-publish-workflow.bats): step bodies now run under `bash -e`, as in CI; a release JSON without isPrerelease is refused; the guard-tree body with a planted refusing guard writes the summary and exits 1; the mutation that drops `|| rc=$?` reproduces the finding (exit 1, empty summary). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…ndle holds on the published digests; the size floor keeps no default, so nothing expands until an operator sets one (backend#3147) (#1045) Chart 1.9.115. per_rank_split is the engine default (tracebloc-engine#1010, #1014 on the :dev label), a decline falls back to one GPU (client-runtime#553, jobs-manager:dev = d1a58d6). TRACEBLOC_DDP=0 per edge is the rollback lever. MIGRATION 1.9.115 section; the no-edge-armed-by-upgrade case is replaced by three pins (default ON, no floor, explicit OFF wins). Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…D2) (#1016) * [settings-naming] rename resourceMonitor -> resourceMonitor.enabled (D2) RFC-0076 S14 (client#1009). `resourceMonitor: <bool>` becomes `resourceMonitor.enabled: <bool>` (D2: component booleans are `<component>.enabled`). This is a bool->object change, so every stored values.yaml and every `--set resourceMonitor=true` still arrives as a SCALAR. A template that read `.Values.resourceMonitor.enabled` blindly would `fail` ("can't evaluate field enabled in interface {}") on the scalar, and the old `ne .Values.resourceMonitor false` gate crashes ("incompatible types for comparison: map and bool") on the new object. So all reads now route through a single new helper, tracebloc.resourceMonitorEnabled, which resolves the effective flag from whichever shape is present (kindIs), preferring the new `.enabled` form and defaulting absent/`{}` to enabled to match the historical default. Effective behaviour is unchanged: resourceMonitor.enabled=true does exactly what resourceMonitor=true did. - values.yaml default is now the object form; legacy scalar still honoured through the alias window (remove_by: 2026-12-31). - values.schema.json accepts both a boolean and an object; the object CLOSES its keys (additionalProperties: false) so a mistyped `enabled` is refused at chart load instead of silently staying enabled. - migrated the eight readers (daemonset, rbac, scc, secrets, rbac.yaml, jobs-manager NODE_AGENTS_NAMESPACE, NOTES.txt, and the two _helpers predicates resourceMonitorRefreshPinned / nodeAgentsInUse). - Chart.yaml version+appVersion bumped 1.9.107 -> 1.9.108 (chart-version-guard). - helm-unittest: added legacy-scalar, new-object, unset, null, `{}`, both-disabled, and mistyped-key cases. Closes #1009 Part of tracebloc/backend#3391 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(chart): bump to 1.9.109 (one patch above develop, chart-version-guard) — client#1009 Develop advanced to 1.9.108 after this branch bumped there too; re-bump so the version stays above develop. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
release-train: develop -> staging
…tart block (#1039) The #563 flap guard does WARN + FLAP_KEY + exit 0 once refresh-attempt >= MAX_REFRESH_ATTEMPTS, before the end-of-tick digest annotate. The stale-pin clears were batched into that final annotate, so on a tick that is both off-digest (restart_needed=1) and latched they were dropped, leaving a false "pin is stale" finding to persist. Move the stale-pin writes (clears + finding-set) into their own accumulator, annotated in a bounded, non-fatal call above the restart block. They touch only stale-pin-<image> keys, never last-refreshed-<image>-digest, so writing them before the rollout can't affect the recorded==latest skip; the digest record stays below, after a successful rollout. Tests: image-refresh-latched-annotate.bats (new, extracts the shipped tail and asserts the clear lands while the digest record does not on a latched tick, and both land on a healthy tick); image-refresh-stale-pin.bats harness updated for the accumulator; image_refresh_test.yaml render guard covering all four stale-pin writes on both axes. Split from #1008 (item 2). Bumps chart to 1.9.117.
… sha256sum -c (backend#3676) (#1047) A mangled, empty, or truncated pin piped straight into `sha256sum -c` can pass verification: coreutils reports a malformed checksum line as "no properly formatted checksum lines found", and whether that is a non-zero exit is build-dependent (GNU exits 1; some builds exit 0). So a bad pin can let an UNVERIFIED gitleaks binary get extracted and run, defeating the point of pinning the scanner. Add the same fail-closed hex-digest guard the kubeconform install already uses in helm-ci.yaml, adjusted for this variable and placed right after the curl download and before the sha256sum line, inside the existing set -euo pipefail block. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
release-train: develop -> staging
Contributor
Author
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 209575a. Configure here.
tracebloc-release-train
Bot
added
gate-nudge
Contributor
Author
|
Promoted with 1 open Medium/Low Bugbot finding(s), per the severity policy in release-train's README (High stops the line; Medium/Low are recorded and ship, at both hops): Mirror omits documented list-images script This is a third look at twice-reviewed code -- it passed per-feature review and functional review on the source branch before the promotion. Fix forward on What the train did with each:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated promotion by the release train (RFC-0008 D14). Head is the train-managed
release-train/to-mainbranch (a mirror ofstaging), so it never collides with a human PR. Merged only when the fr-gate is green.Note
High Risk
Mirror publishing is security-critical (public exposure, secrets, gitleaks); chart defaults change training image registry and DDP, affecting live cluster behavior on upgrade.
Overview
Adds a mirror publish pipeline so only an allowlisted, scanned deliverable reaches a separate public repo: new workflow (triggered after Release Helm Chart),
publish-guard.sh/publish-mirror.sh, and.publish-include/.publish-forbiddenpolicy files (gitleaks, path/string guards, prerelease vs newest-stable tree rules). Bats cover the workflow shape and guard behavior.Client chart 1.9.117 (from 1.9.113): training pods use
JOB_IMAGE_HOSTviatracebloc.tbRegistry(default ghcr.io with control plane);env.TRACEBLOC_DDPdefaults to"1"(still needsMULTI_GPU_MIN_PARAMETERSto expand GPUs);resourceMonitorgainsresourceMonitor.enabledwith a legacy scalar alias viatracebloc.resourceMonitorEnabled. Image-refresh CronJob writes stale-pin annotations before the restart block so they are not dropped on a latched flap (#1008). Docs/tests/schema updated accordingly.Reviewed by Cursor Bugbot for commit 209575a. Bugbot is set up for automated code reviews on this repo. Configure here.