Skip to content

release-train: develop -> staging - #1037

Merged
tracebloc-release-train[bot] merged 2 commits into
stagingfrom
release-train/to-staging
Sep 10, 2026
Merged

release-train: develop -> staging#1037
tracebloc-release-train[bot] merged 2 commits into
stagingfrom
release-train/to-staging

Conversation

@LukasWodka

@LukasWodka LukasWodka commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Automated promotion by the release train (RFC-0008 D14). Head is the train-managed release-train/to-staging branch (a mirror of develop), so it never collides with a human PR. Merged only when the fr-gate is green.


Note

Medium Risk
Changes when the image-refresh CronJob triggers control-plane rollouts (jobs-manager Recreate, resource-monitor DaemonSet flap path); logic is heavily gated and tested, but upgrades still force a one-time repair roll on unmarked legacy edges.

Overview
Bumps the client Helm chart to 1.9.113 and changes the image-refresh CronJob so fresh installs no longer pay an automatic digest re-pin rollout when the registry digest is unchanged but workloads still run the floating :tag.

The script now stamps tracebloc.io/first-observed-<image> on first observation and tracebloc.io/digest-applied-<image> after a successful re-image. When recorded == latest and the live workload is off-digest, it skips rolling only if both markers show a fresh install (first_observed set, applied absent). Established edges (digest was applied before) still re-pin after helm re-renders revert pins; legacy edges with neither marker still take a one-time repair roll on upgrade so the fleet is not left on a possibly stale tag. Digest annotation reads are fail-closed (no || true) so API errors cannot mis-stamp first_observed.

Helm unittest and bats coverage lock the marker keys, skip condition, and the three behaviors (fresh / established / legacy).

Reviewed by Cursor Bugbot for commit 186c494. Bugbot is set up for automated code reviews on this repo. Configure here.

…plied marker (#1033)

On recorded==latest with the workload on :tag, the image-refresh loop could not tell a fresh install from a helm-revert of an applied pin, so every fresh install paid the shared #563 flap path (resource-monitor DaemonSet rollout under tolerations: Exists) plus a Recreate downtime on jobs-manager for byte-identical content.

Gate the roll on two durable per-image markers: first-observed-<image> (stamped by first-observation) and digest-applied-<image> (stamped only after a successful re-image). Skip the roll only on positive fresh-install evidence (first_observed present AND applied absent); an established edge (applied present) rolls the client-runtime#199 repair, and a pre-marker/legacy edge (neither marker) also rolls so the existing fleet is not stranded on :tag on the upgrade hop. Both marker reads and the digest-record read are fail-closed.

Split from #1008 (item 2). Bumps chart to 1.9.112.
@LukasWodka

Copy link
Copy Markdown
Contributor Author

bugbot run

@saqlainsyed007
saqlainsyed007 removed the request for review from saadqbal September 10, 2026 18:03

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

… same 1.9.112 that #1032 already shipped (#1040)

The image-refresh CronJob fix (#1033) bumped Chart.yaml 1.9.111 -> 1.9.112 on its own
branch, and so did the control-plane trim (#1032). Both merged; the second bump was a
no-op, so develop's chart version now equals staging's while the templates differ, and
the chart-version guard rightly refuses the staging promotion. This bump carries #1033
to 1.9.113. No template change.
@LukasWodka

Copy link
Copy Markdown
Contributor Author

bugbot run

@tracebloc-release-train tracebloc-release-train Bot added the gate-nudge Toggled by the release train to (re-)fire the fr-gate label Sep 10, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 186c494. Configure here.

@tracebloc-release-train tracebloc-release-train Bot removed the gate-nudge Toggled by the release train to (re-)fire the fr-gate label Sep 10, 2026
@tracebloc-release-train
tracebloc-release-train Bot merged commit e2ad3a7 into staging Sep 10, 2026
47 checks passed
@tracebloc-release-train
tracebloc-release-train Bot deleted the release-train/to-staging branch September 10, 2026 19:17
@LukasWodka

Copy link
Copy Markdown
Contributor Author

Functional review — staging batch of 2026-09-10/11 (hops 18:00, 19:07, 20:40 UTC + overnight mirrors) — PASS (FR reviewer: LukasWodka; self-signoff per RFC-BACKEND-1405 D6)

Decision: PASS. All 39 cards at FR on staging — e2e-test-agent 15, backend 9, tracebloc-engine 6, client 5, docs 2, client-runtime 2 — advance to Ready for prod. Evidence pulled 05:37–06:11 UTC against the current staging tips (backend 458de261, client 6acc2f9, client-runtime eb412941, docs ea8a47f5, e2e-test-agent aebcf75b, tracebloc-engine 4b503980).

Journey on the staging chart (governs). Two tier-a baselines with chart_ref=v1.9.113-rc.2 — the rc staging carries, which includes the ghcr.io control-plane registry default (#1038):

Customer-path control. Fleet on the published chart 1.9.112, 16 tasks, https://github.com/tracebloc/e2e-test-agent/actions/runs/34566675826: 13 green through Train; seq2seq red at Train with the training pod exiting 137 at the 3Gi envelope — the known model-zoo template case (also red on the daily develop run), tracked upstream, not this batch; the tabular and image legs were superseded by the queued baselines above (same concurrency group), which cover them.

Channels. fr-assist run 34532009992 (21:24 UTC): all OK, 1 SKIPPED (credentials).

Per-repo staging deploys. Green on every tip: backend Tests / docker-build / Migrations check / Lint; client CodeQL / E2E mysql / Release Helm Chart (rc.2 published); client-runtime Publish images / Tests; tracebloc-engine Run Tests / Promote task images by digest; e2e-test-agent Unit tests / CLI surface; docs CodeQL. Staging API answers (401 unauthenticated on /). Frontend smoke on the staging branch https://github.com/tracebloc/frontend-app/actions/runs/34566891772 green.

Per-ticket read. All 39 PR bodies scanned for holds, interactive/TTY behaviour and migrations: none of the batch changes guided prompts; no body declares a hold on shipping. The one Medium Bugbot finding that shipped with the e2e mirror is recorded by the train on its own ticket.

What this evidence cannot tell you (README → Functional review): per-ticket behaviour beyond the journey's path, anything after a failing step, and steps a run skipped. The docs pages (docs#174) have no staging preview and were reviewed as content only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants