PR-env deploys: stop phantom image-tag pins#33
Merged
lucas-koontz merged 2 commits intoJul 20, 2026
Merged
Conversation
…nce guard + Deploys lint)
Improve the logic for selecting image tags by adding an existence check for ECR tags when the ECR_CHECKS flag is enabled. This change aims to prevent phantom image tags that were previously pinned and aligns with pre-hardening behavior under infrastructure errors. Addresses ENG-877.
mindsdb-devops
approved these changes
Jul 20, 2026
lucas-koontz
deleted the
lucas/eng-877-pr-env-deploys-stop-phantom-image-tag-pins-head-sha-tags-ecr
branch
July 20, 2026 18:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stops PR-env deploys from pinning image tags that were never built, and makes the
two silent failure modes from the M3 cutover week loud. Fixes the class of
incident where
pr-auth-168/pr-mindshub-frontend-1368sat inImagePullBackOfffor 25h on a phantom tag.Why
Images are tagged
development-<merge SHA>, but GitHub regenerates a PR'ssynthetic merge commit (new hash, identical content) whenever base or head
moves — we observed two twins born one second apart.
argocd-pr-env-deployresolves linked PRs'
merge_commit_shavia the API at deploy time, so across-PR pin can name a twin no build pushed. Separately,
Deploys:lines inURL form were silently ignored, quietly running envs on defaults.
Changes
build-push-ecr — on pull_request events, additionally tags
<env>-head-<pr head sha>. Head SHAs are immutable, so cross-PR resolutionbecomes race-free. Existing tags, outputs, and non-PR builds are unchanged.
argocd-pr-env-deploy
development-head-<head sha>, fall back to themerge-SHA tag for images built before dual-tagging, and fail loudly
(naming both candidate tags and the re-run fix) when neither exists —
replacing the silent phantom pin.
aws ecr describe-images. Checks fail-open with a single warning on infraerrors (missing CLI, auth, throttle) so a degraded runner can't block a
deploy that would have worked; ImageNotFound is authoritative.
Deploys:-looking lines that don't parse (Deploys: <PR URL>, lowercase)now emit
::warning::naming the accepted forms.mute
set -edeath.Notes
linked repos rebuild once with the new build action (fallback covers the gap).
development-head-*specifically; PR builds forother
build-for-environmentvalues get a head tag deploy doesn't use(degrades to today's merge-tag behavior, now with a loud failure when stale).
hard-fail.
Verification
Both scripts extracted and executed in a containerized harness (bash 5.2,
stubbed
gh_api/aws/argocd): 36/36 assertions across 8 scenarios — head-taghit, merge fallback, neither-exists hard-fail (proving the
exit 1propagatesfrom the main shell, argocd never invoked),
#stagingpassthrough, fail-openwarns exactly once with one AWS call, self-link skip, own-tag-missing message,
URL-form lint. Buildx argv checked: PR events gain exactly two args
(
-t, the head tag); push events gain zero (no empty-string arg). shellcheck:zero findings on the deploy script; only pre-existing style findings on the
build script. YAML parse +
bash -ngreen.Fixes ENG-877