Skip to content

ci: add tests for upgrading azldev #17703

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dmcilvaney wants to merge 2 commits into
base: 4.0
Choose a base branch
from
+213 −79
Open

ci: add tests for upgrading azldev #17703

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
be108b3
ci: add smoketest on updated azldev pin
dmcilvaney Jun 10, 2026
d795767
ci: do full re-render when azldev version changes
dmcilvaney Jun 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 86 additions & 0 deletions .github/workflows/azldev-smoke.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Smoke-test the azldev version pinned in .azldev-version.
#
# When a PR bumps .azldev-version (or touches the runner image / this workflow),
# build the runner container with that exact pin and confirm the resulting
# binary can (a) run and (b) parse every component definition in the repo via
# `azldev component list`. This catches the two failure modes of a version bump:
# the pin doesn't `go install`, or the new version breaks on the repo's TOMLs.
name: "azldev Smoke Test"

on:
pull_request:
branches: ["4.0"]
paths:
- ".azldev-version"
- ".github/workflows/containers/azldev-runner.Dockerfile"
- ".github/workflows/azldev-smoke.yml"
workflow_dispatch:

# Cancel in-progress runs of this workflow if a new run is triggered.
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true

permissions: {}

jobs:
smoke:
name: "comp list"
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false

- name: Validate .azldev-version format
run: |
set -euo pipefail
version="$(tr -d '\n' < .azldev-version)"
if [ -z "$version" ]; then
echo "::error::.azldev-version is empty"
exit 1
fi
# Restrict to the charset Go module versions use (commit SHAs, tags,
# pseudo-versions). This blocks shell metacharacters so the value is
# safe to pass straight through to `docker build --build-arg` below.
if ! printf '%s' "$version" | grep -Eq '^[0-9A-Za-z._+-]+$'; then
echo "::error::.azldev-version contains unexpected characters: '$version'"
exit 1
fi
echo "azldev version pin: $version"

- name: Build azldev runner container
run: |
set -euo pipefail
docker build \
--build-arg UID="$(id -u)" \
--build-arg AZLDEV_VERSION="$(cat .azldev-version)" \
-t localhost/azldev-runner \
-f .github/workflows/containers/azldev-runner.Dockerfile \
.github/workflows/containers/

# `component list` only parses TOML, so no mock sandbox flags are needed
# here (contrast with the render/build checks). Mount the checkout rw to
# match the documented /workdir convention and avoid surprises if azldev
# writes a cache.
- name: Smoke-test azldev
run: |
set -euo pipefail
docker run --rm \
-v "$GITHUB_WORKSPACE:/workdir" \
localhost/azldev-runner \
bash -eu -o pipefail -c '
echo "=== azldev version ==="
azldev --version
echo "=== azldev component list ==="
count=$(azldev component list -a -q -O json | jq length)
echo "azldev resolved ${count} component(s)"
if [ "${count}" -le 0 ]; then
echo "::error::azldev component list returned no components"
exit 1
fi
'
206 changes: 127 additions & 79 deletions .github/workflows/check-rendered-specs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,13 @@
# stale lock would steer it toward the wrong scope. When locks are dirty,
# render is skipped and only the locks comment is posted.
#
# Security: the PR checkout is data-only. We never execute code from the PR —
# azldev is installed from upstream, scripts come from the base branch checkout.
# Security: the PR checkout is data-only. We never run PR scripts, and the
# runner Dockerfile + build context come from the trusted base checkout. The
# azldev binary is built from the version pinned in the PR's .azldev-version
# (so a version-bump PR is validated with the binary that will actually run
# post-merge), but the Dockerfile hardcodes the module path to the
# Microsoft-owned github.com/microsoft/azure-linux-dev-tools repo -- the PR
# can only select a ref within that repo's network, not redirect the install.
name: "Check Rendered Specs"

on:
Expand Down Expand Up @@ -83,9 +88,23 @@ jobs:

- name: Build azldev runner container
run: |
set -euo pipefail
# Build with the azldev version the PR proposes (pr-head/.azldev-version),
# NOT the base branch's. For a version-bump PR these differ, and the whole
# point of the drift check is to run with the binary that will actually run
# on 4.0 post-merge -- the base binary would pass the check and then drift
# the moment the bump lands. Only the version string comes from the PR; the
# Dockerfile + build context are the trusted base checkout, and the
# Dockerfile hardcodes the module path to the Microsoft-owned repo.
AZLDEV_VERSION="$(tr -d '\n' < pr-head/.azldev-version)"
# Hygiene: reject a malformed/garbage version before it reaches docker build.
if ! printf '%s' "$AZLDEV_VERSION" | grep -Eq '^[0-9A-Za-z._+-]+$'; then
echo "::error::pr-head/.azldev-version is empty or has unexpected characters"
exit 1
fi
docker build \
--build-arg UID=$(id -u) \
--build-arg AZLDEV_VERSION="$(cat .azldev-version)" \
--build-arg UID="$(id -u)" \
--build-arg AZLDEV_VERSION="$AZLDEV_VERSION" \
-t localhost/azldev-runner \
-f .github/workflows/containers/azldev-runner.Dockerfile \
.github/workflows/containers/
Expand Down Expand Up @@ -297,9 +316,23 @@ jobs:

- name: Build azldev runner container
run: |
set -euo pipefail
# Build with the azldev version the PR proposes (pr-head/.azldev-version),
# NOT the base branch's. For a version-bump PR these differ, and the whole
# point of the drift check is to run with the binary that will actually run
# on 4.0 post-merge -- the base binary would pass the check and then drift
# the moment the bump lands. Only the version string comes from the PR; the
# Dockerfile + build context are the trusted base checkout, and the
# Dockerfile hardcodes the module path to the Microsoft-owned repo.
AZLDEV_VERSION="$(tr -d '\n' < pr-head/.azldev-version)"
# Hygiene: reject a malformed/garbage version before it reaches docker build.
if ! printf '%s' "$AZLDEV_VERSION" | grep -Eq '^[0-9A-Za-z._+-]+$'; then
echo "::error::pr-head/.azldev-version is empty or has unexpected characters"
exit 1
fi
docker build \
--build-arg UID=$(id -u) \
--build-arg AZLDEV_VERSION="$(cat .azldev-version)" \
--build-arg UID="$(id -u)" \
--build-arg AZLDEV_VERSION="$AZLDEV_VERSION" \
-t localhost/azldev-runner \
-f .github/workflows/containers/azldev-runner.Dockerfile \
.github/workflows/containers/
Expand Down Expand Up @@ -348,86 +381,101 @@ jobs:
-e BASE_REPO \
localhost/azldev-runner \
bash -eu -o pipefail -c '
# Stale forks may not have the upstream base SHA in their
# object store. Fetch it explicitly so `compute_changed.sh`
# can resolve `--from <base-sha>`. The base repo is public
# so no auth is needed.
SPECS_DIR=$(azldev config dump -q -f json | jq -r .project.renderedSpecsDir)

# Stale forks may not have the upstream base SHA in their object
# store. Fetch it explicitly so the version-bump diff below and
# `compute_change_set.sh` can both resolve the base commit. The
# base repo is public so no auth is needed.
git -C /workdir remote add base "https://github.com/$BASE_REPO.git" 2>/dev/null || true
git -C /workdir fetch --no-tags base "$PR_BASE_SHA"

# Compute the render set (PR-touched components). Safe here
# because the locks gate has already run successfully -- input
# fingerprints used by `azldev component changed` reflect the
# current head.
/scripts-components/compute_change_set.sh \
--output-dir /output/change-set \
--source-commit "$PR_HEAD_SHA" \
--target-commit "$PR_BASE_SHA"

SPECS_DIR=$(azldev config dump -q -f json | jq -r .project.renderedSpecsDir)

# Components removed in this PR no longer have a comp.toml, so
# `azldev component render` will not touch (or clean) their old
# specs directory. Delete `<specs>/<first-char>/<name>/` for
# each deleted component in the working tree so the missing
# files surface as drift in the existing
# `check_rendered_specs.py` report (and end up in the fix
# patch as deletions). This replaces the orphan-cleanup that
# `azldev component render -a --clean-stale` used to do for
# us; with a scoped render the deleted components are not in
# the render set, so nothing else cleans them.
#
# Scope caveat: this only catches components deleted IN THIS
# PR. Pre-existing orphans (stale spec dirs whose component
# was removed in an earlier PR without a matching render)
# are not caught here.
jq -r ".[] | select(.changeType == \"deleted\") | .component" \
/output/change-set/changed-components.json \
| while IFS= read -r name; do
[ -n "$name" ] || continue
# Defense-in-depth: refuse component names with path
# metacharacters before constructing the rm -rf target.
# `azldev` does not currently validate names against
# this set, and `.comp.toml` quoted keys allow `/`,
# `..`, etc.
if [[ ! "$name" =~ ^[A-Za-z0-9][A-Za-z0-9._+-]*$ ]]; then
echo "::warning::Refusing suspicious component name from changed-components.json: $name"
continue
fi
# Spec bucket dirs are lowercase (specs/a/, specs/r/, ...)
# while component names can be uppercase (e.g. `R`, `AMF`,
# `CGAL`). Lowercase the first char so the path matches.
first_char="${name:0:1}"
first_char="${first_char,,}"
orphan_dir="$SPECS_DIR/$first_char/$name"
if [ -d "$orphan_dir" ]; then
echo "Cleaning orphan specs dir for deleted component: $orphan_dir"
rm -rf "$orphan_dir"
fi
done

if [ ! -s /output/change-set/render-set.txt ]; then
echo "No PR-scoped components -- skipping render."
# Synthesize an empty render-output.json so the artifact
# upload (if: always()) and downstream comment job have a
# well-formed payload.
echo "[]" > /output/render-output.json
else
# `-x` fails loudly if the arg list would split into multiple
# xargs invocations -- each batch after the first would
# overwrite the previous JSON output to render-output.json.
# Current component counts are well below ARG_MAX, but
# `-n 50000` gives `-x` teeth (without `-n`/`-L`, GNU xargs
# never splits). `--` ends azldev option parsing so component
# names beginning with `-` (none today) are unambiguous.
xargs -x -n 50000 -d "\n" azldev component render -q -O json -- \
< /output/change-set/render-set.txt \
# A change to .azldev-version swaps the azldev binary, which can
# alter rendered output for ANY component -- not just the ones
# this PR edits -- so a PR-scoped render would miss that drift.
# Detect it with a plain git diff (both commits are local after
# the fetch above) and render everything when it moved. Using git
# here -- rather than the REST "list PR files" endpoint -- avoids
# the 3000-file cap and 30-results-per-page pagination of that
# endpoint, so detection stays correct for arbitrarily large PRs.
if git -C /workdir diff --name-only "$PR_BASE_SHA" "$PR_HEAD_SHA" -- .azldev-version | grep -q .; then
# `--clean-stale` prunes orphaned spec dirs globally, which
# subsumes the targeted deleted-component cleanup the scoped
# branch does below.
echo ".azldev-version changed -- rendering ALL components."
azldev component render -q -a --clean-stale -O json \
> /output/render-output.json
else
# Compute the render set (PR-touched components). Safe here
# because the locks gate has already run successfully -- input
# fingerprints used by `azldev component changed` reflect the
# current head.
/scripts-components/compute_change_set.sh \
--output-dir /output/change-set \
--source-commit "$PR_HEAD_SHA" \
--target-commit "$PR_BASE_SHA"

# Components removed in this PR no longer have a comp.toml, so
# `azldev component render` will not touch (or clean) their old
# specs directory. Delete `<specs>/<first-char>/<name>/` for
# each deleted component in the working tree so the missing
# files surface as drift in the existing
# `check_rendered_specs.py` report (and end up in the fix
# patch as deletions). The render-all branch above does not
# need this: `render -a --clean-stale` already prunes orphans.
#
# Scope caveat: this only catches components deleted IN THIS
# PR. Pre-existing orphans (stale spec dirs whose component
# was removed in an earlier PR without a matching render)
# are not caught here.
jq -r ".[] | select(.changeType == \"deleted\") | .component" \
/output/change-set/changed-components.json \
| while IFS= read -r name; do
[ -n "$name" ] || continue
# Defense-in-depth: refuse component names with path
# metacharacters before constructing the rm -rf target.
# `azldev` does not currently validate names against
# this set, and `.comp.toml` quoted keys allow `/`,
# `..`, etc.
if [[ ! "$name" =~ ^[A-Za-z0-9][A-Za-z0-9._+-]*$ ]]; then
echo "::warning::Refusing suspicious component name from changed-components.json: $name"
continue
fi
# Spec bucket dirs are lowercase (specs/a/, specs/r/, ...)
# while component names can be uppercase (e.g. `R`, `AMF`,
# `CGAL`). Lowercase the first char so the path matches.
first_char="${name:0:1}"
first_char="${first_char,,}"
orphan_dir="$SPECS_DIR/$first_char/$name"
if [ -d "$orphan_dir" ]; then
echo "Cleaning orphan specs dir for deleted component: $orphan_dir"
rm -rf "$orphan_dir"
fi
done

if [ ! -s /output/change-set/render-set.txt ]; then
echo "No PR-scoped components -- skipping render."
# Synthesize an empty render-output.json so the artifact
# upload (if: always()) and downstream comment job have a
# well-formed payload.
echo "[]" > /output/render-output.json
else
# `-x` fails loudly if the arg list would split into multiple
# xargs invocations -- each batch after the first would
# overwrite the previous JSON output to render-output.json.
# Current component counts are well below ARG_MAX, but
# `-n 50000` gives `-x` teeth (without `-n`/`-L`, GNU xargs
# never splits). `--` ends azldev option parsing so component
# names beginning with `-` (none today) are unambiguous.
xargs -x -n 50000 -d "\n" azldev component render -q -O json -- \
< /output/change-set/render-set.txt \
> /output/render-output.json
fi
fi

# check_rendered_specs.py diffs the working tree (now containing
# any scoped re-render output and any orphan-cleanup deletions)
# against HEAD and produces the user-facing patch.
# any re-render output and any orphan-cleanup deletions) against
# HEAD and produces the user-facing patch.
python3 /scripts-render/check_rendered_specs.py \
--specs-dir "$SPECS_DIR" \
--report /output/render-check-report.json \
Expand Down
Loading