ci: enable best-effort delta builds using ADO pipelines#17684
Open
dmcilvaney wants to merge 1 commit into
Open
ci: enable best-effort delta builds using ADO pipelines#17684dmcilvaney wants to merge 1 commit into
dmcilvaney wants to merge 1 commit into
Conversation
Contributor
Author
|
Note: Planning on renaming the file to something more accurate, but will need to combine that with ADO changes, leaving as-is to allow testing in the meantime. |
PawelWMS
reviewed
Jun 10, 2026
dmcilvaney
force-pushed
the
damcilva/4.0/pipelines/enable_v1_delta_build
branch
2 times, most recently
from
ed24ecf to
11afe71
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Enables best-effort post-merge “delta” package builds in Azure DevOps by introducing a shared step template and a new helper that determines (target, source) commit ranges via the Azure DevOps control-plane API (rather than git heuristics), while also removing the package-build submission from the legacy source-upload pipeline and avoiding env-var name shadowing.
Changes:
- Add
determine_commit_range.py(Azure DevOps SDK-based) plus docs/deps to compute a stable, non-overlapping commit range for delta builds. - Introduce a shared ADO step sub-template (
common-steps.yml) and refactor stages templates to consume it. - Add a new
package-buildADO wrapper + stages template for post-merge delta builds; adjust lock verification expectations accordingly.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
scripts/ci/control-tower/verify_locks.sh |
Stops self-unshallowing; relies on the pipeline’s shared “Ensure full git history” step. |
scripts/ci/ado/requirements.txt |
Pins Azure DevOps Python SDK dependency for ADO control-plane queries. |
scripts/ci/ado/README.md |
Documents conventions and caller contract for ADO helper scripts. |
scripts/ci/ado/determine_commit_range.py |
New helper to compute commit range using ADO build history (best-effort fallback included). |
ruff.toml |
Adds per-file ignore to allow namespace-style execution of scripts/ci/** helpers without __init__.py. |
.github/workflows/ado/templates/steps/common-steps.yml |
New shared step template: unshallow, install deps, determine range, verify locks, compute change set, verify rendered specs. |
.github/workflows/ado/templates/sources-upload-stages.yml |
Refactors to use common-steps.yml, removes package-build submission, and fixes reserved env var shadowing. |
.github/workflows/ado/templates/package-build-stages.yml |
New raw stages template for post-merge delta package builds via Control Tower. |
.github/workflows/ado/sources-upload.yml |
Updates wrapper metadata and parameters (e.g., artifact base name). |
.github/workflows/ado/package-build.yml |
New wrapper pipeline wiring for the delta package-build flow. |
.github/instructions/ado-pipeline.instructions.md |
Extends applyTo glob and documents shared step sub-templates + ADO control-plane auth via System.AccessToken. |
Automatic source upload is being reworked, but we don't need to wait on that to enable the initial version of delta builds. Disable the source upload component in the delta build pipeline, and configure it to calculate the commit range using the ADO REST API instead of git commands (since this will be post-merge). Also fix a bug where the environment variable names were shadowed by ADO predefined variables.
dmcilvaney
force-pushed
the
damcilva/4.0/pipelines/enable_v1_delta_build
branch
from
11afe71 to
9ca3207
Compare
Comment on lines
+197
to
+220
| # The job is fail-loud, so reaching this step means "Prepare change | ||
| # set" succeeded and renderSetFile points at a real (possibly empty) | ||
| # file. An empty render set just means no PR-scoped components | ||
| # changed -- nothing to verify. | ||
| if [[ ! -s "$(renderSetFile)" ]]; then | ||
| echo "No changed components -- skipping render." | ||
| exit 0 | ||
| fi | ||
| echo "##[group]Render set" | ||
| sed 's/^/ - /' "$(renderSetFile)" | ||
| echo "##[endgroup]" | ||
| echo "##[group]Specs rendering + verification" | ||
| # `-x` fails loudly if the arg list would split into multiple | ||
| # xargs invocations -- a multi-batch render-check would | ||
| # silently mask drift in the later batches. `-n 50000` is | ||
| # required for `-x` to have any effect (without `-n`/`-L`, | ||
| # GNU xargs never splits). `--` ends azldev option parsing so | ||
| # component names beginning with `-` (none today) are | ||
| # unambiguous. `env AZLDEV_ALLOW_ROOT=1` is inline per | ||
| # .github/instructions/ado-pipeline.instructions.md. | ||
| xargs -x -n 50000 -d '\n' env AZLDEV_ALLOW_ROOT=1 azldev component render --check-only -- \ | ||
| < "$(renderSetFile)" | ||
| echo "##[endgroup]" | ||
| displayName: "Verify rendered specs are up to date" |
Comment on lines
+169
to
+187
| echo "##[group]Preparing change set" | ||
| scripts/ci/components/compute_change_set.sh \ | ||
| --output-dir "$change_set_dir" \ | ||
| --source-commit "$(sourceCommit)" \ | ||
| --target-commit "$(targetCommit)" | ||
| echo "##[endgroup]" | ||
|
|
||
| echo "##[group]Upload set (sourcesChange == true, changeType in {added, changed})" | ||
| upload_count=$(jq -r '[.[] | select(.sourcesChange == true and (.changeType | IN("added", "changed")))] | length' "$json_file") | ||
| jq -r '.[] | select(.sourcesChange == true and (.changeType | IN("added", "changed"))) | .component' "$json_file" | sort | ||
| echo "Total: $upload_count component(s) to upload." | ||
| echo "##[endgroup]" | ||
|
|
||
| # Publish the downstream pipeline variables now that the change set is | ||
| # ready (consumed by the render-verify step and the trailing Control | ||
| # Tower call). | ||
| echo "##vso[task.setvariable variable=changedComponentsFile;isreadonly=true]$json_file" | ||
| echo "##vso[task.setvariable variable=renderSetFile;isreadonly=true]$render_set_file" | ||
| displayName: "Prepare change set" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automatic source upload is being reworked, but we don't need to wait on that to enable the initial version of delta builds. Disable the source upload component in the delta build pipeline, and configure it to calculate the commit range using the ADO REST API instead of git commands (since this will be post-merge).
To make future pipeline work easier, split the common components into their own template.
Also fix a bug where the environment variable names were shadowed by ADO predefined variables.