diff --git a/.github/instructions/ado-pipeline.instructions.md b/.github/instructions/ado-pipeline.instructions.md index 2c07de5c78f..5718e28098f 100644 --- a/.github/instructions/ado-pipeline.instructions.md +++ b/.github/instructions/ado-pipeline.instructions.md @@ -1,5 +1,5 @@ --- -applyTo: ".github/workflows/ado/*.yml,.github/workflows/ado/templates/*.yml,scripts/ci/**" +applyTo: ".github/workflows/ado/*.yml,.github/workflows/ado/templates/**/*.yml,scripts/ci/**" description: "Authoring and maintenance rules for Azure DevOps YAML pipelines under .github/workflows/ado/ (wrappers + raw stage templates under templates/) and their helper scripts under scripts/ci/ that run as GitHub PR checks or in the merge queue. Apply when creating or modifying any pipeline in that folder, or any script invoked by one — covers the wrapper/raw split, required OneBranch templates (Official vs NonOfficial), Workload Identity Federation service connections, Control Tower audience URIs, internal-only dependency sources (Go / Python / container images), Python-over-shell scripting, and security hardening." --- @@ -39,6 +39,8 @@ File-pairing convention: a wrapper at `.github/workflows/ado/.yml` pairs w See [.github/workflows/ado/sources-upload.yml](.github/workflows/ado/sources-upload.yml) and [.github/workflows/ado/templates/sources-upload-stages.yml](.github/workflows/ado/templates/sources-upload-stages.yml) for the canonical example. +Shared **step sub-templates** live under `.github/workflows/ado/templates/steps/.yml` and are spliced into a job's `steps:` via `- template: steps/.yml` (path relative to the including stages template). Use these to share step sequences across stages templates that differ only in a trailing pipeline-specific step. Splicing as **steps** (not a separate job/stage) keeps job-scoped pipeline variables and on-disk files flowing to the steps that follow — a separate job would force output variables + artifact upload/download. The including job must define any job-scope variables the shared steps reference (e.g. `ob_outputDirectory`). See [.github/workflows/ado/templates/steps/common-steps.yml](.github/workflows/ado/templates/steps/common-steps.yml), shared by the `package-build` and `sources-upload` pipelines. + ## OneBranch templates (MANDATORY — wrapper only) The rules in this section apply to the **wrapper** file. The raw stages template MUST NOT reference OneBranch at all. @@ -180,6 +182,8 @@ Avoid shell scripts beyond the smallest possible wiring (env exports, `##vso[... Python scripts are easier to test locally, easier to review, and avoid the foot-guns of bash quoting / globbing. +**Reference pipeline variables in bash via the `env:` block** (as in the example above: `env: FOO: $(foo)`, then `"$FOO"` in the script) — never inline `$(foo)` in a script body. An inline macro that resolves empty is left as the literal string `$(foo)`, which bash then evaluates as command substitution; an `env:` value is inert text, so an unset variable fails loud instead of silently running a command. + ### `azldev` in OneBranch OneBranch runs all steps as `root`. `azldev` refuses to run many commands as root by default (a safety measure for developer workstations). To allow it in CI, set `AZLDEV_ALLOW_ROOT=1` -- but **set it inline as a prefix on each `azldev` invocation**, not at the step level. Inline scoping makes it obvious which calls actually need the override and avoids leaking the flag to unrelated commands in the same step. @@ -192,6 +196,25 @@ Symptom that you need this: `ERR Error: this command may not be run as root`. This is NOT safe for general use, only for disposable CI environments. +### Reading ADO build metadata (control plane) + +To read the pipeline's own control-plane data (build history, definitions) from a helper script, use the official [`azure-devops`](https://github.com/microsoft/azure-devops-python-api) SDK rather than a hand-rolled REST client. Pin it in the area's `requirements.txt` and install it through the standard `PipAuthenticate@1` + `pip install` step. + +Authenticate with the pipeline's `System.AccessToken`, which the ADO REST API accepts as a PAT-equivalent credential via the SDK's documented `BasicAuthentication` pattern: + +```python +from azure.devops.connection import Connection +from msrest.authentication import BasicAuthentication + +connection = Connection(base_url=collection_uri, creds=BasicAuthentication("", access_token)) +build_client = connection.clients.get_build_client() +``` + +- This is the ADO **control plane** (the pipeline's own builds/repos), **not** Azure Resource Manager or Control Tower — so the Workload Identity Federation service-connection rule does **not** apply. The build identity is the correct least-privilege caller. +- Map the token in via the step `env:` block (`SYSTEM_ACCESSTOKEN: $(System.AccessToken)`), never on the command line. In a **YAML** pipeline there is no "Allow scripts to access the OAuth token" toggle (that is Classic-only) — the `env:` mapping is all that is needed. +- Reading builds of the pipeline's **own** definition in the same project is covered by the default **project** job-authorization scope and the `{Project} Build Service ({Org})` identity's default build-read permission. Only revisit if your org has tightened the defaults. +- Because the SDK is a pip dependency (not stdlib), any step that imports it MUST run **after** the dependency-install step (which itself follows `PipAuthenticate@1`). + ## Security hardening Apply all of these unless there is a documented reason not to: diff --git a/.github/workflows/ado/package-build.yml b/.github/workflows/ado/package-build.yml new file mode 100644 index 00000000000..6eae5748ac1 --- /dev/null +++ b/.github/workflows/ado/package-build.yml @@ -0,0 +1,87 @@ +# Microsoft Corporation +# +# Wrapper pipeline — passed to ADO as the entry point for the package-build +# pipeline. This file owns all OneBranch-specific wiring (governed templates +# repo, Official vs NonOfficial variant, featureFlags) and delegates the actual +# stages/jobs/steps to the raw stages template at: +# .github/workflows/ado/templates/package-build-stages.yml +# +# Authenticates via Workload Identity Federation (OIDC) and calls the Control +# Tower APIs to run the v1 post-merge delta build (stopgap until source upload +# is reworked): +# 1. Resolve the (target, source) commit range for this push from the +# previous CI build (ADO Builds API). +# 2. Submit official package builds for the components that changed across +# that range, via Workload Identity Federation (OIDC) to Control Tower. +# +# The setup + change-detection + validation steps are shared with the +# source-upload pipeline via templates/steps/common-steps.yml. +# +# Helper scripts live under: +# - scripts/ci/control-tower/ - (Control Tower-specific) +# - scripts/ci/ado/ - (ADO API: commit-range detection) +# - scripts/ci/components/ - (cross-pipeline azldev helpers shared with the GH Actions PR gates). +# +# Prerequisites (ADO / Azure Portal): +# 1. Entra ID App Registration with audience URI +# "api://" (see variable group below). +# 2. Federated identity credential on the app registration for the ADO +# service connection (issuer: https://vstoken.dev.azure.com/, +# subject: sc:////). +# 3. ARM service connection in ADO project settings using Workload Identity +# Federation (manual). +# 4. CI trigger configured (in ADO pipeline settings) to fire on pushes to +# the target branch. +# +# Variable Group (ADO Pipelines > Library): +# Name: "ControlTower-PRCheck" +# Required variables: +# - ApiAudience : Entra ID audience URI for the Control Tower app +# - ApiBaseDirectUrl : Direct base URL of the Control Tower APIM endpoint (bypasses Azure Front Door) + +# Trigger controlled by ADO branch policy — not YAML triggers. +trigger: none + +pr: none + +resources: + repositories: + - repository: templates + type: git + name: OneBranch.Pipelines/GovernedTemplates + ref: refs/heads/main + +extends: + template: v2/OneBranch.Official.CrossPlat.yml@templates + parameters: + featureFlags: + golang: + internalModuleProxy: + enabled: true + LinuxHostVersion: + Network: R1 + runOnHost: true + EnableCDPxPAT: false + + # https://aka.ms/obpipelines/sdl + globalSdl: + disableLegacyManifest: true + sbom: + enabled: false + tsa: + enabled: false + + stages: + - template: /.github/workflows/ado/templates/package-build-stages.yml@self + parameters: + outputDirectory: $(Build.ArtifactStagingDirectory)/output + artifactBaseName: packagebuild + containerImage: mcr.microsoft.com/onebranch/azurelinux/build:3.0 + poolType: linux + serviceConnection: CT-Endpoints-Access-ServiceConnection-DEV + variableGroup: ControlTower-PRCheck + # Control Tower package target for the 4.0 branch. + packageTarget: azl4 + # Must exceed the script's --poll-timeout-seconds (default 600s = 10m) + # with enough headroom for setup steps and the final API call. + timeoutInMinutes: 60 diff --git a/.github/workflows/ado/sources-upload.yml b/.github/workflows/ado/sources-upload.yml index 9d7f562e8be..46d233f1469 100644 --- a/.github/workflows/ado/sources-upload.yml +++ b/.github/workflows/ado/sources-upload.yml @@ -1,21 +1,23 @@ # Microsoft Corporation # -# Wrapper pipeline — passed to ADO as the entry point. This file owns all -# OneBranch-specific wiring (governed templates repo, Official vs NonOfficial -# variant, featureFlags) and delegates the actual stages/jobs/steps to the -# raw stages template at: +# Wrapper pipeline — passed to ADO as the entry point for the source-upload +# pipeline. This file owns all OneBranch-specific wiring (governed templates +# repo, Official vs NonOfficial variant, featureFlags) and delegates the actual +# stages/jobs/steps to the raw stages template at: # .github/workflows/ado/templates/sources-upload-stages.yml # # Authenticates via Workload Identity Federation (OIDC) and calls the Control -# Tower APIs to: -# 1. Validate that the rendered sources of every changed component can be -# fetched from the lookaside (prcheck). The actual upload happens later -# from the merge queue, not here. -# 2. Submit scratch package builds for changed components. +# Tower 'prcheck' API to upload changed component sources. prcheck returns early +# (no upload) on pull-request triggers -- unmerged code should not consume +# capacity. # -# Helper scripts live under scripts/ci/control-tower/ -# (Control Tower-specific) and scripts/ci/components/ -# (cross-pipeline azldev helpers shared with the GH Actions PR gates). +# The setup + change-detection + validation steps are shared with the +# package-build pipeline via templates/steps/common-steps.yml. +# +# Helper scripts live under: +# - scripts/ci/control-tower/ - (Control Tower-specific) +# - scripts/ci/ado/ - (ADO API: commit-range detection) +# - scripts/ci/components/ - (cross-pipeline azldev helpers shared with the GH Actions PR gates). # # Prerequisites (ADO / Azure Portal): # 1. Entra ID App Registration with audience URI @@ -25,7 +27,8 @@ # subject: sc:////). # 3. ARM service connection in ADO project settings using Workload Identity # Federation (manual). -# 4. ADO branch policy or pipeline PR trigger configured to fire on PRs. +# 4. CI trigger configured (in ADO pipeline settings) to fire on pushes to +# the target branch. # # Variable Group (ADO Pipelines > Library): # Name: "ControlTower-PRCheck" @@ -69,7 +72,7 @@ extends: - template: /.github/workflows/ado/templates/sources-upload-stages.yml@self parameters: outputDirectory: $(Build.ArtifactStagingDirectory)/output - artifactBaseName: prcheck + artifactBaseName: sourceupload containerImage: mcr.microsoft.com/onebranch/azurelinux/build:3.0 poolType: linux serviceConnection: CT-Endpoints-Access-ServiceConnection-DEV diff --git a/.github/workflows/ado/templates/package-build-stages.yml b/.github/workflows/ado/templates/package-build-stages.yml new file mode 100644 index 00000000000..e0cecb9f226 --- /dev/null +++ b/.github/workflows/ado/templates/package-build-stages.yml @@ -0,0 +1,89 @@ +# Microsoft Corporation +# +# Raw stages template for the package-build pipeline (v1 post-merge delta +# build). OneBranch-agnostic: declares the stages/jobs/steps and exposes the +# OneBranch-coupled knobs as parameters. The wrapper at +# .github/workflows/ado/package-build.yml supplies concrete values. +# +# The setup + change-detection + validation steps are shared with the +# source-upload pipeline via templates/steps/common-steps.yml; this template +# appends only the package-build-specific Control Tower call. + +parameters: + - name: outputDirectory + type: string + - name: artifactBaseName + type: string + - name: containerImage + type: string + - name: poolType + type: string + default: linux + - name: serviceConnection + type: string + - name: variableGroup + type: string + - name: timeoutInMinutes + type: number + # Control Tower package target for builds submitted from this pipeline + # (e.g. azl4 for the 4.0 branch, azl5 for 5.0). Bound per-branch by the + # wrapper so a branch's builds land in the correct target. + - name: packageTarget + type: string + +stages: + - stage: PackageBuild + jobs: + - job: PackageBuild + # This is a post-merge build: code has already merged, so there is no PR + # or merge queue to gate. The job fails loud -- any failing step turns + # the run red so submission/validation breakage is visible and alertable + # rather than masked as a partial success. The build itself runs + # asynchronously in Control Tower/Koji; this pipeline only submits it and + # confirms acceptance (see run_package_build.py). + # Must exceed the script's --poll-timeout-seconds (default 600s = 10m) + # with enough headroom for setup steps and the final API call. + timeoutInMinutes: ${{ parameters.timeoutInMinutes }} + pool: + type: ${{ parameters.poolType }} + variables: + - group: ${{ parameters.variableGroup }} + - name: ob_outputDirectory + value: ${{ parameters.outputDirectory }} + - name: ob_artifactBaseName + value: ${{ parameters.artifactBaseName }} + - name: LinuxContainerImage + value: ${{ parameters.containerImage }} + steps: + # Shared setup + change detection + validation (PipAuthenticate, + # install deps, determine commit range, verify locks, prepare change + # set, verify rendered specs). Produces the job-scope variables and + # change-set files the submit step below consumes. + - template: steps/common-steps.yml + + - task: AzureCLI@2 + displayName: "Submit package build to Control Tower" + inputs: + azureSubscription: ${{ parameters.serviceConnection }} + scriptType: bash + scriptLocation: inlineScript + inlineScript: | + set -euo pipefail + + python3 scripts/ci/control-tower/run_package_build.py \ + --api-audience "$API_AUDIENCE" \ + --api-base-url "$API_BASE_URL" \ + --build-reason "$CT_BUILD_REASON" \ + --changed-components-file "$CHANGED_COMPONENTS_FILE" \ + --package-target "${{ parameters.packageTarget }}" \ + --official-build \ + --commit-sha "$SOURCE_COMMIT" \ + --repo-uri "$UPSTREAM_REPO_URL" + env: + API_AUDIENCE: $(ApiAudience) + API_BASE_URL: $(ApiBaseDirectUrl) + # Non-reserved name: an `env:` override of the reserved BUILD_REASON var is silently ignored by the agent. + CT_BUILD_REASON: $(Build.Reason) + CHANGED_COMPONENTS_FILE: $(changedComponentsFile) + SOURCE_COMMIT: $(sourceCommit) + UPSTREAM_REPO_URL: $(Build.Repository.Uri) diff --git a/.github/workflows/ado/templates/sources-upload-stages.yml b/.github/workflows/ado/templates/sources-upload-stages.yml index e0ef83cd99c..7c2adb5913c 100644 --- a/.github/workflows/ado/templates/sources-upload-stages.yml +++ b/.github/workflows/ado/templates/sources-upload-stages.yml @@ -1,13 +1,18 @@ # Microsoft Corporation # -# Raw stages template for the Control Tower integration pipeline. +# Raw stages template for the source-upload pipeline (Control Tower prcheck). +# OneBranch-agnostic: declares the stages/jobs/steps and exposes the +# OneBranch-coupled knobs as parameters. The wrapper at +# .github/workflows/ado/sources-upload.yml supplies concrete values. # -# This template is OneBranch-agnostic: it declares the stages/jobs/steps that -# do the actual work and exposes the OneBranch-coupled knobs as parameters. -# The wrapper at .github/workflows/ado/sources-upload.yml is responsible for -# choosing the OneBranch governed template variant (Official vs NonOfficial), -# configuring its featureFlags, and supplying concrete values for the -# parameters declared below. +# The setup + change-detection + validation steps are shared with the +# package-build pipeline via templates/steps/common-steps.yml; this template +# appends only the source-upload-specific Control Tower call (prcheck). +# +# NOTE: source upload is the original Control Tower flow. It runs prcheck, which +# returns early (no upload) on pull-request triggers -- unmerged code should not +# consume capacity. This pipeline is retained for when source upload returns to +# active duty; the v1 stopgap runs builds via package-build.yml instead. parameters: - name: outputDirectory @@ -27,9 +32,9 @@ parameters: type: number stages: - - stage: Integration + - stage: SourceUpload jobs: - - job: UploadAndBuild + - job: SourceUpload # Non-blocking PR check: failing steps still render red error annotations # and surface in the build-issues view, but the job resolves to # SucceededWithIssues and the run to PartiallySucceeded, which the Azure @@ -39,8 +44,8 @@ stages: # block PRs and the merge queue. # ADO task: 19179 continueOnError: true - # Must exceed the script's --poll-timeout-seconds (default 7200s = 120m) - # with enough headroom for setup steps and the final API call. + # Must exceed the script's --poll-timeout-seconds (run_prcheck.py default + # 7200s = 120m) with enough headroom for setup steps and the final API call. timeoutInMinutes: ${{ parameters.timeoutInMinutes }} pool: type: ${{ parameters.poolType }} @@ -53,175 +58,11 @@ stages: - name: LinuxContainerImage value: ${{ parameters.containerImage }} steps: - - script: | - set -euo pipefail - - # For both triggers, Build.SourceVersion is the relevant source commit: - # - PR trigger: ADO sets it to the GitHub-created merge commit (refs/pull/{n}/merge). - # - CI/merge-queue trigger: it is the commit that landed on the base branch. - source_hash="$SOURCE_COMMIT" - - if [[ "$BUILD_REASON" == "PullRequest" ]]; then - # PR trigger: ADO provides the target branch directly. - target_hash=$(git ls-remote "$UPSTREAM_REPO_URL" "$PR_TARGET_BRANCH" | cut -f1) - else - # CI/merge-queue trigger: extract the base branch from the merge-queue branch name. - # Branch format: refs/heads/gh-readonly-queue/{base_branch}/pr-{pr_number}-{head_sha} - if ! base_branch=$(grep -oP '(?<=^refs/heads/gh-readonly-queue/).+(?=/pr-[^/]+$)' <<< "$SOURCE_BRANCH"); then - echo "##[error]Unsupported SOURCE_BRANCH '$SOURCE_BRANCH' for non-PullRequest build. Expected 'refs/heads/gh-readonly-queue//pr--'." - exit 1 - fi - target_hash=$(git ls-remote "$UPSTREAM_REPO_URL" "refs/heads/$base_branch" | cut -f1) - fi - - echo "Source commit: $source_hash" - echo "Target commit: $target_hash" - - echo "##vso[task.setvariable variable=sourceCommit]$source_hash" - echo "##vso[task.setvariable variable=targetCommit]$target_hash" - env: - BUILD_REASON: $(Build.Reason) - PR_TARGET_BRANCH: $(System.PullRequest.TargetBranch) - SOURCE_BRANCH: $(Build.SourceBranch) - SOURCE_COMMIT: $(Build.SourceVersion) - UPSTREAM_REPO_URL: $(Build.Repository.Uri) - displayName: "Determine source and target commit hashes" - - - task: PipAuthenticate@1 - displayName: "Authenticate pip" - inputs: - artifactFeeds: "azl/ControlTowerFeed" - - - script: | - set -euo pipefail - - echo "##[group]Mock" - tdnf install -y mock mock-rpmautospec python3-chardet - sudo usermod -aG mock "$(whoami)" - echo "##[endgroup]" - - echo "##[group]Azldev" - AZLDEV_VERSION=$(cat .azldev-version) - echo "Installing azldev@${AZLDEV_VERSION}..." - go install "github.com/microsoft/azure-linux-dev-tools/cmd/azldev@${AZLDEV_VERSION}" - - go_bin_path="$(go env GOPATH)/bin" - echo "##vso[task.prependpath]$go_bin_path" - - "$go_bin_path/azldev" --version - echo "##[endgroup]" - - echo "##[group]Python dependencies" - pip install -r scripts/ci/control-tower/requirements.txt - echo "##[endgroup]" - displayName: "Install dependencies" - - # Verify lock files are current. --check-only validates without - # writing, exits nonzero if any lock would change. - - script: | - set -euo pipefail - scripts/ci/control-tower/verify_locks.sh \ - --output-file "$(Build.ArtifactStagingDirectory)/lock-update.json" \ - --publish-dir "$(ob_outputDirectory)" - displayName: "Verify lock files are up to date" - - # Detect changed components AND prepare the render set. The same - # changed-components JSON is consumed by the downstream - # render-verify step (below) and by the CT API calls (further - # down -- prcheck + package-build), so we compute it once here - # and publish it as a triage artifact on every exit path. - # - # azldev hard-fails if any component has sourcesChange == true - # without a corresponding identity change (changeType not in - # {added, changed, deleted}). That combination would let - # attacker-supplied bytes ride into the lookaside under an - # unchanged component identity, so we treat it as hostile and - # fail closed (supply-chain drift tripwire). --include-unchanged - # (set inside compute_changed.sh) ensures the full component - # list is available for downstream consumers. - - script: | - set -euo pipefail - change_set_dir="$(Build.ArtifactStagingDirectory)/change-set" - json_file="$change_set_dir/changed-components.json" - render_set_file="$change_set_dir/render-set.txt" - - # Declare the downstream pipeline variables UP FRONT with empty - # values. ADO's `$(varName)` substitution leaves the literal - # string unchanged when the variable is undefined; we want - # downstream `[[ ! -f "$(renderSetFile)" ]]` assertions to fail - # cleanly on a missing file instead of relying on bash's - # "command not found" behavior for a literal `$(renderSetFile)` - # path. Re-set after success below. - echo "##vso[task.setvariable variable=changedComponentsFile;isreadonly=false]" - echo "##vso[task.setvariable variable=renderSetFile;isreadonly=false]" - - # Publish the changed-components JSON for post-mortem triage on - # EVERY exit path, not just success -- if azldev hard-fails on a - # consistency tripwire the partial JSON is exactly what an - # operator needs to investigate. - publish_artifact() { - if [[ -s "$json_file" ]]; then - mkdir -p "$(ob_outputDirectory)/changed-components" - cp "$json_file" "$(ob_outputDirectory)/changed-components/" || true - fi - } - trap publish_artifact EXIT - - echo "##[group]Preparing change set" - scripts/ci/components/compute_change_set.sh \ - --output-dir "$change_set_dir" \ - --source-commit "$(sourceCommit)" \ - --target-commit "$(targetCommit)" - echo "##[endgroup]" - - echo "##[group]Upload set (sourcesChange == true, changeType in {added, changed})" - upload_count=$(jq -r '[.[] | select(.sourcesChange == true and (.changeType | IN("added", "changed")))] | length' "$json_file") - jq -r '.[] | select(.sourcesChange == true and (.changeType | IN("added", "changed"))) | .component' "$json_file" | sort - echo "Total: $upload_count component(s) to upload." - echo "##[endgroup]" - - # Re-set the variables now that the script has succeeded. - echo "##vso[task.setvariable variable=changedComponentsFile;isreadonly=true]$json_file" - echo "##vso[task.setvariable variable=renderSetFile;isreadonly=true]$render_set_file" - displayName: "Prepare change set" - - # Render the components flagged by the change set above - # (azldev-flagged plus any with hand-edited spec files) in - # --check-only mode: azldev renders to a staging area and diffs - # against the on-disk output without writing. Exits nonzero if - # any component's rendered output would change, catching both - # stale renders and direct hand-edits. - - script: | - set -euo pipefail - # Distinguish "render-set file is missing" (Prepare step - # crashed; supply-chain tripwire or other hard failure) from - # "render-set is empty" (no PR-scoped components). The former - # MUST fail loud so a `continueOnError` job does not silently - # mask the upstream crash. - if [[ ! -f "$(renderSetFile)" ]]; then - echo "##[error]renderSetFile is unset or missing -- 'Prepare change set' did not complete cleanly. Treating as failure rather than silently skipping render." - exit 1 - fi - if [[ ! -s "$(renderSetFile)" ]]; then - echo "No changed components -- skipping render." - exit 0 - fi - echo "##[group]Render set" - sed 's/^/ - /' "$(renderSetFile)" - echo "##[endgroup]" - echo "##[group]Specs rendering + verification" - # `-x` fails loudly if the arg list would split into multiple - # xargs invocations -- a multi-batch render-check would - # silently mask drift in the later batches. `-n 50000` is - # required for `-x` to have any effect (without `-n`/`-L`, - # GNU xargs never splits). `--` ends azldev option parsing so - # component names beginning with `-` (none today) are - # unambiguous. `env AZLDEV_ALLOW_ROOT=1` is inline per - # .github/instructions/ado-pipeline.instructions.md. - xargs -x -n 50000 -d '\n' env AZLDEV_ALLOW_ROOT=1 azldev component render --check-only -- \ - < "$(renderSetFile)" - echo "##[endgroup]" - displayName: "Verify rendered specs are up to date" + # Shared setup + change detection + validation (PipAuthenticate, + # install deps, determine commit range, verify locks, prepare change + # set, verify rendered specs). Produces the job-scope variables and + # change-set files the prcheck step below consumes. + - template: steps/common-steps.yml - task: AzureCLI@2 displayName: "Call Control Tower 'prcheck' API" @@ -235,43 +76,18 @@ stages: python3 scripts/ci/control-tower/run_prcheck.py \ --api-audience "$API_AUDIENCE" \ --api-base-url "$API_BASE_URL" \ - --build-reason "$BUILD_REASON" \ + --build-reason "$CT_BUILD_REASON" \ --changed-components-file "$CHANGED_COMPONENTS_FILE" \ --source-commit "$SOURCE_COMMIT" \ --repo-uri "$UPSTREAM_REPO_URL" env: API_AUDIENCE: $(ApiAudience) API_BASE_URL: $(ApiBaseDirectUrl) - BUILD_REASON: $(Build.Reason) + # Non-reserved name: an `env:` override of the reserved BUILD_REASON var is silently ignored by the agent. + CT_BUILD_REASON: $(Build.Reason) CHANGED_COMPONENTS_FILE: $(changedComponentsFile) SOURCE_COMMIT: $(sourceCommit) # TODO: Target commit is not used. Will be needed once we move detection of affected components to CT. # ADO task: 18816 TARGET_COMMIT: $(targetCommit) UPSTREAM_REPO_URL: $(Build.Repository.Uri) - - - task: AzureCLI@2 - displayName: "Submit package build to Control Tower" - inputs: - azureSubscription: ${{ parameters.serviceConnection }} - scriptType: bash - scriptLocation: inlineScript - inlineScript: | - set -euo pipefail - - python3 scripts/ci/control-tower/run_package_build.py \ - --api-audience "$API_AUDIENCE" \ - --api-base-url "$API_BASE_URL" \ - --build-reason "$BUILD_REASON" \ - --changed-components-file "$CHANGED_COMPONENTS_FILE" \ - --package-target azl4 \ - --official-build \ - --commit-sha "$SOURCE_COMMIT" \ - --repo-uri "$UPSTREAM_REPO_URL" - env: - API_AUDIENCE: $(ApiAudience) - API_BASE_URL: $(ApiBaseDirectUrl) - BUILD_REASON: $(Build.Reason) - CHANGED_COMPONENTS_FILE: $(changedComponentsFile) - SOURCE_COMMIT: $(sourceCommit) - UPSTREAM_REPO_URL: $(Build.Repository.Uri) diff --git a/.github/workflows/ado/templates/steps/common-steps.yml b/.github/workflows/ado/templates/steps/common-steps.yml new file mode 100644 index 00000000000..7b67c98643c --- /dev/null +++ b/.github/workflows/ado/templates/steps/common-steps.yml @@ -0,0 +1,228 @@ +# Microsoft Corporation +# +# Shared step template for the Control Tower integration pipelines (package +# build + source upload). Spliced into each pipeline's single job via: +# +# steps: +# - template: steps/common-steps.yml +# - +# +# Keeping these as STEPS (not a separate job/stage) is deliberate: the +# change-detection steps emit job-scoped pipeline variables (sourceCommit, +# targetCommit, changedComponentsFile, renderSetFile) and write the change-set +# files to disk. +# Those flow freely to later steps in the SAME job but would need output +# variables + artifact upload/download to cross a job boundary. Splicing the +# common steps inline keeps that wiring trivial. +# +# Contract for the including job: +# * It MUST define the job-scope variable `ob_outputDirectory` (both stages +# templates set it from their `outputDirectory` parameter); the verify-locks +# and prepare-change-set steps publish triage artifacts there. +# * These steps assume normal step sequencing -- a failed step stops the ones +# after it (the default; do NOT set step-level `continueOnError` on them). +# Later steps depend on earlier ones, e.g. the render step trusts that +# "Prepare change set" ran and set renderSetFile to a real path. A job-level +# `continueOnError` (which only affects how the JOB result is reported, not +# intra-job step flow) is fine -- the source-upload pipeline uses one. +# +# The steps below are identical across both pipelines: ensure full git history, +# authenticate + install dependencies, resolve the (target, source) commit +# range, and validate (lock files + rendered specs). The pipeline-specific +# Control Tower call (package build or prcheck) is appended by the including +# stages template. + +steps: + # Ensure the full git history is present, ONCE, before anything that needs it. + # The OneBranch checkout may be shallow (depth 1), but lock resolution, spec + # rendering (rpmautospec derives Release/changelog from `git log` over the + # lock files), and commit-range detection all require the complete history. + # Doing the unshallow here means the helper scripts can assume full history + # and never fetch themselves -- a `git fetch --depth=N` inside a script would + # re-shallow a full clone, which silently corrupts the rpmautospec Release + # calculation (and is a footgun when running the scripts locally). + - script: | + set -euo pipefail + if [ "$(git rev-parse --is-shallow-repository)" = "true" ]; then + echo "##[group]Fetching full git history" + git fetch --unshallow + echo "##[endgroup]" + fi + displayName: "Ensure full git history" + + - task: PipAuthenticate@1 + displayName: "Authenticate pip" + inputs: + artifactFeeds: "azl/ControlTowerFeed" + + - script: | + set -euo pipefail + + echo "##[group]Mock" + tdnf install -y mock mock-rpmautospec python3-chardet + sudo usermod -aG mock "$(whoami)" + echo "##[endgroup]" + + echo "##[group]Azldev" + AZLDEV_VERSION=$(cat .azldev-version) + echo "Installing azldev@${AZLDEV_VERSION}..." + go install "github.com/microsoft/azure-linux-dev-tools/cmd/azldev@${AZLDEV_VERSION}" + + go_bin_path="$(go env GOPATH)/bin" + echo "##vso[task.prependpath]$go_bin_path" + + "$go_bin_path/azldev" --version + echo "##[endgroup]" + + echo "##[group]Python dependencies" + pip install -r scripts/ci/control-tower/requirements.txt + pip install -r scripts/ci/ado/requirements.txt + echo "##[endgroup]" + displayName: "Install dependencies" + + # Resolve the (target, source) commit range for this post-merge build: + # source = the commit that triggered this run (Build.SourceVersion). + # target = the previous CI build's commit on this branch, looked up + # via the ADO Builds API. Pairing each build with the + # one immediately before it keeps concurrent runs from + # overlapping and captures every commit a rebase merge + # appends. Runs after the dependency-install step because + # it imports the azure-devops SDK. See + # scripts/ci/ado/determine_commit_range.py for the full + # rationale and the first-run (source^1) fallback. + - script: | + set -euo pipefail + # The helper prints the resolved range to stdout as two lines: + # sourceCommit= + # targetCommit= + # (all diagnostics go to stderr). We parse them and set the + # pipeline variables HERE so the variable wiring is visible in + # the YAML rather than hidden in the script. + range_output="$(python3 scripts/ci/ado/determine_commit_range.py \ + --definition-id "$SYSTEM_DEFINITIONID" \ + --current-build-id "$BUILD_BUILDID" \ + --branch "$DELTA_QUERY_BRANCH" \ + --source-commit "$BUILD_SOURCEVERSION")" + echo "$range_output" + + source_commit="$(sed -n 's/^sourceCommit=//p' <<< "$range_output")" + target_commit="$(sed -n 's/^targetCommit=//p' <<< "$range_output")" + if [[ -z "$source_commit" || -z "$target_commit" ]]; then + echo "##[error]determine_commit_range.py did not emit a source/target commit range." + exit 1 + fi + + echo "##vso[task.setvariable variable=sourceCommit]$source_commit" + echo "##vso[task.setvariable variable=targetCommit]$target_commit" + env: + SYSTEM_COLLECTIONURI: $(System.CollectionUri) + SYSTEM_TEAMPROJECT: $(System.TeamProject) + SYSTEM_ACCESSTOKEN: $(System.AccessToken) + SYSTEM_DEFINITIONID: $(System.DefinitionId) + BUILD_BUILDID: $(Build.BuildId) + DELTA_QUERY_BRANCH: $(Build.SourceBranch) + BUILD_SOURCEVERSION: $(Build.SourceVersion) + displayName: "Determine source and target commit range" + + # Verify lock files are current. --check-only validates without + # writing, exits nonzero if any lock would change. + - script: | + set -euo pipefail + scripts/ci/control-tower/verify_locks.sh \ + --output-file "$(Build.ArtifactStagingDirectory)/lock-update.json" \ + --publish-dir "$(ob_outputDirectory)" + displayName: "Verify lock files are up to date" + + # Detect changed components AND prepare the render set. The same + # changed-components JSON is consumed by the downstream + # render-verify step (below) and by the CT API calls (appended by + # the including stages template -- prcheck or package-build), so we + # compute it once here and publish it as a triage artifact on every + # exit path. + # + # azldev hard-fails if any component has sourcesChange == true + # without a corresponding identity change (changeType not in + # {added, changed, deleted}). That combination would let + # attacker-supplied bytes ride into the lookaside under an + # unchanged component identity, so we treat it as hostile and + # fail closed (supply-chain drift tripwire). --include-unchanged + # (set inside compute_changed.sh) ensures the full component + # list is available for downstream consumers. + - script: | + set -euo pipefail + change_set_dir="$(Build.ArtifactStagingDirectory)/change-set" + json_file="$change_set_dir/changed-components.json" + render_set_file="$change_set_dir/render-set.txt" + + # Publish the changed-components JSON for post-mortem triage on + # EVERY exit path, not just success -- if azldev hard-fails on a + # consistency tripwire the partial JSON is exactly what an + # operator needs to investigate. + publish_artifact() { + if [[ -s "$json_file" ]]; then + mkdir -p "$(ob_outputDirectory)/changed-components" + cp "$json_file" "$(ob_outputDirectory)/changed-components/" || true + fi + } + trap publish_artifact EXIT + + echo "##[group]Preparing change set" + scripts/ci/components/compute_change_set.sh \ + --output-dir "$change_set_dir" \ + --source-commit "$SOURCE_COMMIT" \ + --target-commit "$TARGET_COMMIT" + echo "##[endgroup]" + + echo "##[group]Upload set (sourcesChange == true, changeType in {added, changed})" + upload_count=$(jq -r '[.[] | select(.sourcesChange == true and (.changeType | IN("added", "changed")))] | length' "$json_file") + jq -r '.[] | select(.sourcesChange == true and (.changeType | IN("added", "changed"))) | .component' "$json_file" | sort + echo "Total: $upload_count component(s) to upload." + echo "##[endgroup]" + + # Publish the downstream pipeline variables now that the change set is + # ready (consumed by the render-verify step and the trailing Control + # Tower call). + echo "##vso[task.setvariable variable=changedComponentsFile;isreadonly=true]$json_file" + echo "##vso[task.setvariable variable=renderSetFile;isreadonly=true]$render_set_file" + env: + SOURCE_COMMIT: $(sourceCommit) + TARGET_COMMIT: $(targetCommit) + displayName: "Prepare change set" + + # Render the components flagged by the change set above + # (azldev-flagged plus any with hand-edited spec files) in + # --check-only mode: azldev renders to a staging area and diffs + # against the on-disk output without writing. Exits nonzero if + # any component's rendered output would change, catching both + # stale renders and direct hand-edits. + - script: | + set -euo pipefail + # A missing render-set file means "Prepare change set" did not complete; + # fail loud rather than silently skipping the render check. + if [[ ! -f "$RENDER_SET_FILE" ]]; then + echo "##[error]render-set file '$RENDER_SET_FILE' is missing -- 'Prepare change set' did not complete cleanly." + exit 1 + fi + # An empty render set just means no PR-scoped components changed. + if [[ ! -s "$RENDER_SET_FILE" ]]; then + echo "No changed components -- skipping render." + exit 0 + fi + echo "##[group]Render set" + sed 's/^/ - /' "$RENDER_SET_FILE" + echo "##[endgroup]" + echo "##[group]Specs rendering + verification" + # `-x` fails loudly if the arg list would split into multiple + # xargs invocations -- a multi-batch render-check would + # silently mask drift in the later batches. `-n 50000` is + # required for `-x` to have any effect (without `-n`/`-L`, + # GNU xargs never splits). `--` ends azldev option parsing so + # component names beginning with `-` (none today) are + # unambiguous. `env AZLDEV_ALLOW_ROOT=1` is inline per + # .github/instructions/ado-pipeline.instructions.md. + xargs -x -n 50000 -d '\n' env AZLDEV_ALLOW_ROOT=1 azldev component render --check-only -- \ + < "$RENDER_SET_FILE" + echo "##[endgroup]" + env: + RENDER_SET_FILE: $(renderSetFile) + displayName: "Verify rendered specs are up to date" diff --git a/ruff.toml b/ruff.toml index e264df7ff23..a22db0a2bc8 100644 --- a/ruff.toml +++ b/ruff.toml @@ -18,6 +18,12 @@ extend-select = ["D401"] fixable = ["ALL"] +[lint.per-file-ignores] +# Helpers under scripts/ci/ are executed directly (python3 path/to/script.py) +# with sibling-module imports, not imported as a package, so they intentionally +# have no __init__.py. Silence the implicit-namespace-package rule for that tree. +"scripts/ci/**/*.py" = ["INP001"] + [lint.isort] # Just force this, its better than dealing with a mishmash of modules that have annotations and those that don't required-imports = ["from __future__ import annotations"] diff --git a/scripts/ci/ado/README.md b/scripts/ci/ado/README.md new file mode 100644 index 00000000000..2c857b3ce56 --- /dev/null +++ b/scripts/ci/ado/README.md @@ -0,0 +1,45 @@ +# Azure DevOps pipeline helpers + +Helpers for talking to the **Azure DevOps control plane** (the pipelines' own +build metadata) from ADO YAML pipelines, using the official +[`azure-devops`](https://github.com/microsoft/azure-devops-python-api) SDK. + +| File | Purpose | +| ---- | ------- | +| `determine_commit_range.py` | Resolves the `(target, source)` commit range for the post-merge delta build and prints it to stdout as `sourceCommit=` / `targetCommit=` lines. The calling step sets the pipeline variables. | +| `requirements.txt` | Python dependencies (`azure-devops`), installed from the internal feed. | + +## Conventions + +- **Use the SDK.** Talk to ADO through the `azure-devops` package, not a + bespoke REST layer. Pinned in `requirements.txt`; bump deliberately. +- **Auth + step ordering** follow the "Reading ADO build metadata" section of + [`ado-pipeline.instructions.md`](../../../.github/instructions/ado-pipeline.instructions.md): + build-identity `System.AccessToken` auth, and the helper runs **after** the + dependency-install step because the SDK is a pip dependency. + +## Caller contract + +`determine_commit_range.py` expects: + +- **Args:** `--definition-id`, `--current-build-id`, `--branch` (full ref, e.g. + `refs/heads/4.0`), `--source-commit`, optional `--top`. +- **Env:** `SYSTEM_COLLECTIONURI`, `SYSTEM_TEAMPROJECT`, `SYSTEM_ACCESSTOKEN`. +- **Git:** read-only. It assumes the full history is already present (the + pipeline's "Ensure full git history" step fetches it once up front) and never + fetches — a `git fetch --depth=N` would re-shallow a full clone. +- **Output:** two `key=value` lines on **stdout** (`sourceCommit=` and + `targetCommit=`); all diagnostics go to **stderr**. The caller parses + stdout and owns the `##vso[task.setvariable]` wiring, so pipeline-variable + assignment stays visible in the YAML. + +It is best-effort: if the previous build cannot be found (first run) or the ADO +query fails, it falls back to `target = source^1` (warning on stderr) rather +than failing the pipeline. A hard failure (invalid source SHA, or no parent +found for the fallback) exits non-zero so the calling step fails. + +## Callers + +- `templates/steps/common-steps.yml` "Determine source and target commit range" + step → `determine_commit_range.py` (shared by the package-build and + source-upload pipelines). diff --git a/scripts/ci/ado/determine_commit_range.py b/scripts/ci/ado/determine_commit_range.py new file mode 100644 index 00000000000..51b18151ea4 --- /dev/null +++ b/scripts/ci/ado/determine_commit_range.py @@ -0,0 +1,227 @@ +"""Resolve the ``(target, source)`` commit range for a post-merge delta build. + +Strategy (see ``.github/workflows/ado/templates/steps/common-steps.yml``): + +* ``source`` is the commit that triggered this run (``Build.SourceVersion``). +* ``target`` is the ``sourceVersion`` of the immediately-preceding CI build of + this pipeline definition on the same branch, selected by build id — + regardless of that build's result. + +Selecting the immediately-preceding build *by id* (not by success) is what +keeps concurrent runs from overlapping: build N always pairs with build N-1, so +successive merges produce ADJACENT, non-overlapping commit ranges even when an +earlier run is still in flight or has failed. A failed/cancelled run still +"claims" its range — those commits are skipped until the weekly true-up job — +which is the accepted bias-to-miss tradeoff. Overlapping ranges would cause +NEVR collisions in the build system, which is far worse than a transient gap. + +Rebase-merge aware: because the range is a two-commit span (previous tip → +current tip), it captures EVERY commit a rebase merge appends, not just the +tip. ``azldev component changed`` then tree-diffs the two endpoints. + +Fallback (first run, or no prior CI build found): ``target = source^1`` with a +warning. That run only builds the single tip commit's components, self- +correcting on the next push. + +The resolved hashes are printed to stdout as two ``key=value`` lines +(``sourceCommit=`` and ``targetCommit=``). The calling pipeline step +reads them and sets the corresponding ADO pipeline variables, so the +variable wiring stays visible in the YAML rather than hidden here. All +diagnostic output goes to stderr to keep stdout machine-readable. + +This script is read-only with respect to git: it assumes the full history is +already present (the pipeline fetches it up front in a single "Ensure full git +history" step) and never fetches itself. A ``git fetch --depth=N`` here would +re-shallow a full clone -- a footgun, especially when running locally. +""" + +from __future__ import annotations + +import argparse +import os +import re +import subprocess +import sys + +from azure.devops.connection import Connection # pyright: ignore[reportMissingTypeStubs] +from azure.devops.exceptions import ClientException # pyright: ignore[reportMissingTypeStubs] +from msrest.authentication import BasicAuthentication + +# ADO predefined variables required to reach the control-plane REST API, read +# from the step environment. +_ENV_COLLECTION_URI = "SYSTEM_COLLECTIONURI" +_ENV_PROJECT = "SYSTEM_TEAMPROJECT" +_ENV_TOKEN = "SYSTEM_ACCESSTOKEN" # noqa: S105 - env var NAME, not a secret value + +# A build is eligible as a baseline only if it was itself a CI build of the +# branch. Manual / PR / scheduled runs are excluded so that a one-off manual +# test run of this pipeline cannot become the baseline for the next real CI +# build (which would skip everything in between). +_BASELINE_REASONS = frozenset({"individualCI", "batchedCI"}) + +_SHA_RE = re.compile(r"^[0-9a-f]{40}$") + + +def _log(message: str) -> None: + """Print a diagnostic message to stderr, keeping stdout machine-readable.""" + print(message, file=sys.stderr) + + +def _emit_range(source_commit: str, target_commit: str) -> None: + """Print the resolved range to stdout as ``key=value`` lines. + + The calling pipeline step parses these two lines and sets the + ``sourceCommit`` / ``targetCommit`` pipeline variables, so the + variable wiring lives in the YAML rather than in this script. + """ + _log(f"Resolved range: target={target_commit} source={source_commit}") + print(f"sourceCommit={source_commit}") + print(f"targetCommit={target_commit}") + + +def _run_git(args: list[str]) -> subprocess.CompletedProcess[str]: + """Run a ``git`` command, capturing text output without raising on failure.""" + return subprocess.run(["git", *args], check=False, capture_output=True, text=True) + + +def _commit_present(commit: str) -> bool: + """Return whether ``commit`` exists as a commit object in the local clone.""" + return _run_git(["cat-file", "-e", f"{commit}^{{commit}}"]).returncode == 0 + + +def _parent_commit(commit: str) -> str | None: + """Return ``commit^1`` (40-hex), or None if it cannot be resolved. + + Assumes full history is present (the pipeline fetches it up front), so no + fetch is performed here. + """ + result = _run_git(["rev-parse", "--verify", "--quiet", f"{commit}^1"]) + parent = result.stdout.strip() + if result.returncode != 0 or not _SHA_RE.match(parent): + return None + return parent + + +def _fetch_recent_builds(*, definition_id: int, branch: str, top: int) -> list[object]: + """Return recent builds for ``definition_id`` on ``branch`` via the ADO SDK. + + Authenticates with the pipeline's ``System.AccessToken`` (read from the + environment) using the SDK's documented ``BasicAuthentication`` pattern; the + Azure DevOps REST API accepts the job access token as a PAT-equivalent + credential. This reads the pipeline's own build history on the ADO control + plane, so the default project job-authorization scope is sufficient and the + Workload Identity Federation service-connection rule does not apply. + + Args: + definition_id: Build definition (pipeline) id to filter by. + branch: Full source branch ref, e.g. ``refs/heads/4.0``. + top: Maximum number of builds to return (most recent first). + + Returns: + The list of ``Build`` objects returned by the SDK. + + Raises: + RuntimeError: If a required environment variable is missing or empty. + """ + missing = [name for name in (_ENV_COLLECTION_URI, _ENV_PROJECT, _ENV_TOKEN) if not os.environ.get(name)] + if missing: + msg = f"Missing required ADO environment variable(s): {', '.join(missing)}." + raise RuntimeError(msg) + credentials = BasicAuthentication("", os.environ[_ENV_TOKEN]) + connection = Connection(base_url=os.environ[_ENV_COLLECTION_URI], creds=credentials) + build_client = connection.clients.get_build_client() + return build_client.get_builds( + os.environ[_ENV_PROJECT], + definitions=[definition_id], + branch_name=branch, + top=top, + query_order="queueTimeDescending", + ) + + +def _select_baseline(builds: list[object], current_build_id: int) -> str | None: + """Pick the source commit of the immediately-preceding eligible CI build. + + Args: + builds: ``Build`` objects from :func:`_fetch_recent_builds`. + current_build_id: Id of the running build; only earlier builds qualify. + + Returns: + The ``source_version`` of the highest-id eligible build, or None. + """ + candidates: list[tuple[int, str]] = [] + for build in builds: + build_id = getattr(build, "id", None) + if not isinstance(build_id, int) or build_id >= current_build_id: + continue + if getattr(build, "reason", None) not in _BASELINE_REASONS: + continue + source_version = getattr(build, "source_version", None) + if isinstance(source_version, str) and _SHA_RE.match(source_version.lower()): + candidates.append((build_id, source_version.lower())) + if not candidates: + return None + return max(candidates, key=lambda item: item[0])[1] + + +def _parse_args() -> argparse.Namespace: + """Parse command-line arguments.""" + parser = argparse.ArgumentParser(description="Resolve the post-merge delta build commit range.") + parser.add_argument("--definition-id", type=int, required=True, help="Build definition (pipeline) id.") + parser.add_argument("--current-build-id", type=int, required=True, help="Id of the running build.") + parser.add_argument("--branch", required=True, help="Full source branch ref, e.g. refs/heads/4.0.") + parser.add_argument("--source-commit", required=True, help="The triggering commit SHA (Build.SourceVersion).") + parser.add_argument("--top", type=int, default=100, help="How many recent builds to inspect.") + return parser.parse_args() + + +def main() -> int: + """Resolve the range, print it to stdout, and return a process exit code.""" + args = _parse_args() + + source_commit = str(args.source_commit).strip().lower() + if not _SHA_RE.match(source_commit): + _log(f"ERROR: --source-commit is not a 40-character hex SHA: {source_commit!r}") + return 1 + + target_commit: str | None = None + + try: + builds = _fetch_recent_builds( + definition_id=args.definition_id, + branch=args.branch, + top=args.top, + ) + target_commit = _select_baseline(builds, args.current_build_id) + except (ClientException, OSError, RuntimeError) as exc: + _log(f"WARNING: Could not query previous builds ({exc}); falling back to source^1.") + + if target_commit is None: + _log( + "WARNING: No previous CI build found for this branch; building only " + "the tip commit (target = source^1). The weekly true-up job covers " + "any gap." + ) + target_commit = _parent_commit(source_commit) + if target_commit is None: + _log("ERROR: Unable to determine a parent of the source commit; cannot compute a build range.") + return 1 + _emit_range(source_commit, target_commit) + return 0 + + # Both endpoints must be present for the downstream tree diff in the + # change-set step. Full history is fetched once by the pipeline before this + # step runs, so we only sanity-check presence here (no fetching). + missing = [commit for commit in (target_commit, source_commit) if not _commit_present(commit)] + if missing: + _log( + f"WARNING: commit(s) not present locally: {', '.join(missing)}; the change-set step may be " + "unable to diff them. Ensure the full git history was fetched before this step." + ) + + _emit_range(source_commit, target_commit) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/ci/ado/requirements.txt b/scripts/ci/ado/requirements.txt new file mode 100644 index 00000000000..a133e4e8f97 --- /dev/null +++ b/scripts/ci/ado/requirements.txt @@ -0,0 +1,2 @@ +# azure-devops ships beta-only; b4 is the latest 7.1 release (not a typo). +azure-devops==7.1.0b4 diff --git a/scripts/ci/control-tower/verify_locks.sh b/scripts/ci/control-tower/verify_locks.sh index 736046b49b2..8dfe3d478b7 100755 --- a/scripts/ci/control-tower/verify_locks.sh +++ b/scripts/ci/control-tower/verify_locks.sh @@ -21,12 +21,10 @@ done # The config key may not be present on every agent image, so tolerate its absence. git config --unset extensions.worktreeConfig || true -# Full history is needed for lock resolution and spec rendering. -if [ "$(git rev-parse --is-shallow-repository)" = "true" ]; then - echo "##[group]Fetching full git history" - git fetch --unshallow - echo "##[endgroup]" -fi +# NOTE: full git history (needed for lock resolution and rpmautospec Release +# calculation) is ensured ONCE by the pipeline's "Ensure full git history" step +# (.github/workflows/ado/templates/steps/common-steps.yml) before this script +# runs. This script assumes it is present and does not fetch. mkdir -p "$(dirname "$output_file")"