Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion .github/instructions/ado-pipeline.instructions.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
applyTo: ".github/workflows/ado/*.yml,.github/workflows/ado/templates/*.yml,scripts/ci/**"
applyTo: ".github/workflows/ado/*.yml,.github/workflows/ado/templates/**/*.yml,scripts/ci/**"
description: "Authoring and maintenance rules for Azure DevOps YAML pipelines under .github/workflows/ado/ (wrappers + raw stage templates under templates/) and their helper scripts under scripts/ci/ that run as GitHub PR checks or in the merge queue. Apply when creating or modifying any pipeline in that folder, or any script invoked by one — covers the wrapper/raw split, required OneBranch templates (Official vs NonOfficial), Workload Identity Federation service connections, Control Tower audience URIs, internal-only dependency sources (Go / Python / container images), Python-over-shell scripting, and security hardening."
---

Expand Down Expand Up @@ -39,6 +39,8 @@ File-pairing convention: a wrapper at `.github/workflows/ado/<name>.yml` pairs w

See [.github/workflows/ado/sources-upload.yml](.github/workflows/ado/sources-upload.yml) and [.github/workflows/ado/templates/sources-upload-stages.yml](.github/workflows/ado/templates/sources-upload-stages.yml) for the canonical example.

Shared **step sub-templates** live under `.github/workflows/ado/templates/steps/<name>.yml` and are spliced into a job's `steps:` via `- template: steps/<name>.yml` (path relative to the including stages template). Use these to share step sequences across stages templates that differ only in a trailing pipeline-specific step. Splicing as **steps** (not a separate job/stage) keeps job-scoped pipeline variables and on-disk files flowing to the steps that follow — a separate job would force output variables + artifact upload/download. The including job must define any job-scope variables the shared steps reference (e.g. `ob_outputDirectory`). See [.github/workflows/ado/templates/steps/common-steps.yml](.github/workflows/ado/templates/steps/common-steps.yml), shared by the `package-build` and `sources-upload` pipelines.

## OneBranch templates (MANDATORY — wrapper only)

The rules in this section apply to the **wrapper** file. The raw stages template MUST NOT reference OneBranch at all.
Expand Down Expand Up @@ -192,6 +194,25 @@ Symptom that you need this: `ERR Error: this command may not be run as root`.

This is NOT safe for general use, only for disposable CI environments.

### Reading ADO build metadata (control plane)

To read the pipeline's own control-plane data (build history, definitions) from a helper script, use the official [`azure-devops`](https://github.com/microsoft/azure-devops-python-api) SDK rather than a hand-rolled REST client. Pin it in the area's `requirements.txt` and install it through the standard `PipAuthenticate@1` + `pip install` step.

Authenticate with the pipeline's `System.AccessToken`, which the ADO REST API accepts as a PAT-equivalent credential via the SDK's documented `BasicAuthentication` pattern:

```python
from azure.devops.connection import Connection
from msrest.authentication import BasicAuthentication

connection = Connection(base_url=collection_uri, creds=BasicAuthentication("", access_token))
build_client = connection.clients.get_build_client()
```

- This is the ADO **control plane** (the pipeline's own builds/repos), **not** Azure Resource Manager or Control Tower — so the Workload Identity Federation service-connection rule does **not** apply. The build identity is the correct least-privilege caller.
- Map the token in via the step `env:` block (`SYSTEM_ACCESSTOKEN: $(System.AccessToken)`), never on the command line. In a **YAML** pipeline there is no "Allow scripts to access the OAuth token" toggle (that is Classic-only) — the `env:` mapping is all that is needed.
- Reading builds of the pipeline's **own** definition in the same project is covered by the default **project** job-authorization scope and the `{Project} Build Service ({Org})` identity's default build-read permission. Only revisit if your org has tightened the defaults.
- Because the SDK is a pip dependency (not stdlib), any step that imports it MUST run **after** the dependency-install step (which itself follows `PipAuthenticate@1`).

## Security hardening

Apply all of these unless there is a documented reason not to:
Expand Down
87 changes: 87 additions & 0 deletions .github/workflows/ado/package-build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
# Microsoft Corporation
#
# Wrapper pipeline — passed to ADO as the entry point for the package-build
# pipeline. This file owns all OneBranch-specific wiring (governed templates
# repo, Official vs NonOfficial variant, featureFlags) and delegates the actual
# stages/jobs/steps to the raw stages template at:
# .github/workflows/ado/templates/package-build-stages.yml
#
# Authenticates via Workload Identity Federation (OIDC) and calls the Control
# Tower APIs to run the v1 post-merge delta build (stopgap until source upload
# is reworked):
# 1. Resolve the (target, source) commit range for this push from the
# previous CI build (ADO Builds API).
# 2. Submit official package builds for the components that changed across
# that range, via Workload Identity Federation (OIDC) to Control Tower.
#
# The setup + change-detection + validation steps are shared with the
# source-upload pipeline via templates/steps/common-steps.yml.
#
# Helper scripts live under:
# - scripts/ci/control-tower/ - (Control Tower-specific)
# - scripts/ci/ado/ - (ADO API: commit-range detection)
# - scripts/ci/components/ - (cross-pipeline azldev helpers shared with the GH Actions PR gates).
#
# Prerequisites (ADO / Azure Portal):
# 1. Entra ID App Registration with audience URI
# "api://<ControlTower-ClientId>" (see variable group below).
# 2. Federated identity credential on the app registration for the ADO
# service connection (issuer: https://vstoken.dev.azure.com/<org-id>,
# subject: sc://<org>/<project>/<service-connection-name>).
# 3. ARM service connection in ADO project settings using Workload Identity
# Federation (manual).
# 4. CI trigger configured (in ADO pipeline settings) to fire on pushes to
# the target branch.
#
# Variable Group (ADO Pipelines > Library):
# Name: "ControlTower-PRCheck"
# Required variables:
# - ApiAudience : Entra ID audience URI for the Control Tower app
# - ApiBaseDirectUrl : Direct base URL of the Control Tower APIM endpoint (bypasses Azure Front Door)

# Trigger controlled by ADO branch policy — not YAML triggers.
trigger: none

pr: none

resources:
repositories:
- repository: templates
type: git
name: OneBranch.Pipelines/GovernedTemplates
ref: refs/heads/main

extends:
template: v2/OneBranch.Official.CrossPlat.yml@templates
parameters:
featureFlags:
golang:
internalModuleProxy:
enabled: true
LinuxHostVersion:
Network: R1
runOnHost: true
EnableCDPxPAT: false

# https://aka.ms/obpipelines/sdl
globalSdl:
disableLegacyManifest: true
sbom:
enabled: false
tsa:
enabled: false

stages:
- template: /.github/workflows/ado/templates/package-build-stages.yml@self
parameters:
outputDirectory: $(Build.ArtifactStagingDirectory)/output
artifactBaseName: packagebuild
containerImage: mcr.microsoft.com/onebranch/azurelinux/build:3.0
poolType: linux
serviceConnection: CT-Endpoints-Access-ServiceConnection-DEV
variableGroup: ControlTower-PRCheck
# Control Tower package target for the 4.0 branch.
packageTarget: azl4
# Must exceed the script's --poll-timeout-seconds (default 600s = 10m)
# with enough headroom for setup steps and the final API call.
timeoutInMinutes: 60
31 changes: 17 additions & 14 deletions .github/workflows/ado/sources-upload.yml
Original file line number Diff line number Diff line change
@@ -1,21 +1,23 @@
# Microsoft Corporation
#
# Wrapper pipeline — passed to ADO as the entry point. This file owns all
# OneBranch-specific wiring (governed templates repo, Official vs NonOfficial
# variant, featureFlags) and delegates the actual stages/jobs/steps to the
# raw stages template at:
# Wrapper pipeline — passed to ADO as the entry point for the source-upload
# pipeline. This file owns all OneBranch-specific wiring (governed templates
# repo, Official vs NonOfficial variant, featureFlags) and delegates the actual
# stages/jobs/steps to the raw stages template at:
# .github/workflows/ado/templates/sources-upload-stages.yml
#
# Authenticates via Workload Identity Federation (OIDC) and calls the Control
# Tower APIs to:
# 1. Validate that the rendered sources of every changed component can be
# fetched from the lookaside (prcheck). The actual upload happens later
# from the merge queue, not here.
# 2. Submit scratch package builds for changed components.
# Tower 'prcheck' API to upload changed component sources. prcheck returns early
# (no upload) on pull-request triggers -- unmerged code should not consume
# capacity.
#
# Helper scripts live under scripts/ci/control-tower/
# (Control Tower-specific) and scripts/ci/components/
# (cross-pipeline azldev helpers shared with the GH Actions PR gates).
# The setup + change-detection + validation steps are shared with the
# package-build pipeline via templates/steps/common-steps.yml.
#
# Helper scripts live under:
# - scripts/ci/control-tower/ - (Control Tower-specific)
# - scripts/ci/ado/ - (ADO API: commit-range detection)
# - scripts/ci/components/ - (cross-pipeline azldev helpers shared with the GH Actions PR gates).
#
# Prerequisites (ADO / Azure Portal):
# 1. Entra ID App Registration with audience URI
Expand All @@ -25,7 +27,8 @@
# subject: sc://<org>/<project>/<service-connection-name>).
# 3. ARM service connection in ADO project settings using Workload Identity
# Federation (manual).
# 4. ADO branch policy or pipeline PR trigger configured to fire on PRs.
# 4. CI trigger configured (in ADO pipeline settings) to fire on pushes to
# the target branch.
#
# Variable Group (ADO Pipelines > Library):
# Name: "ControlTower-PRCheck"
Expand Down Expand Up @@ -69,7 +72,7 @@ extends:
- template: /.github/workflows/ado/templates/sources-upload-stages.yml@self
parameters:
outputDirectory: $(Build.ArtifactStagingDirectory)/output
artifactBaseName: prcheck
artifactBaseName: sourceupload
containerImage: mcr.microsoft.com/onebranch/azurelinux/build:3.0
poolType: linux
serviceConnection: CT-Endpoints-Access-ServiceConnection-DEV
Expand Down
89 changes: 89 additions & 0 deletions .github/workflows/ado/templates/package-build-stages.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Microsoft Corporation
#
# Raw stages template for the package-build pipeline (v1 post-merge delta
# build). OneBranch-agnostic: declares the stages/jobs/steps and exposes the
# OneBranch-coupled knobs as parameters. The wrapper at
# .github/workflows/ado/package-build.yml supplies concrete values.
#
# The setup + change-detection + validation steps are shared with the
# source-upload pipeline via templates/steps/common-steps.yml; this template
# appends only the package-build-specific Control Tower call.

parameters:
- name: outputDirectory
type: string
- name: artifactBaseName
type: string
- name: containerImage
type: string
- name: poolType
type: string
default: linux
- name: serviceConnection
type: string
- name: variableGroup
type: string
- name: timeoutInMinutes
type: number
# Control Tower package target for builds submitted from this pipeline
# (e.g. azl4 for the 4.0 branch, azl5 for 5.0). Bound per-branch by the
# wrapper so a branch's builds land in the correct target.
- name: packageTarget
type: string

stages:
- stage: PackageBuild
jobs:
- job: PackageBuild
# This is a post-merge build: code has already merged, so there is no PR
# or merge queue to gate. The job fails loud -- any failing step turns
# the run red so submission/validation breakage is visible and alertable
# rather than masked as a partial success. The build itself runs
# asynchronously in Control Tower/Koji; this pipeline only submits it and
# confirms acceptance (see run_package_build.py).
# Must exceed the script's --poll-timeout-seconds (default 600s = 10m)
# with enough headroom for setup steps and the final API call.
timeoutInMinutes: ${{ parameters.timeoutInMinutes }}
pool:
type: ${{ parameters.poolType }}
variables:
- group: ${{ parameters.variableGroup }}
- name: ob_outputDirectory
value: ${{ parameters.outputDirectory }}
- name: ob_artifactBaseName
value: ${{ parameters.artifactBaseName }}
- name: LinuxContainerImage
value: ${{ parameters.containerImage }}
steps:
# Shared setup + change detection + validation (PipAuthenticate,
# install deps, determine commit range, verify locks, prepare change
# set, verify rendered specs). Produces the job-scope variables and
# change-set files the submit step below consumes.
- template: steps/common-steps.yml

- task: AzureCLI@2
displayName: "Submit package build to Control Tower"
inputs:
azureSubscription: ${{ parameters.serviceConnection }}
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
set -euo pipefail

python3 scripts/ci/control-tower/run_package_build.py \
--api-audience "$API_AUDIENCE" \
--api-base-url "$API_BASE_URL" \
--build-reason "$CT_BUILD_REASON" \
--changed-components-file "$CHANGED_COMPONENTS_FILE" \
--package-target "${{ parameters.packageTarget }}" \
--official-build \
--commit-sha "$SOURCE_COMMIT" \
--repo-uri "$UPSTREAM_REPO_URL"
env:
API_AUDIENCE: $(ApiAudience)
API_BASE_URL: $(ApiBaseDirectUrl)
# Non-reserved name: an `env:` override of the reserved BUILD_REASON var is silently ignored by the agent.
CT_BUILD_REASON: $(Build.Reason)
CHANGED_COMPONENTS_FILE: $(changedComponentsFile)
SOURCE_COMMIT: $(sourceCommit)
UPSTREAM_REPO_URL: $(Build.Repository.Uri)
Loading
Loading