feat: gate custom cloud cert pull on RCV 1P opt-in by rchincha · Pull Request #7958 · Azure/AgentBaker

rchincha · 2026-02-26T01:13:21Z

add opt-in check via acms/isOptedInForRootCerts before certificate retrieval
use operation-request endpoints for RCV 1P root/intermediate certificate pull
keep fallback to legacy cacertificates flow when not opted in
harden opt-in detection with curl --fail and JSON boolean match
normalize Flatcar cert installation by converting .crt artifacts to .pem during copy

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Copilot

Pull request overview

This PR introduces RCV 1P (Root Certificate Validation 1st Party) opt-in gating for custom cloud certificate retrieval in AKS. The change modifies the init-aks-custom-cloud.sh script to check whether a VM is opted into RCV 1P via a new wireserver endpoint (/acms/isOptedInForRootCerts). When opted in, the script uses new operation-request endpoints to pull root and intermediate certificates; otherwise, it falls back to the legacy cacertificates flow.

Changes:

Added RCV 1P opt-in detection via wireserver endpoint with JSON boolean matching
Implemented new certificate retrieval path using operationrequestsroot and operationrequestsintermediate endpoints
Preserved backward compatibility with legacy certificate retrieval for non-opted-in VMs
Normalized Flatcar certificate installation by converting .crt files to .pem during copy operation

parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh

Copilot

Pull request overview

Copilot reviewed 23 out of 24 changed files in this pull request and generated 4 comments.

parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh

parts/windows/kuberneteswindowssetup.ps1

parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh

Copilot

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated 3 comments.

You can also share your feedback on Copilot code review. Take the survey.

Copilot · 2026-03-03T22:23:53Z

parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh

+            # Save the certificate to the appropriate location
+            echo "$cert_content" > "/root/AzureCACertificates/$cert_filename"
+            echo "Successfully saved certificate: $cert_filename"


In the opted-in (operation-requests) path, cert filenames are saved exactly as returned (e.g., .cer/.crt/.pem), but the install step later only copies /root/AzureCACertificates/*.crt. If WireServer returns .cer (as the legacy flow suggests), no files will match and no certs will be installed. Normalize the saved filenames/extensions (e.g., always write .crt on Ubuntu/Mariner and handle Flatcar separately) or broaden the install glob to include the actual extensions returned.

Suggested change

# Save the certificate to the appropriate location

echo "$cert_content" > "/root/AzureCACertificates/$cert_filename"

echo "Successfully saved certificate: $cert_filename"

# Determine the filename to use when saving the certificate.

# On non-Flatcar systems, normalize to a .crt extension so that later

# install steps that use /root/AzureCACertificates/*.crt will pick it up.

local save_filename

if [ "$IS_FLATCAR" -eq 1 ]; then

save_filename="$cert_filename"

else

save_filename="${filename}.crt"

fi

# Save the certificate to the appropriate location

echo "$cert_content" > "/root/AzureCACertificates/$save_filename"

echo "Successfully saved certificate: $save_filename (original: $cert_filename)"

parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh

Copilot · 2026-03-03T22:23:53Z

parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh

+    for cert in /root/AzureCACertificates/*.crt; do
+        destcert="${cert##*/}"
+        destcert="${destcert%.*}.pem"
+        cp "$cert" /etc/ssl/certs/"$destcert"
+    done


Flatcar install loop iterates over /root/AzureCACertificates/*.crt, but the non-opted-in flow writes .pem files (and the opted-in flow may write .cer). This loop will no-op (and then run update-ca-certificates) if no .crt files exist. Align the Flatcar copy/convert logic with the actual extensions produced by both flows (e.g., convert whatever was downloaded into .pem during the copy step, or standardize downloads to .crt).

parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh

cameronmeissner · 2026-03-04T20:15:14Z

parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh

+}
+
+# Function to process certificate operations from a given endpoint
+process_cert_operations() {


nit: wondering if we can actually rename this to something more meaningful, process_cert_operations was originally chosen since we were trying to get things done quickly

parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh

cameronmeissner · 2026-03-04T20:18:43Z

parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh

+else
+    echo "Not opted in for root certs, skipping CA cert pull and install"
+    # http://168.63.129.16 is a constant for the host's wireserver endpoint
+    certs=$(curl "http://168.63.129.16/machine?comp=acmspackage&type=cacertificates&ext=json")


just to clarify - in the opt-out (or not opt-in) case, we download certificates from a different wireserver endpoint, so in either case we're grabbing certs from wireserver? I guess up until now this logic has only ever been executed in custom clouds, which makes sense

though now that this would be running in all clouds, this should this always be executed?

rchincha · 2026-03-04T23:02:30Z

@microsoft-github-policy-service agree company="Microsoft"

Copilot

Pull request overview

Copilot reviewed 36 out of 40 changed files in this pull request and generated 2 comments.

You can also share your feedback on Copilot code review. Take the survey.

Copilot · 2026-03-05T23:01:10Z

parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh

+        echo "Clearing existing CA trust store to ensure only certs from wireserver are trusted"
+        rm -rf /etc/ssl/certs/*
+    fi
    cp /root/AzureCACertificates/*.pem /etc/ssl/certs/


In the opted-in RCV 1P path, operation-request downloads are saved using the server-provided filename (very likely .crt). Flatcar installation here copies only *.pem, so a Flatcar node that is opted-in can end up installing zero certificates. Mirror the prior behavior by converting/copying .crt artifacts to .pem during the Flatcar copy step (or copy both extensions).

Suggested change

cp /root/AzureCACertificates/*.pem /etc/ssl/certs/

if ls /root/AzureCACertificates/*.pem >/dev/null 2>&1; then

cp /root/AzureCACertificates/*.pem /etc/ssl/certs/

fi

if ls /root/AzureCACertificates/*.crt >/dev/null 2>&1; then

cp /root/AzureCACertificates/*.crt /etc/ssl/certs/

fi

parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh

- add opt-in check via acms/isOptedInForRootCerts before certificate retrieval - use operation-request endpoints for RCV 1P root/intermediate certificate pull - keep fallback to legacy cacertificates flow when not opted in - harden opt-in detection with curl --fail and JSON boolean match - normalize Flatcar cert installation by converting .crt artifacts to .pem during copy

- add a dedicated CARefresh parameter set with -CARefreshOnly mode for certificate-only execution - implement RCV 1P cert retrieval from WireServer, including opt-in detection via isOptedInForRootCerts - add operation-request certificate download path (operationrequestsroot/operationrequestsintermediate) - keep backward-compatible fallback to legacy cacertificates endpoint when VM is not opted in - add retry/backoff wrapper for WireServer calls and structured RCV1P logging helper - install downloaded certs into LocalMachine certificate stores (Root for self-signed, CA for intermediates) - register a daily SYSTEM scheduled task (aks-rcv1p-cert-refresh) to rerun cert refresh via this script - wire AKS custom cloud base-prep flow to run RCV 1P refresh and task registration - document RCV 1P intent with inline comment block for maintainability

…t initialization and updating certificate handling

…d Mariner

…ted logic

…and system configuration (incoming from main branch)

…t-aks-custom-cloud script

…ion and hardening CA trust store

…ud script

…ng in init-aks-custom-cloud script

…ud script

…cert_operations function

…ustom-cloud script

…aks-custom-cloud script

Copilot AI review requested due to automatic review settings February 26, 2026 01:13

rchincha temporarily deployed to test February 26, 2026 01:13 — with GitHub Actions Inactive

Copilot started reviewing on behalf of rchincha February 26, 2026 01:14 View session

Copilot AI reviewed Feb 26, 2026

View reviewed changes

rchincha temporarily deployed to test February 26, 2026 02:19 — with GitHub Actions Inactive

Copilot AI review requested due to automatic review settings March 3, 2026 17:55

Copilot started reviewing on behalf of rchincha March 3, 2026 17:55 View session

Copilot AI reviewed Mar 3, 2026

View reviewed changes

rchincha force-pushed the origin/rchinchani/rcv1p branch from 9677792 to 31145fe Compare March 3, 2026 21:44

rchincha had a problem deploying to test March 3, 2026 21:44 — with GitHub Actions Failure

Copilot AI review requested due to automatic review settings March 3, 2026 22:17

rchincha temporarily deployed to test March 3, 2026 22:17 — with GitHub Actions Inactive

Copilot started reviewing on behalf of rchincha March 3, 2026 22:18 View session

Copilot AI reviewed Mar 3, 2026

View reviewed changes

rchincha force-pushed the origin/rchinchani/rcv1p branch from aa14963 to 40fa5ed Compare March 4, 2026 17:01

rchincha temporarily deployed to test March 4, 2026 17:01 — with GitHub Actions Inactive

cameronmeissner reviewed Mar 4, 2026

View reviewed changes

parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh Outdated Show resolved Hide resolved

cameronmeissner reviewed Mar 4, 2026

View reviewed changes

parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh Outdated Show resolved Hide resolved

cameronmeissner reviewed Mar 4, 2026

View reviewed changes

parts/linux/cloud-init/artifacts/init-aks-custom-cloud.sh Outdated Show resolved Hide resolved

cameronmeissner reviewed Mar 4, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings March 5, 2026 22:53

rchincha force-pushed the origin/rchinchani/rcv1p branch from 40fa5ed to 7c77e4e Compare March 5, 2026 22:53

Copilot AI reviewed Mar 5, 2026

View reviewed changes

rchincha temporarily deployed to test March 5, 2026 23:03 — with GitHub Actions Inactive

Copilot started reviewing on behalf of rchincha March 5, 2026 23:30 View session

rchincha changed the title ~~feat(linux): gate custom cloud cert pull on RCV 1P opt-in~~ feat: gate custom cloud cert pull on RCV 1P opt-in Mar 9, 2026

Ramkumar Chinchani added 2 commits March 9, 2026 13:47

Ramkumar Chinchani added 20 commits March 9, 2026 13:48

fix: update testdata

67cd54a

feat: implement retry logic for HTTP requests in RCV 1P cert retrieval

7d46547

feat: add directory creation for RCV1P certificates if it doesn't exist

b0f4832

fix: update testdata

c54aeb2

feat: enhance support for Mariner and Azure Linux by adding repo depo…

01245cc

…t initialization and updating certificate handling

refactor: simplify initAKSCustomCloud logic for Azure Linux Distro an…

2805e5d

…d Mariner

refactor: remove deprecated AKS custom cloud initialization scripts

a55e3f8

fix: align certificate handling for Mariner and Azure Linux with upda…

35ba7c6

…ted logic

feat: add support for ACL Linux distribution in certificate handling …

807f9ef

…and system configuration (incoming from main branch)

fix: correct condition order for CA trust store initialization in ini…

8729b45

…t-aks-custom-cloud script

feat: enhance certificate handling by adding action-based initializat…

042a171

…ion and hardening CA trust store

fix: correct syntax for conditional statements in init-aks-custom-clo…

988af2a

…ud script

fix: update compressed content in CustomData for MarinerV2+CustomCloud

ddf908c

fix: improve certificate handling by using printf for better formatti…

a2192f5

…ng in init-aks-custom-cloud script

fix: correct syntax for conditional statements in init-aks-custom-clo…

d1e7c29

…ud script

fix: clarify intentional typo in ResourceFileName comment in process_…

12efe6e

…cert_operations function

fix: simplify grep usage for checking root cert opt-in status

4f3ef4a

fix: remove redundant comment about wireserver endpoint in init-aks-c…

afb4f6d

…ustom-cloud script

fix: correct extension replacement for certificate filenames in init-…

28ff700

…aks-custom-cloud script

fix: update compressed content in CustomData for MarinerV2+CustomCloud

5b34df8

rchincha force-pushed the origin/rchinchani/rcv1p branch from 7c77e4e to 5b34df8 Compare March 9, 2026 21:02

rchincha temporarily deployed to test March 9, 2026 21:02 — with GitHub Actions Inactive

-            # Save the certificate to the appropriate location
-            echo "$cert_content" > "/root/AzureCACertificates/$cert_filename"
-            echo "Successfully saved certificate: $cert_filename"
+            # Determine the filename to use when saving the certificate.
+            # On non-Flatcar systems, normalize to a .crt extension so that later
+            # install steps that use /root/AzureCACertificates/*.crt will pick it up.
+            local save_filename
+            if [ "$IS_FLATCAR" -eq 1 ]; then
+                save_filename="$cert_filename"
+            else
+                save_filename="${filename}.crt"
+            fi
+            # Save the certificate to the appropriate location
+            echo "$cert_content" > "/root/AzureCACertificates/$save_filename"
+            echo "Successfully saved certificate: $save_filename (original: $cert_filename)"

-    cp /root/AzureCACertificates/*.pem /etc/ssl/certs/
+    if ls /root/AzureCACertificates/*.pem >/dev/null 2>&1; then
+        cp /root/AzureCACertificates/*.pem /etc/ssl/certs/
+    fi
+    if ls /root/AzureCACertificates/*.crt >/dev/null 2>&1; then
+        cp /root/AzureCACertificates/*.crt /etc/ssl/certs/
+    fi

Conversation

rchincha commented Feb 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cameronmeissner Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cameronmeissner Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

rchincha commented Mar 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants