Skip to content

aro-hcp: add app registration cleanup prow job #77621

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bennerv wants to merge 1 commit into
base: main
Choose a base branch
from
Open

aro-hcp: add app registration cleanup prow job #77621

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions ci-operator/config/Azure/ARO-HCP/Azure-ARO-HCP-main__periodic.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,28 @@ tests:
steps:
test:
- ref: aro-hcp-deprovision-kusto-role-assignments
- as: delete-expired-red-hat-tenant-app-registrations
cron: 0 5 * * *
steps:
env:
VAULT_SECRET_PROFILE: red-hat-tenant
leases:
- count: 1
env: ENV_QUOTA_LEASED_RESOURCE
resource_type: aro-hcp-red-hat-tenant-quota-slice
test:
- ref: aro-hcp-deprovision-expired-app-registrations
- as: delete-expired-msft-test-tenant-app-registrations
cron: 0 5 * * *
steps:
env:
VAULT_SECRET_PROFILE: msft-test-tenant
leases:
- count: 1
env: ENV_QUOTA_LEASED_RESOURCE
resource_type: aro-hcp-msft-test-tenant-quota-slice
test:
- ref: aro-hcp-deprovision-expired-app-registrations
- as: delete-expired-development-resource-groups
cron: 7,37 * * * *
steps:
Expand Down
146 changes: 146 additions & 0 deletions ci-operator/jobs/Azure/ARO-HCP/Azure-ARO-HCP-main-periodics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -307,6 +307,79 @@ periodics:
- name: result-aggregator
secret:
secretName: result-aggregator
- agent: kubernetes
cluster: build09
cron: 0 5 * * *
decorate: true
decoration_config:
skip_cloning: true
extra_refs:
- base_ref: main
org: Azure
repo: ARO-HCP
labels:
ci-operator.openshift.io/variant: periodic
ci.openshift.io/generator: prowgen
pj-rehearse.openshift.io/can-be-rehearsed: "true"
name: periodic-ci-Azure-ARO-HCP-main-periodic-delete-expired-msft-test-tenant-app-registrations
spec:
containers:
- args:
- --gcs-upload-secret=/secrets/gcs/service-account.json
- --image-import-pull-secret=/etc/pull-secret/.dockerconfigjson
- --lease-server-credentials-file=/etc/boskos/credentials
- --report-credentials-file=/etc/report/credentials
- --target=delete-expired-msft-test-tenant-app-registrations
- --variant=periodic
command:
- ci-operator
env:
- name: HTTP_SERVER_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
image: quay-proxy.ci.openshift.org/openshift/ci:ci_ci-operator_latest
imagePullPolicy: Always
name: ""
ports:
- containerPort: 8080
name: http
resources:
requests:
cpu: 10m
volumeMounts:
- mountPath: /etc/boskos
name: boskos
readOnly: true
- mountPath: /secrets/gcs
name: gcs-credentials
readOnly: true
- mountPath: /secrets/manifest-tool
name: manifest-tool-local-pusher
readOnly: true
- mountPath: /etc/pull-secret
name: pull-secret
readOnly: true
- mountPath: /etc/report
name: result-aggregator
readOnly: true
serviceAccountName: ci-operator
volumes:
- name: boskos
secret:
items:
- key: credentials
path: credentials
secretName: boskos-credentials
- name: manifest-tool-local-pusher
secret:
secretName: manifest-tool-local-pusher
- name: pull-secret
secret:
secretName: registry-pull-credentials
- name: result-aggregator
secret:
secretName: result-aggregator
- agent: kubernetes
cluster: build09
cron: 7,37 * * * *
Expand Down Expand Up @@ -388,6 +461,79 @@ periodics:
- name: result-aggregator
secret:
secretName: result-aggregator
- agent: kubernetes
cluster: build09
cron: 0 5 * * *
decorate: true
decoration_config:
skip_cloning: true
extra_refs:
- base_ref: main
org: Azure
repo: ARO-HCP
labels:
ci-operator.openshift.io/variant: periodic
ci.openshift.io/generator: prowgen
pj-rehearse.openshift.io/can-be-rehearsed: "true"
name: periodic-ci-Azure-ARO-HCP-main-periodic-delete-expired-red-hat-tenant-app-registrations
spec:
containers:
- args:
- --gcs-upload-secret=/secrets/gcs/service-account.json
- --image-import-pull-secret=/etc/pull-secret/.dockerconfigjson
- --lease-server-credentials-file=/etc/boskos/credentials
- --report-credentials-file=/etc/report/credentials
- --target=delete-expired-red-hat-tenant-app-registrations
- --variant=periodic
command:
- ci-operator
env:
- name: HTTP_SERVER_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
image: quay-proxy.ci.openshift.org/openshift/ci:ci_ci-operator_latest
imagePullPolicy: Always
name: ""
ports:
- containerPort: 8080
name: http
resources:
requests:
cpu: 10m
volumeMounts:
- mountPath: /etc/boskos
name: boskos
readOnly: true
- mountPath: /secrets/gcs
name: gcs-credentials
readOnly: true
- mountPath: /secrets/manifest-tool
name: manifest-tool-local-pusher
readOnly: true
- mountPath: /etc/pull-secret
name: pull-secret
readOnly: true
- mountPath: /etc/report
name: result-aggregator
readOnly: true
serviceAccountName: ci-operator
volumes:
- name: boskos
secret:
items:
- key: credentials
path: credentials
secretName: boskos-credentials
- name: manifest-tool-local-pusher
secret:
secretName: manifest-tool-local-pusher
- name: pull-secret
secret:
secretName: registry-pull-credentials
- name: result-aggregator
secret:
secretName: result-aggregator
- agent: kubernetes
cluster: build09
cron: 7,37 * * * *
Expand Down
14 changes: 14 additions & 0 deletions ci-operator/step-registry/aro-hcp/deprovision/expired-app-registrations/OWNERS
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
approvers:
- geoberle
- jharrington22
- mmazur
- roivaz
- venkateshsredhat
- deads2k
reviewers:
- geoberle
- jharrington22
- mmazur
- roivaz
- venkateshsredhat
- deads2k
16 changes: 16 additions & 0 deletions ...ision/expired-app-registrations/aro-hcp-deprovision-expired-app-registrations-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#!/bin/bash
set -o errexit
set -o nounset
set -o pipefail

export CLUSTER_PROFILE_DIR="/var/run/aro-hcp-${VAULT_SECRET_PROFILE}"

# Disable tracing due to credential handling
export AZURE_CLIENT_ID; AZURE_CLIENT_ID=$(cat "${CLUSTER_PROFILE_DIR}/client-id")
export AZURE_TENANT_ID; AZURE_TENANT_ID=$(cat "${CLUSTER_PROFILE_DIR}/tenant")
export AZURE_CLIENT_SECRET; AZURE_CLIENT_SECRET=$(cat "${CLUSTER_PROFILE_DIR}/client-secret")
export AZURE_TOKEN_CREDENTIALS=prod

set -o xtrace

./test/aro-hcp-tests cleanup app-registrations
21 changes: 21 additions & 0 deletions ...expired-app-registrations/aro-hcp-deprovision-expired-app-registrations-ref.metadata.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"path": "aro-hcp/deprovision/expired-app-registrations/aro-hcp-deprovision-expired-app-registrations-ref.yaml",
"owners": {
"approvers": [
"geoberle",
"jharrington22",
"mmazur",
"roivaz",
"venkateshsredhat",
"deads2k"
],
"reviewers": [
"geoberle",
"jharrington22",
"mmazur",
"roivaz",
"venkateshsredhat",
"deads2k"
]
}
}
22 changes: 22 additions & 0 deletions ...rovision/expired-app-registrations/aro-hcp-deprovision-expired-app-registrations-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
ref:
as: aro-hcp-deprovision-expired-app-registrations
from: aro-hcp-e2e-tools
commands: aro-hcp-deprovision-expired-app-registrations-commands.sh
resources:
requests:
cpu: 100m
memory: 300Mi
credentials:
- namespace: test-credentials
name: cluster-secrets-aro-hcp-red-hat-tenant
mount_path: /var/run/aro-hcp-red-hat-tenant
- namespace: test-credentials
name: cluster-secrets-aro-hcp-msft-test-tenant
mount_path: /var/run/aro-hcp-msft-test-tenant
Comment on lines +9 to +15
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Reduce credential blast radius: do not mount both tenant secrets in one step.

Line 9-15 mounts both cluster-secrets-aro-hcp-red-hat-tenant and cluster-secrets-aro-hcp-msft-test-tenant for every run. This over-privileges each job and weakens tenant isolation. Split this into tenant-specific refs (or equivalent wiring) so each periodic job mounts only its required secret.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In
`@ci-operator/step-registry/aro-hcp/deprovision/expired-app-registrations/aro-hcp-deprovision-expired-app-registrations-ref.yaml`
around lines 9 - 15, The current credentials block mounts both tenant secrets
(cluster-secrets-aro-hcp-red-hat-tenant and
cluster-secrets-aro-hcp-msft-test-tenant) into every run, over-privileging jobs;
change the step so each periodic job only mounts its required secret by
splitting the single credentials list into tenant-specific credential refs
(e.g., create separate steps or variant refs that include only
cluster-secrets-aro-hcp-red-hat-tenant with mount_path
/var/run/aro-hcp-red-hat-tenant for Red Hat jobs and only
cluster-secrets-aro-hcp-msft-test-tenant with mount_path
/var/run/aro-hcp-msft-test-tenant for MSFT jobs), and wire the appropriate
step/ref into the corresponding periodic job definitions so no job mounts the
other tenant's secret.

env:
- name: VAULT_SECRET_PROFILE
default: "red-hat-tenant"
documentation: |-
Selects which environment's cluster secrets to use (red-hat-tenant, msft-test-tenant).
documentation: |-
Clean up expired e2e app registrations that were left by test runs.
8 changes: 8 additions & 0 deletions core-services/prow/02_config/_boskos.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,10 @@ resources:
min-count: 1
state: free
type: aro-hcp-int-quota-slice
- max-count: 1
min-count: 1
state: free
type: aro-hcp-msft-test-tenant-quota-slice
- names:
- aro-hcp-msi-mock-cs-sp-dev-0
- aro-hcp-msi-mock-cs-sp-dev-1
Expand Down Expand Up @@ -94,6 +98,10 @@ resources:
min-count: 10
state: free
type: aro-hcp-prod-quota-slice
- max-count: 1
min-count: 1
state: free
type: aro-hcp-red-hat-tenant-quota-slice
- max-count: 1
min-count: 1
state: free
Expand Down
6 changes: 6 additions & 0 deletions core-services/prow/02_config/generate-boskos.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -291,6 +291,12 @@
'aro-hcp-dev-global-pipeline-quota-slice': {
'default': 1,
},
'aro-hcp-red-hat-tenant-quota-slice': {
'default': 1,
},
'aro-hcp-msft-test-tenant-quota-slice': {
'default': 1,
},
'aro-hcp-test-msi-containers-dev': {},
'aro-hcp-test-msi-containers-int': {},
'aro-hcp-test-msi-containers-stg': {},
Expand Down