test(monorepo): adds adhoc package selection for CI system tests - #17931
test(monorepo): adds adhoc package selection for CI system tests#17931chalmerlowe wants to merge 25 commits into
Conversation
chalmerlowe
changed the title
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces an ad-hoc testing integration into the Kokoro CI pipeline, allowing dynamic package selection based on PR labels and configuration files. The review feedback highlights several crucial robustness improvements, including handling potential grep failures under set -o pipefail in the ad-hoc test runner, authenticating GitHub API requests to prevent rate-limiting, safely parsing the API response in the inline Python snippet, and quoting variables to avoid pathname expansion.
chalmerlowe
added
test:adhoc
yoshi-kokoro
removed
the
kokoro:force-run
bcb4c15 to
d376408
Compare
chalmerlowe
changed the title
chalmerlowe
added
the
do not merge
|
Changes were made to two packages to ensure that the combination and deduplication of diff'ed packages and ad hoc packages functioned as expected. |
| headers=(-H "User-Agent: Kokoro") | ||
| if [[ -n "${GITHUB_TOKEN:-${GH_TOKEN}}" ]]; then | ||
| headers+=(-H "Authorization: token ${GITHUB_TOKEN:-${GH_TOKEN}}") | ||
| fi |
There was a problem hiding this comment.
Is the GitHub token already being populated in kokoro? Or is this a new secret we need to add for this feature?
There was a problem hiding this comment.
GITHUB_TOKEN is not yet populated in kokoro but kokoro is set up to inject them if/when any GITHUB tokens are provided in SECRET MANAGER in Google Cloud Console.
This PR and the ad hoc label succeeds because we are currently relying on the fact that GitHub allows 60 unauthorized API calls from a single IP per hour.
We have no way to know if others in the firm will be in a kokoro instance behind the same IP and calling github. So it in our long-term best interest to add the token to SECRET MANAGER. Completion of that task is not required for this PR to be merged.
This Issue tracks this limitation.
:EX:
| # For this prototype: | ||
| # we use a small inline Python snippet here because parsing JSON in pure Bash is difficult/error-prone, | ||
| # and we cannot guarantee that tools like 'jq' or 'gh' are installed in the test environment. | ||
| # Python and its built-in 'json' module are guaranteed to be available in this repository. |
There was a problem hiding this comment.
we cannot guarantee that tools like 'jq' or 'gh' are installed in the test environment.
Are you sure? It looks like kokoro is using the gcr.io/cloud-devrel-kokoro-resources/python-multi image. I just tested it locally, and it seems to have jq available
There was a problem hiding this comment.
:RESOLVED:
Removed the python snippet and replaced with BASH command using jq.
…l parameter confusion
f75555e to
d10e510
Compare
chalmerlowe
added
automerge
|
All misc dummy edits have been removed now that the internal kokoro config has been updated. |
parthea
removed
the
automerge
|
Switching to draft based on offline discussion |
4 participants
Problem
Currently, there is no easy way to run tests for specific packages in our continuous integration (CI) pipeline without making temporary changes to those packages (like adding a space to a comment or inserting a blank line) to trigger detection of a
diff. This causes noise in commit history and extra toil to revert the changes used solely for triggering tests prior to merge.Solution
We have implemented an "Ad-Hoc Package Selection" feature for
systemtests. This allows developers to specify which packages or groups of packages they want to test by editing simple configuration files within their Pull Request (PR). The feature is triggered by applying a specific GitHub label (test:adhoc) to the PR.A new script parses these configuration files and merges the ad hoc packages with those that have actual code diffs, ensuring that no modified packages are missed.
Changes
ci/adhoc/.standalone_package_list.txtfor listing individual packagesci/adhoc/.package_groups.txtfor defining groups of packages (e.g.,handwritten,core`).ci/adhoc/adhoc_test_runner.shscript to parse and deduplicate requested packages..kokoro/system.shto integrate this ad-hoc logic when thetest:adhoclabel is present.Note
Notes to Reviewers
ci/adhoc/are intended to be modified in your PR to select target packages and DO NOT require cleanup before merging.google-resumable-mediaand a trigger comment ingoogle-cloud-speech) strictly for verifying the ad-hoc mechanism works with diff'ed packages and to confirm output formatting in successful and failed situations. These will be reverted before final merge.