experiment: [WIP] add ad-hoc testing integration and configs w/o edits to other packages.

[chalmerlowe]

WIP as an experiment.

[gemini-code-assist]

Code Review

This pull request introduces an ad-hoc testing integration that allows running CI tests for a specific subset of packages or predefined groups based on a GitHub PR label. It adds configuration files, a runner script, and documentation under ci/adhoc/, and updates .kokoro/system.sh to parse these configurations and execute the tests in parallel using null-terminated arguments. The review feedback highlights three key improvement opportunities: handling empty package arrays to prevent xargs from executing with an empty argument, improving curl error handling to gracefully capture network connection failures, and stripping carriage returns from configuration files to ensure cross-platform compatibility.

[gemini-code-assist]

When ${PACKAGES_TO_TEST[@]} is empty, printf '%s\0' will still output a single null byte (\0). Because xargs -0 treats null bytes as delimiters, it will interpret this as a single empty string argument ("") and execute the bash command once with pkg="". This can lead to unexpected behavior or failures (e.g., trying to run run_package_test "").

To prevent this, we can short-circuit the pipeline if the array is empty.

Suggested change

	printf '%s\0' "${PACKAGES_TO_TEST[@]}" \
	| xargs -0 -n 1 -P "$MAX_JOBS" \
	[ ${#PACKAGES_TO_TEST[@]} -eq 0 ] || printf '%s\\0' "${PACKAGES_TO_TEST[@]}" \\
	| xargs -0 -n 1 -P "$MAX_JOBS" \\

[gemini-code-assist]

Using || echo "[]" on the curl command masks any network or connection failures. If curl fails completely (e.g., due to DNS issues or network timeout), LABELS_JSON becomes [], and the script will silently report that the test:adhoc label was not found instead of warning the user about the connection failure.

We can handle connection failures gracefully by checking the exit status of the curl command directly.

    # Fetch PR labels from GitHub API, handling connection failures gracefully
    if ! LABELS_JSON=$(curl -s "${headers[@]}" "https://api.github.com/repos/googleapis/google-cloud-python/issues/${KOKORO_GITHUB_PULL_REQUEST_NUMBER}/labels"); then
        echo "==============================================================="
        echo "WARNING: Failed to connect to GitHub API!"
        echo "Ad-hoc tests will NOT be triggered."
        echo "==============================================================="
    else
        # Use jq to parse github labels (works as long as jq is available in python-multi image).
        IS_ADHOC=$(echo "$LABELS_JSON" | jq -r 'if type == "array" then any(.name == "test:adhoc") else false end' 2>/dev/null)

        if [[ "$IS_ADHOC" == "true" ]]; then
            TRIGGER_ADHOC="true"
            echo "Adhoc test label 'test:adhoc' found!"
        else
            if [[ "$LABELS_JSON" != "["* ]]; then
                API_ERR_MSG=$(echo "$LABELS_JSON" | jq -r '.message // "Unknown error"' 2>/dev/null)
                echo "==============================================================="
                echo "WARNING: Failed to fetch PR labels from GitHub API!"
                echo "Error Message: $API_ERR_MSG"
                echo "This might be due to API Rate Limiting."
                echo "Ad-hoc tests will NOT be triggered."
                echo "==============================================================="
            else
                echo "Adhoc test label 'test:adhoc' not found."
            fi
        fi
    fi

[gemini-code-assist]

If the configuration files .standalone_package_list.txt or .package_groups.txt are edited on Windows, they may contain carriage return characters (\r). This will cause the extracted package or group names to have a trailing \r, which will prevent the script from finding the correct package directories.

We should strip carriage returns using tr -d '\r' to ensure cross-platform compatibility.

Suggested change

	adhoc_packages=$(grep "^package:" "$STANDALONE_LIST" | cut -d':' -f2 | xargs || true)
	# Grab requested groups
	requested_groups=$(grep "^group:" "$STANDALONE_LIST" | cut -d':' -f2 | xargs || true)
	# Expand groups
	for group in $requested_groups; do
	group_pkgs=$(grep "^$group:" "$GROUPS_FILE" | cut -d':' -f2 | xargs || true)
	adhoc_packages="$adhoc_packages $group_pkgs"
	done
	adhoc_packages=$(grep "^package:" "$STANDALONE_LIST" | cut -d':' -f2 | tr -d '\\r' | xargs || true)
	# Grab requested groups
	requested_groups=$(grep "^group:" "$STANDALONE_LIST" | cut -d':' -f2 | tr -d '\\r' | xargs || true)
	# Expand groups
	for group in $requested_groups; do
	group_pkgs=$(grep "^$group:" "$GROUPS_FILE" | cut -d':' -f2 | tr -d '\\r' | xargs || true)
	adhoc_packages="$adhoc_packages $group_pkgs"
	done