Skip to content

ci: avoid running split ITs unexpectedly on unrelated changes (#12029) - #14217

Merged
lqiu96 merged 1 commit into
googleapis:mainfrom
lqiu96:fix-split-it-module-detection
Sep 1, 2026
Merged

ci: avoid running split ITs unexpectedly on unrelated changes (#12029)#14217
lqiu96 merged 1 commit into
googleapis:mainfrom
lqiu96:fix-split-it-module-detection

Conversation

@lqiu96

@lqiu96 lqiu96 commented Aug 31, 2026

Copy link
Copy Markdown
Member

Fixes #12029 (b/487770623)

This PR is part 1 of 2 in a stacked series:

  1. This PR (ci: avoid running split ITs unexpectedly on unrelated changes (#12029) #14217): fix: avoid running split ITs unexpectedly on unrelated changes (#12029)
  2. perf(ci): replace mvn help:evaluate with native bash and sed extraction #14218: perf(ci): replace mvn help:evaluate with native bash and sed extraction

Problem

In split integration and GraalVM tests (e.g. integration-single and graalvm-single), module modification detection previously checked if the module name was present as a substring in modified_module_list. For modules with shared prefixes (such as java-bigquery vs java-bigquerystorage, java-bigquery-jdbc, or java-bigqueryconnection), modifying one module could trigger tests for other prefix-sharing modules.

Changes

  • Module Boundary Anchoring (is_module_modified): Added is_module_modified in .kokoro/common.sh using exact directory prefix matching (^${module}/).
  • Global Overrides: is_module_modified respects parent POM modifications (google-cloud-(pom|jar)-parent/pom.xml), shared dependency modifications (sdk-platform-java/java-shared-dependencies), and TEST_ALL_MODULES="true" so that dependency and parent updates properly verify downstream integration suites.
  • Short-Circuit in Split Jobs: Updated integration-single and graalvm-single in .kokoro/build.sh to check is_module_modified "${BUILD_SUBDIR}" directly.
  • Updated Shared Dependencies Path: Updated shared_dependencies_modified in .kokoro/common.sh to match sdk-platform-java/java-shared-dependencies.
  • Unit Tests: Added unit test coverage in .kokoro/common_test.sh for is_module_modified covering empty inputs, prefix collision prevention, parent pom modification, shared dependencies modification, and TEST_ALL_MODULES.

@lqiu96
lqiu96 requested review from a team as code owners August 31, 2026 15:19
@lqiu96 lqiu96 changed the title fix: avoid running split ITs unexpectedly on unrelated changes ci: avoid running split ITs unexpectedly on unrelated changes Aug 31, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request optimizes Kokoro CI scripts by replacing slow Maven evaluation commands (mvn help:evaluate) with fast, native Bash and sed parsing of pom.xml files, significantly reducing CI boot times. It also introduces a new is_module_modified helper to check for module-specific changes. Feedback highlights a critical bug where is_module_modified bypasses global override checks (such as parent POM changes or TEST_ALL_MODULES), which would cause tests to be skipped when they should run. Additionally, improvements are suggested to make the unit tests more robust by resolving paths relative to the script's directory instead of relying on the current working directory.

Comment thread .kokoro/common.sh
Comment thread .kokoro/common_test.sh Outdated
Comment thread .kokoro/common_test.sh Outdated
@lqiu96
lqiu96 force-pushed the fix-split-it-module-detection branch from 42ce494 to c66781c Compare August 31, 2026 15:35
@lqiu96 lqiu96 changed the title ci: avoid running split ITs unexpectedly on unrelated changes fix: avoid running split ITs unexpectedly on unrelated changes (#12029) Aug 31, 2026
lqiu96 added a commit to lqiu96/google-cloud-java that referenced this pull request Aug 31, 2026
This PR is part 2 of 2 in a stacked series:
1. googleapis#14217: fix: avoid running split ITs unexpectedly on unrelated changes (googleapis#12029)
2. This PR: perf(ci): replace mvn help:evaluate with native bash and sed extraction

## Problem
In Kokoro CI scripts, mvn help:evaluate was used to parse POM values:
- In generate_modified_modules_list (.kokoro/common.sh), evaluating
  project.modules launched a full JVM and evaluated the monorepo POMs,
  taking 20–30+ seconds on every single CI run.
- In downstream-build.sh (.kokoro/presubmit/downstream-build.sh),
  evaluating gapic-showcase.version launched another Maven JVM process.

## Changes
- Pure-Bash Module Extraction: Updated generate_modified_modules_list to use
  extract_pom_modules pom.xml, extracting all 270 modules in ~0.02s without JVM
  boot overhead.
- Sed Showcase Version Extraction: Updated downstream-build.sh to parse
  <gapic-showcase.version> directly using sed with fail-fast validation.
- Unit Tests: Added automated unit tests in .kokoro/common_test.sh for
  extract_pom_modules and generate_modified_modules_list.
- Refactoring & Best Practices: Scoped all temporary variables as local,
  used read -r module <<< "${module}" for pure-bash whitespace trimming,
  and anchored paths with ${scriptDir}/..
@lqiu96
lqiu96 force-pushed the fix-split-it-module-detection branch from c66781c to 4a6a76b Compare August 31, 2026 15:53
lqiu96 added a commit to lqiu96/google-cloud-java that referenced this pull request Aug 31, 2026
This PR is part 2 of 2 in a stacked series:
1. googleapis#14217: fix: avoid running split ITs unexpectedly on unrelated changes (googleapis#12029)
2. This PR: perf(ci): replace mvn help:evaluate with native bash and sed extraction

In Kokoro CI scripts, mvn help:evaluate was used to parse POM values:
- In generate_modified_modules_list (.kokoro/common.sh), evaluating
  project.modules launched a full JVM and evaluated the monorepo POMs,
  taking 20–30+ seconds on every single CI run.
- In downstream-build.sh (.kokoro/presubmit/downstream-build.sh),
  evaluating gapic-showcase.version launched another Maven JVM process.

- Pure-Bash Module Extraction: Updated generate_modified_modules_list to use
  extract_pom_modules pom.xml, extracting all 270 modules in ~0.02s without JVM
  boot overhead.
- Sed Showcase Version Extraction: Updated downstream-build.sh to parse
  <gapic-showcase.version> directly using sed with fail-fast validation.
- Unit Tests: Added automated unit tests in .kokoro/common_test.sh for
  extract_pom_modules and generate_modified_modules_list.
- Refactoring & Best Practices: Scoped all temporary variables as local,
  used read -r module <<< "${module}" for pure-bash whitespace trimming,
  and anchored paths with ${scriptDir}/..
@lqiu96
lqiu96 force-pushed the fix-split-it-module-detection branch from 4a6a76b to a15b6a5 Compare August 31, 2026 16:02
lqiu96 added a commit to lqiu96/google-cloud-java that referenced this pull request Aug 31, 2026
This PR is part 2 of 2 in a stacked series:
1. googleapis#14217: fix: avoid running split ITs unexpectedly on unrelated changes (googleapis#12029)
2. This PR: perf(ci): replace mvn help:evaluate with native bash and sed extraction

In Kokoro CI scripts, mvn help:evaluate was used to parse POM values:
- In generate_modified_modules_list (.kokoro/common.sh), evaluating
  project.modules launched a full JVM and evaluated the monorepo POMs,
  taking 20–30+ seconds on every single CI run.
- In downstream-build.sh (.kokoro/presubmit/downstream-build.sh),
  evaluating gapic-showcase.version launched another Maven JVM process.

- Pure-Bash Module Extraction: Updated generate_modified_modules_list to use
  extract_pom_modules pom.xml, extracting all 270 modules in ~0.02s without JVM
  boot overhead.
- Sed Showcase Version Extraction: Updated downstream-build.sh to parse
  <gapic-showcase.version> directly using sed with fail-fast validation.
- Unit Tests: Added automated unit tests in .kokoro/common_test.sh for
  extract_pom_modules and generate_modified_modules_list.
- Refactoring & Best Practices: Scoped all temporary variables as local,
  used read -r module <<< "${module}" for pure-bash whitespace trimming,
  and anchored paths with ${scriptDir}/..
lqiu96 added a commit to lqiu96/google-cloud-java that referenced this pull request Aug 31, 2026
…eapis#12029)

This PR is part 1 of 2 in a stacked series:
1. This PR (googleapis#14217): fix: avoid running split ITs unexpectedly on unrelated changes (googleapis#12029)
2. googleapis#14218: perf(ci): replace mvn help:evaluate with native bash and sed extraction

## Problem
In split integration and GraalVM tests (e.g. integration-single and graalvm-single),
module modification detection previously checked if the module name was present as a
substring in modified_module_list. For modules with shared prefixes (such as
java-bigquery vs java-bigquerystorage, java-bigquery-jdbc, or java-bigqueryconnection),
modifying one module could trigger tests for other prefix-sharing modules.

## Changes
- **Modular Helper Functions with Single Responsibility**:
  - `populate_modified_files`: Encapsulates diff retrieval, caching in `modified_files`,
    and configuring git `safe.directory` for Docker container ownership mismatches.
  - `should_test_all_modules`: Purely checks global overrides that require testing all
    modules (parent POMs, core shared dependencies, and `TEST_ALL_MODULES="true"`).
  - `is_module_modified`: Strictly checks if files within a specific module directory
    were modified (`^${module}/`), avoiding prefix collisions.
- **Explicit Job Control in Split Jobs**:
  - In `.kokoro/build.sh` (`integration-single` and `graalvm-single`), runs tests if
    either monorepo-wide testing is required or the specific module was modified:
    `! should_test_all_modules && ! is_module_modified "${BUILD_SUBDIR}"` -> skip.
- **Updated generate_modified_modules_list**:
  - Reuses `populate_modified_files` and `should_test_all_modules`.
- **Unit Tests**:
  - Added unit tests in `.kokoro/common_test.sh` for `is_module_modified` and
    `should_test_all_modules`.

Fixes googleapis#12029
@lqiu96
lqiu96 force-pushed the fix-split-it-module-detection branch from a15b6a5 to 54fb41a Compare August 31, 2026 16:20
lqiu96 added a commit to lqiu96/google-cloud-java that referenced this pull request Aug 31, 2026
This PR is part 2 of 2 in a stacked series:
1. googleapis#14217: fix: avoid running split ITs unexpectedly on unrelated changes (googleapis#12029)
2. This PR: perf(ci): replace mvn help:evaluate with native bash and sed extraction

In Kokoro CI scripts, mvn help:evaluate was used to parse POM values:
- In generate_modified_modules_list (.kokoro/common.sh), evaluating
  project.modules launched a full JVM and evaluated the monorepo POMs,
  taking 20–30+ seconds on every single CI run.
- In downstream-build.sh (.kokoro/presubmit/downstream-build.sh),
  evaluating gapic-showcase.version launched another Maven JVM process.

- Pure-Bash Module Extraction: Updated generate_modified_modules_list to use
  extract_pom_modules pom.xml, extracting all 270 modules in ~0.02s without JVM
  boot overhead.
- Sed Showcase Version Extraction: Updated downstream-build.sh to parse
  <gapic-showcase.version> directly using sed with fail-fast validation.
- Unit Tests: Added automated unit tests in .kokoro/common_test.sh for
  extract_pom_modules and generate_modified_modules_list.
- Refactoring & Best Practices: Scoped all temporary variables as local,
  used read -r module <<< "${module}" for pure-bash whitespace trimming,
  and anchored paths with ${scriptDir}/..
lqiu96 added a commit to lqiu96/google-cloud-java that referenced this pull request Aug 31, 2026
…eapis#12029)

This PR is part 1 of 2 in a stacked series:
1. This PR (googleapis#14217): fix: avoid running split ITs unexpectedly on unrelated changes (googleapis#12029)
2. googleapis#14218: perf(ci): replace mvn help:evaluate with native bash and sed extraction

## Problem
In split integration and GraalVM tests (e.g. integration-single and graalvm-single),
module modification detection previously checked if the module name was present as a
substring in modified_module_list. For modules with shared prefixes (such as
java-bigquery vs java-bigquerystorage, java-bigquery-jdbc, or java-bigqueryconnection),
modifying one module could trigger tests for other prefix-sharing modules.

## Changes
- **Modular Helper Functions with Single Responsibility**:
  - `populate_modified_files`: Encapsulates diff retrieval, caching in `modified_files`,
    and configuring git `safe.directory` for Docker container ownership mismatches.
  - `should_test_all_modules`: Purely checks global overrides that require testing all
    modules (parent POMs, core shared dependencies, and `TEST_ALL_MODULES="true"`).
  - `is_module_modified`: Strictly checks if files within a specific module directory
    were modified (`^${module}/`), avoiding prefix collisions.
- **Explicit Job Control in Split Jobs**:
  - In `.kokoro/build.sh` (`integration-single` and `graalvm-single`), runs tests if
    either monorepo-wide testing is required or the specific module was modified:
    `! should_test_all_modules && ! is_module_modified "${BUILD_SUBDIR}"` -> skip.
- **Updated generate_modified_modules_list**:
  - Reuses `populate_modified_files` and `should_test_all_modules`.
- **Unit Tests**:
  - Added unit tests in `.kokoro/common_test.sh` for `is_module_modified` and
    `should_test_all_modules`.

Fixes googleapis#12029
@lqiu96
lqiu96 force-pushed the fix-split-it-module-detection branch from 54fb41a to 16b1841 Compare August 31, 2026 16:28
lqiu96 added a commit to lqiu96/google-cloud-java that referenced this pull request Aug 31, 2026
This PR is part 2 of 2 in a stacked series:
1. googleapis#14217: fix: avoid running split ITs unexpectedly on unrelated changes (googleapis#12029)
2. This PR: perf(ci): replace mvn help:evaluate with native bash and sed extraction

In Kokoro CI scripts, mvn help:evaluate was used to parse POM values:
- In generate_modified_modules_list (.kokoro/common.sh), evaluating
  project.modules launched a full JVM and evaluated the monorepo POMs,
  taking 20–30+ seconds on every single CI run.
- In downstream-build.sh (.kokoro/presubmit/downstream-build.sh),
  evaluating gapic-showcase.version launched another Maven JVM process.

- Pure-Bash Module Extraction: Updated generate_modified_modules_list to use
  extract_pom_modules pom.xml, extracting all 270 modules in ~0.02s without JVM
  boot overhead.
- Sed Showcase Version Extraction: Updated downstream-build.sh to parse
  <gapic-showcase.version> directly using sed with fail-fast validation.
- Unit Tests: Added automated unit tests in .kokoro/common_test.sh for
  extract_pom_modules and generate_modified_modules_list.
- Refactoring & Best Practices: Scoped all temporary variables as local,
  used read -r module <<< "${module}" for pure-bash whitespace trimming,
  and anchored paths with ${scriptDir}/..
@lqiu96
lqiu96 marked this pull request as draft August 31, 2026 17:34
lqiu96 added a commit to lqiu96/google-cloud-java that referenced this pull request Aug 31, 2026
…eapis#12029)

This PR is part 1 of 2 in a stacked series:
1. This PR (googleapis#14217): fix: avoid running split ITs unexpectedly on unrelated changes (googleapis#12029)
2. googleapis#14218: perf(ci): replace mvn help:evaluate with native bash and sed extraction

## Problem
In split integration and GraalVM tests (e.g. integration-single and graalvm-single),
module modification detection previously checked if the module name was present as a
substring in modified_module_list. For modules with shared prefixes (such as
java-bigquery vs java-bigquerystorage, java-bigquery-jdbc, or java-bigqueryconnection),
modifying one module could trigger tests for other prefix-sharing modules.

## Changes
- **Modular Helper Functions with Single Responsibility**:
  - `populate_modified_files`: Encapsulates diff retrieval, caching in `modified_files`,
    and configuring git `safe.directory` for Docker container ownership mismatches.
  - `should_test_all_modules`: Purely checks global overrides that require testing all
    modules (parent POMs, core shared dependencies, and `TEST_ALL_MODULES="true"`).
  - `is_module_modified`: Strictly checks if files within a specific module directory
    were modified (`^${module}/`), avoiding prefix collisions.
- **Explicit Job Control in Split Jobs**:
  - In `.kokoro/build.sh` (`integration-single` and `graalvm-single`), runs tests if
    either monorepo-wide testing is required or the specific module was modified:
    `! should_test_all_modules && ! is_module_modified "${BUILD_SUBDIR}"` -> skip.
- **Updated generate_modified_modules_list**:
  - Reuses `populate_modified_files` and `should_test_all_modules`.
- **Unit Tests**:
  - Added unit tests in `.kokoro/common_test.sh` for `is_module_modified` and
    `should_test_all_modules`.

Fixes googleapis#12029
@lqiu96
lqiu96 force-pushed the fix-split-it-module-detection branch from 16b1841 to 6b8aa56 Compare August 31, 2026 19:50
lqiu96 added a commit to lqiu96/google-cloud-java that referenced this pull request Aug 31, 2026
This PR is part 2 of 2 in a stacked series:
1. googleapis#14217: fix: avoid running split ITs unexpectedly on unrelated changes (googleapis#12029)
2. This PR: perf(ci): replace mvn help:evaluate with native bash and sed extraction

In Kokoro CI scripts, mvn help:evaluate was used to parse POM values:
- In generate_modified_modules_list (.kokoro/common.sh), evaluating
  project.modules launched a full JVM and evaluated the monorepo POMs,
  taking 20–30+ seconds on every single CI run.
- In downstream-build.sh (.kokoro/presubmit/downstream-build.sh),
  evaluating gapic-showcase.version launched another Maven JVM process.

- Pure-Bash Module Extraction: Updated generate_modified_modules_list to use
  extract_pom_modules pom.xml, extracting all 270 modules in ~0.02s without JVM
  boot overhead.
- Sed Showcase Version Extraction: Updated downstream-build.sh to parse
  <gapic-showcase.version> directly using sed with fail-fast validation.
- Unit Tests: Added automated unit tests in .kokoro/common_test.sh for
  extract_pom_modules and generate_modified_modules_list.
- Refactoring & Best Practices: Scoped all temporary variables as local,
  used read -r module <<< "${module}" for pure-bash whitespace trimming,
  and anchored paths with ${scriptDir}/..
@lqiu96

lqiu96 commented Aug 31, 2026

Copy link
Copy Markdown
Member Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the CI build scripts to modularize the logic for detecting modified files and determining whether to run tests for specific modules or the entire monorepo. It introduces helper functions get_modified_files, should_test_all_modules, and is_module_modified to prevent prefix collisions (e.g., matching java-bigquery instead of java-bigquerystorage) and adds comprehensive unit tests. The feedback suggests caching the output of git diff to avoid redundant executions and adding a trailing slash to the java-shared-dependencies regex to prevent potential prefix collisions.

Comment thread .kokoro/common.sh
Comment thread .kokoro/common.sh Outdated
…eapis#12029)

This PR is part 1 of 2 in a stacked series:
1. This PR (googleapis#14217): fix: avoid running split ITs unexpectedly on unrelated changes (googleapis#12029)
2. googleapis#14218: perf(ci): replace mvn help:evaluate with native bash and sed extraction

## Problem
In split integration and GraalVM tests (e.g. integration-single and graalvm-single),
module modification detection previously checked if the module name was present as a
substring in modified_module_list. For modules with shared prefixes (such as
java-bigquery vs java-bigquerystorage, java-bigquery-jdbc, or java-bigqueryconnection),
modifying one module could trigger tests for other prefix-sharing modules.

## Changes
- **Modular Helper Functions with Single Responsibility**:
  - `populate_modified_files`: Encapsulates diff retrieval, caching in `modified_files`,
    and configuring git `safe.directory` for Docker container ownership mismatches.
  - `should_test_all_modules`: Purely checks global overrides that require testing all
    modules (parent POMs, core shared dependencies, and `TEST_ALL_MODULES="true"`).
  - `is_module_modified`: Strictly checks if files within a specific module directory
    were modified (`^${module}/`), avoiding prefix collisions.
- **Explicit Job Control in Split Jobs**:
  - In `.kokoro/build.sh` (`integration-single` and `graalvm-single`), runs tests if
    either monorepo-wide testing is required or the specific module was modified:
    `! should_test_all_modules && ! is_module_modified "${BUILD_SUBDIR}"` -> skip.
- **Updated generate_modified_modules_list**:
  - Reuses `populate_modified_files` and `should_test_all_modules`.
- **Unit Tests**:
  - Added unit tests in `.kokoro/common_test.sh` for `is_module_modified` and
    `should_test_all_modules`.

Fixes googleapis#12029
@lqiu96
lqiu96 force-pushed the fix-split-it-module-detection branch from 6b8aa56 to 86b6dfb Compare August 31, 2026 20:05
lqiu96 added a commit to lqiu96/google-cloud-java that referenced this pull request Aug 31, 2026
This PR is part 2 of 2 in a stacked series:
1. googleapis#14217: fix: avoid running split ITs unexpectedly on unrelated changes (googleapis#12029)
2. This PR: perf(ci): replace mvn help:evaluate with native bash and sed extraction

In Kokoro CI scripts, mvn help:evaluate was used to parse POM values:
- In generate_modified_modules_list (.kokoro/common.sh), evaluating
  project.modules launched a full JVM and evaluated the monorepo POMs,
  taking 20–30+ seconds on every single CI run.
- In downstream-build.sh (.kokoro/presubmit/downstream-build.sh),
  evaluating gapic-showcase.version launched another Maven JVM process.

- Pure-Bash Module Extraction: Updated generate_modified_modules_list to use
  extract_pom_modules pom.xml, extracting all 270 modules in ~0.02s without JVM
  boot overhead.
- Sed Showcase Version Extraction: Updated downstream-build.sh to parse
  <gapic-showcase.version> directly using sed with fail-fast validation.
- Unit Tests: Added automated unit tests in .kokoro/common_test.sh for
  extract_pom_modules and generate_modified_modules_list.
- Refactoring & Best Practices: Scoped all temporary variables as local,
  used read -r module <<< "${module}" for pure-bash whitespace trimming,
  and anchored paths with ${scriptDir}/..
@lqiu96
lqiu96 requested review from blakeli0 and whowes August 31, 2026 20:07
@lqiu96
lqiu96 marked this pull request as ready for review August 31, 2026 20:07
@lqiu96 lqiu96 changed the title fix: avoid running split ITs unexpectedly on unrelated changes (#12029) ci: avoid running split ITs unexpectedly on unrelated changes (#12029) Aug 31, 2026

@blakeli0 blakeli0 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
For the bigquery modules, maybe we do want the ITs of downstream modules to be triggered by the upstream modules, otherwise they are only tested in release PRs. For example, the java-bigquery-jdbc-ci is triggered by java-bigquery and java-bigquerystorage changes as well.

@lqiu96

lqiu96 commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

For the bigquery modules, maybe we do want the ITs of downstream modules to be triggered by the upstream modules, otherwise they are only tested in release PRs. For example, the java-bigquery-jdbc-ci is triggered by java-bigquery and java-bigquerystorage changes as well.

Yeah this make sense. I'll create a follow up PR for this. We probably have a few special cases where there is a downstream component for a module

@lqiu96
lqiu96 merged commit f237e41 into googleapis:main Sep 1, 2026
198 checks passed
lqiu96 added a commit to lqiu96/google-cloud-java that referenced this pull request Sep 1, 2026
This PR is part 2 of 2 in a stacked series:
1. googleapis#14217: fix: avoid running split ITs unexpectedly on unrelated changes (googleapis#12029)
2. This PR: perf(ci): replace mvn help:evaluate with native bash and sed extraction

In Kokoro CI scripts, mvn help:evaluate was used to parse POM values:
- In generate_modified_modules_list (.kokoro/common.sh), evaluating
  project.modules launched a full JVM and evaluated the monorepo POMs,
  taking 20–30+ seconds on every single CI run.
- In downstream-build.sh (.kokoro/presubmit/downstream-build.sh),
  evaluating gapic-showcase.version launched another Maven JVM process.

- Pure-Bash Module Extraction: Updated generate_modified_modules_list to use
  extract_pom_modules pom.xml, extracting all 270 modules in ~0.02s without JVM
  boot overhead.
- Sed Showcase Version Extraction: Updated downstream-build.sh to parse
  <gapic-showcase.version> directly using sed with fail-fast validation.
- Unit Tests: Added automated unit tests in .kokoro/common_test.sh for
  extract_pom_modules and generate_modified_modules_list.
- Refactoring & Best Practices: Scoped all temporary variables as local,
  used read -r module <<< "${module}" for pure-bash whitespace trimming,
  and anchored paths with ${scriptDir}/..
lqiu96 added a commit to lqiu96/google-cloud-java that referenced this pull request Sep 1, 2026
This PR is part 2 of 2 in a stacked series:
1. googleapis#14217: fix: avoid running split ITs unexpectedly on unrelated changes (googleapis#12029)
2. This PR: perf(ci): replace mvn help:evaluate with native bash and sed extraction

In Kokoro CI scripts, mvn help:evaluate was used to parse POM values:
- In generate_modified_modules_list (.kokoro/common.sh), evaluating
  project.modules launched a full JVM and evaluated the monorepo POMs,
  taking 20–30+ seconds on every single CI run.
- In downstream-build.sh (.kokoro/presubmit/downstream-build.sh),
  evaluating gapic-showcase.version launched another Maven JVM process.

- Pure-Bash Module Extraction: Updated generate_modified_modules_list to use
  extract_pom_modules pom.xml, extracting all 270 modules in ~0.02s without JVM
  boot overhead.
- Sed Showcase Version Extraction: Updated downstream-build.sh to parse
  <gapic-showcase.version> directly using sed with fail-fast validation.
- Unit Tests: Added automated unit tests in .kokoro/common_test.sh for
  extract_pom_modules and generate_modified_modules_list.
- Refactoring & Best Practices: Scoped all temporary variables as local,
  used read -r module <<< "${module}" for pure-bash whitespace trimming,
  and anchored paths with ${scriptDir}/..
lqiu96 added a commit to lqiu96/google-cloud-java that referenced this pull request Sep 1, 2026
…ative bash, sed, and git optimizations

This PR is part 2 of 2 in a stacked series:
1. googleapis#14217: fix: avoid running split ITs unexpectedly on unrelated changes (googleapis#12029)
2. This PR: perf(ci): replace mvn help:evaluate with native bash and sed extraction

In Kokoro CI and automation scripts, heavy subprocesses and JVM invocations were used:
- In generate_modified_modules_list (.kokoro/common.sh), evaluating
  project.modules launched a full JVM and evaluated the monorepo POMs,
  taking 20–30+ seconds on every single CI run.
- In downstream-build.sh (.kokoro/presubmit/downstream-build.sh) and
  showcase-native.sh (sdk-platform-java/.kokoro/presubmit/showcase-native.sh),
  evaluating gapic-showcase.version launched Maven JVM processes.
- In check_status.sh (java-cloud-bom/tests/release-repository-readiness/check_status.sh),
  mvn help:evaluate was called for shared dependencies and generator versions,
  and table formatting piped echo into awk.
- In update_javadoc.sh (google-auth-library-java/scripts/update_javadoc.sh),
  maven-help-plugin:evaluate was called to obtain the project version.
- In update_googleapis_commit.sh (sdk-platform-java/.github/scripts/update_googleapis_commit.sh),
  a full git clone of googleapis was performed just to inspect HEAD commitish.
- In downstream-protobuf-binary-compatibility.sh (sdk-platform-java/.kokoro/nightly/downstream-protobuf-binary-compatibility.sh),
  cat/grep/cut was run on versions.txt on every loop iteration.
- In check_existing_release_versions.sh (generation/check_existing_release_versions.sh),
  xmllint was run 3 separate times per POM.

Optimizations:
- Pure-Bash Module Extraction: Updated generate_modified_modules_list to use
  extract_pom_modules pom.xml, extracting all 270 modules in ~0.02s without JVM
  boot overhead.
- Sed Showcase Version Extraction: Updated downstream-build.sh and showcase-native.sh
  to parse <gapic-showcase.version> directly using sed with fail-fast validation,
  and added --fail to curl.
- XML Tag Extraction & Printf Formatting: Updated check_status.sh to extract XML tags
  directly with sed and replaced pipe-to-awk with built-in printf formatting.
- Sed Javadoc Version Extraction: Updated update_javadoc.sh to parse project version
  from pom.xml using sed.
- Git Ls-Remote: Updated update_googleapis_commit.sh to query HEAD commit using
  git ls-remote instead of cloning the repository.
- Associative Array Pre-caching: Pre-cached versions.txt into declare -A versions_map
  in downstream-protobuf-binary-compatibility.sh instead of invoking cat/grep/cut per artifact.
- Combined XPath Query: Combined 3 xmllint calls into a single concat() XPath extraction
  in check_existing_release_versions.sh.
- Unit Tests: Verified existing unit tests in .kokoro/common_test.sh and ran bash -n
  and git diff --check across all modified scripts.
- Refactoring & Best Practices: Scoped temporary variables as local, used pure-bash
  whitespace trimming and string manipulations, and anchored relative paths.
lqiu96 added a commit to lqiu96/google-cloud-java that referenced this pull request Sep 1, 2026
…ative bash, sed, and git optimizations

This PR is part 2 of 2 in a stacked series:
1. googleapis#14217: fix: avoid running split ITs unexpectedly on unrelated changes (googleapis#12029)
2. This PR: perf(ci): replace mvn help:evaluate with native bash and sed extraction

In Kokoro CI and automation scripts, heavy subprocesses and JVM invocations were used:
- In generate_modified_modules_list (.kokoro/common.sh), evaluating
  project.modules launched a full JVM and evaluated the monorepo POMs,
  taking 20–30+ seconds on every single CI run.
- In downstream-build.sh (.kokoro/presubmit/downstream-build.sh) and
  showcase-native.sh (sdk-platform-java/.kokoro/presubmit/showcase-native.sh),
  evaluating gapic-showcase.version launched Maven JVM processes.
- In check_status.sh (java-cloud-bom/tests/release-repository-readiness/check_status.sh),
  mvn help:evaluate was called for shared dependencies and generator versions,
  and table formatting piped echo into awk.
- In update_javadoc.sh (google-auth-library-java/scripts/update_javadoc.sh),
  maven-help-plugin:evaluate was called to obtain the project version.
- In update_googleapis_commit.sh (sdk-platform-java/.github/scripts/update_googleapis_commit.sh),
  a full git clone of googleapis was performed just to inspect HEAD commitish.
- In downstream-protobuf-binary-compatibility.sh (sdk-platform-java/.kokoro/nightly/downstream-protobuf-binary-compatibility.sh),
  cat/grep/cut was run on versions.txt on every loop iteration.
- In check_existing_release_versions.sh (generation/check_existing_release_versions.sh),
  xmllint was run 3 separate times per POM.

Optimizations:
- Pure-Bash Module Extraction: Updated generate_modified_modules_list to use
  extract_pom_modules pom.xml, extracting all 270 modules in ~0.02s without JVM
  boot overhead.
- Sed Showcase Version Extraction: Updated downstream-build.sh and showcase-native.sh
  to parse <gapic-showcase.version> directly using sed with fail-fast validation,
  and added --fail to curl.
- XML Tag Extraction & Printf Formatting: Updated check_status.sh to extract XML tags
  directly with sed and replaced pipe-to-awk with built-in printf formatting.
- Sed Javadoc Version Extraction: Updated update_javadoc.sh to parse project version
  from pom.xml using sed.
- Git Ls-Remote: Updated update_googleapis_commit.sh to query HEAD commit using
  git ls-remote instead of cloning the repository.
- Associative Array Pre-caching: Pre-cached versions.txt into declare -A versions_map
  in downstream-protobuf-binary-compatibility.sh instead of invoking cat/grep/cut per artifact.
- Combined XPath Query: Combined 3 xmllint calls into a single concat() XPath extraction
  in check_existing_release_versions.sh.
- Unit Tests: Verified existing unit tests in .kokoro/common_test.sh and ran bash -n
  and git diff --check across all modified scripts.
- Refactoring & Best Practices: Scoped temporary variables as local, used pure-bash
  whitespace trimming and string manipulations, and anchored relative paths.
lqiu96 added a commit that referenced this pull request Sep 1, 2026
… changes (#14222)

## Problem
Following #14217, integration tests for split modules only run when
their own module directory is modified or when global parent POMs
change. However, some modules depend on other sibling modules in the
monorepo:
- `java-bigquery` depends on `java-bigquerystorage`
- `java-bigquery-jdbc` depends on `java-bigquery` and
`java-bigquerystorage`
- `java-spanner` depends on `grpc-gcp-java`
- `java-spanner-jdbc` depends on `java-spanner` and `grpc-gcp-java`
- `java-storage-nio` depends on `java-storage`
- `java-logging-logback` depends on `java-logging`

Additionally, changes in `sdk-platform-java` or
`google-auth-library-java` impact all client libraries across the entire
repository and should trigger all integration tests.

## Changes
- **Global Triggers (`should_test_all_modules`)**: Updated in
`.kokoro/common.sh` to trigger monorepo-wide integration tests if either
`sdk-platform-java` or `google-auth-library-java` is modified.
- **Upstream Mapping (`get_upstream_modules` &
`is_upstream_module_modified`)**: Added in `.kokoro/common.sh` to map
intra-monorepo upstream dependencies and check if any upstream module
was modified in the PR diff.
- **Split IT & GraalVM Jobs (`.kokoro/build.sh`)**: Updated
`integration-single` and `graalvm-single` to execute when an upstream
dependency is modified.
- **Batch Jobs (`generate_modified_modules_list`)**: Updated to include
downstream modules when their upstream dependencies are modified.
- **Unit Tests (`.kokoro/common_test.sh`)**: Added test coverage for
`sdk-platform-java`, `google-auth-library-java`, and all
upstream-downstream combinations.
lqiu96 added a commit that referenced this pull request Sep 1, 2026
…on (#14218)

This PR is part 2 of 2 in a stacked series:

1. #14217: fix: avoid running split ITs unexpectedly on unrelated
changes (#12029)
2. This PR: perf(ci): replace mvn help:evaluate with native bash and sed
extraction

--------

## Problem

In Kokoro CI and automation scripts, heavy Maven JVM invocations were
used to parse POM values:

- In `generate_modified_modules_list` (`.kokoro/common.sh`), evaluating
`project.modules` launched a full JVM and evaluated the monorepo POMs,
taking 20–30+ seconds on every single CI run.
- In `downstream-build.sh` (`.kokoro/presubmit/downstream-build.sh`) and
`showcase-native.sh`
(`sdk-platform-java/.kokoro/presubmit/showcase-native.sh`), evaluating
`gapic-showcase.version` launched Maven JVM processes.
- In `update_javadoc.sh`
(`google-auth-library-java/scripts/update_javadoc.sh`), evaluating
`project.version` called `maven-help-plugin:evaluate`.

## Changes

- **Pure-Bash Module Extraction**: Updated
`generate_modified_modules_list` to use `extract_pom_modules pom.xml`,
extracting all 270 modules in ~0.02s without JVM boot overhead.
- **Sed Showcase Version Extraction**: Updated `downstream-build.sh` and
`showcase-native.sh` to parse `<gapic-showcase.version>` directly using
`sed` with fail-fast validation, and added `--fail` to curl.
- **Sed Javadoc Version Extraction**: Updated `update_javadoc.sh` to
parse project version directly using `sed`, skipping parent POM
declarations.
- **Unit Tests**: Added automated unit tests in `.kokoro/common_test.sh`
for `extract_pom_modules` and `generate_modified_modules_list`.
- **Refactoring & Best Practices**: Scoped all temporary variables as
`local`, used pure parameter expansion for whitespace trimming, and
anchored paths.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Modified modules detection doesn't work correctly for modules with overlapping names

2 participants