[4.1][SPARK-57267][INFRA] Prepare branch-4.1 build workflows for decoupling#56330
Open
gaogaotiantian wants to merge 4 commits into
Open
[4.1][SPARK-57267][INFRA] Prepare branch-4.1 build workflows for decoupling#56330gaogaotiantian wants to merge 4 commits into
gaogaotiantian wants to merge 4 commits into
Conversation
…ontained ### What changes were proposed in this pull request? This PR makes the remaining `branch-4.1` scheduled build workflow files self-contained, following the same approach as SPARK-57115 (which did this for `build_java17.yml`): - Rewrites `build_java21.yml`, `build_non_ansi.yml`, `build_maven.yml`, `build_maven_java21.yml`, `build_python_3.14.yml`, and `build_python_pypy3.10.yml` so they are triggered by `workflow_dispatch` only (the dormant `schedule:` triggers are removed; scheduled runs on a non-default branch never fire anyway), drop the inherited `branch: master` (so they build `branch-4.1`), and use generic, non-branch-tagged names. - Adds `build_python_3.11.yml`, the equivalent of `build_branch41_python.yml`. The per-build configurations are relocated from the active `build_branch41_*.yml` files on `master` so the coverage matches what `branch-4.1` schedules today. Notes: - `build_and_test.yml` defaults `branch` to `branch-4.1`, so those callers omit `branch`. `maven_test.yml` defaults `branch` to `master` and checks out `inputs.branch`, so `build_maven.yml` / `build_maven_java21.yml` keep an explicit `branch: branch-4.1`. - A pre-existing invalid trailing comma in the non-ANSI `envs` JSON was fixed during relocation. ### Why are the changes needed? This is the `branch-4.1` side of decoupling our scheduled CIs (cf. the `branch-4.x` effort). Scheduled workflows only fire from the default branch, so `branch-4.1` CI should consist of self-contained, dispatchable workflow files on `branch-4.1` that a single scheduler on `master` can trigger. This PR prepares those targets; a follow-up on `master` will add the scheduler and remove the `build_branch41_*.yml` files. ### Does this PR introduce _any_ user-facing change? No. CI only. ### How was this patch tested? These workflows can be triggered manually via `workflow_dispatch` once merged. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.8) Co-authored-by: Isaac
Bump the `branch` input default in `maven_test.yml` from `master` to `branch-4.1`, matching `build_and_test.yml` (which was already bumped when the branch was cut). The maven default was missed, so maven callers that omit `branch` were checking out `apache/spark@master`. With the default fixed, `build_maven.yml` and `build_maven_java21.yml` no longer need an explicit `branch: branch-4.1` (now identical to their `branch-4.x` counterparts). This also corrects `build_maven_java21_macos26.yml` and `build_maven_java21_arm.yml`, which omit `branch` and were likewise defaulting to master. The `build_branch40_maven*.yml` files pass `branch: branch-4.0` explicitly and are unaffected. Co-authored-by: Isaac
…branch-4.1 Same fix as `maven_test.yml`: bump the `branch` input default in `python_hosted_runner_test.yml` from `master` to `branch-4.1`. Its callers `build_python_3.11_macos26.yml` and `build_python_3.11_arm.yml` omit `branch`, so they were checking out `apache/spark@master` (the input is used as the checkout `ref`). `release.yml` keeps `default: master` for its `workflow_dispatch` input, where the operator selects the branch to release. Co-authored-by: Isaac
…anch to branch-4.1" This reverts commit fa663f5.
zhengruifeng
approved these changes
Jun 5, 2026
Contributor
zhengruifeng
left a comment
zhengruifeng
left a comment
There was a problem hiding this comment.
Thanks for driving this effort!
It will significantly simplify the CI!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR makes the remaining
branch-4.1scheduled build workflow files self-contained, following the same approach as SPARK-57115 (which did this forbuild_java17.yml):build_java21.yml,build_non_ansi.yml,build_maven.yml,build_maven_java21.yml,build_python_3.14.yml, andbuild_python_pypy3.10.ymlso they are triggered byworkflow_dispatchonly (the dormantschedule:triggers are removed; scheduled runs on a non-default branch never fire anyway), drop the inheritedbranch: master(so they buildbranch-4.1), and use generic, non-branch-tagged names.build_python_3.11.yml, the equivalent ofbuild_branch41_python.yml.The per-build configurations are relocated from the active
build_branch41_*.ymlfiles onmasterso the coverage matches whatbranch-4.1schedules today.Notes:
build_and_test.ymldefaultsbranchtobranch-4.1, so those callers omitbranch.maven_test.ymldefaultsbranchtomasterand checks outinputs.branch, sobuild_maven.yml/build_maven_java21.ymlkeep an explicitbranch: branch-4.1.envsJSON was fixed during relocation.Why are the changes needed?
This is the
branch-4.1side of decoupling our scheduled CIs (cf. thebranch-4.xeffort). Scheduled workflows only fire from the default branch, sobranch-4.1CI should consist of self-contained, dispatchable workflow files onbranch-4.1that a single scheduler onmastercan trigger. This PR prepares those targets; a follow-up onmasterwill add the scheduler and remove thebuild_branch41_*.ymlfiles.Does this PR introduce any user-facing change?
No. CI only.
How was this patch tested?
These workflows can be triggered manually via
workflow_dispatchonce merged.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)