[GLUTEN-12850][VL] Support multiple Dev Container configurations - #12853
[GLUTEN-12850][VL] Support multiple Dev Container configurations#12853felipepessoto wants to merge 4 commits into
Conversation
- Keep the dynamic-link environment as the default for daily development. - Add a static-link configuration for portable jars and vcpkg reproduction. - Share post-create setup while printing variant-specific build instructions. - Persist separate ccache, Maven and vcpkg binary-cache volumes. - Select the arm64 vcpkg triplet automatically in the static environment. - Document configuration selection, limitations and safe switching. - Make the JDK 17 switch explicit for Spark 4.1 builds so Maven does not run the Java 17 release target under JDK 8. Generated-by: GitHub Copilot CLI gpt-5.6-sol Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR extends Gluten’s Dev Container support to a multi-configuration setup (dynamic-link as the default plus an additional static-link option), sharing common post-create initialization while updating developer documentation to describe configuration selection, intended use cases, and build/test workflows (including explicit JDK 17 switching for Spark 4.x in the dynamic image).
Changes:
- Adds a new
.devcontainer/velox-static/devcontainer.jsonconfiguration for static-link packaging and vcpkg reproduction while keeping.devcontainer/devcontainer.jsonas the default dynamic-link workflow. - Updates the shared
.devcontainer/post-create.shto tailor printed instructions and arm64 static vcpkg environment setup based on the selected configuration. - Refreshes developer docs to describe configuration choice in VS Code/Codespaces and to document build/test commands for each configuration.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| docs/developers/NewToGluten.md | Updates Dev Container references to the new multi-configuration layout under .devcontainer/. |
| docs/developers/dev-container.md | Documents configuration selection and provides updated dynamic/static build and test instructions (including Spark 4.x JDK 17 guidance). |
| .devcontainer/devcontainer.json | Marks the default dynamic-link config as the daily-dev option and sets GLUTEN_DEV_CONTAINER_VARIANT=velox-dynamic. |
| .devcontainer/velox-static/devcontainer.json | Introduces a static-link configuration with separate persisted caches (including /var/cache/vcpkg) and higher host requirements. |
| .devcontainer/post-create.sh | Shares setup across configs while printing variant-specific instructions and adding arm64 static vcpkg environment exports. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
/workspaces persists across Dev Container rebuilds and switches between the velox-dynamic and velox-static configs, but ep/build-velox/build/velox_ep/_build and cpp/build bake in the vcpkg toolchain choice (or its absence) at first CMake configure and CMake never re-evaluates it. Reusing a build tree from the other variant does not fail clearly -- it silently resolves dependencies like zlib/zstd from the wrong place and fails much later, e.g. "could not find SnappyConfig.cmake", deep into a build that can take hours. Add a check to post-create.sh that detects this mismatch and warns with the exact cleanup command up front, instead of letting it surface as a cryptic CMake error after a long build. Generated-by: Claude claude-sonnet-5 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
warn() output goes to a script that also installs JDK 17, clang-format, and prints long per-variant build instructions. A warning printed early (e.g. the stale build tree check) can easily scroll past unnoticed before the final banner. Collect warnings as they happen and reprint them in yellow after everything else, so they can't be missed. Generated-by: Claude claude-sonnet-5 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
docs/developers/NewToGluten.md:49
- The sentence refers to a single “pre-built Docker image”, but the repository now ships multiple Dev Container configurations under
.devcontainer/that can use different images. This wording is slightly misleading; consider pluralizing to reflect that the selected configuration determines the image.
To skip the manual environment setup, you can develop Gluten inside a pre-built Docker
image using the [Dev Container](https://containers.dev/) configurations under
`.devcontainer/`. See [Dev Containers](./dev-container.md) for configuration,
build and test instructions.
What changes are proposed in this pull request?
Fixes #12850.
Follow-up to #12778, which made the dynamic-link image the default development environment.
.devcontainer/devcontainer.jsonas the default dynamic-link configuration for daily Velox development and Spark unit tests..devcontainer/velox-static/devcontainer.jsonfor portable jar packaging and vcpkg issue reproduction./var/cache/vcpkgbinary cache.The Spark 4.1 instructions now explicitly switch the dynamic environment from its default JDK 8 to JDK 17:
buildbundle-veloxbe.shadds the Java 17 Maven profile and release target for Spark 4.x, but a Maven profile cannot switch the JDK that is already running. Without this step Scala fails with'17' is not a valid choice for '-release'.How was this patch tested?
Parsed both configurations with
@devcontainers/cli 0.88.0 read-configuration.Ran both post-create variants and verified the dynamic Spark 4.1 instructions, static packaging instructions, unknown-variant failure and idempotent arm64 environment setup.
Built Spark 4.1 under OpenJDK 17 with the same Maven command used by
buildbundle-veloxbe.sh:BUILD SUCCESSin 4:23, producinggluten-package-1.8.0-SNAPSHOT-4.1.jar.Validated the final static command end to end on CentOS 9/GCC 12/JDK 17 using the Maven repository and vcpkg binary cache extracted from
apache/gluten:vcpkg-centos-9:./dev/buildbundle-veloxbe.sh --enable_vcpkg=ON --build_arrow=OFF \ --spark_version=3.5The build processed 124 vcpkg packages, completed in 24:55 and produced
cpp/build/releases/libgluten.soplus the 88 MBgluten-velox-bundle-spark3.5_2.12-linux_amd64-1.8.0-SNAPSHOT.jar.Ran
bash -n, the license-header check,git diff --checkand VS Code diagnostics with no errors.Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot CLI gpt-5.6-sol