[GLUTEN-12863][VL][TEST] Re-enable six GlutenTryCastSuite cases excluded on Spark 3.4+ - #12886
Merged
jackylee-ch merged 1 commit intoAug 25, 2026
Conversation
…ded on Spark 3.4+
VeloxTestSettings excludes six cases from GlutenTryCastSuite on 3.4, 3.5, 4.0 and 4.1.
All six pass once the exclusions are removed, so they cost coverage and buy nothing.
Unlike the neighbouring entries in that block, none of them carries a comment saying
what fails.
.exclude("ANSI mode: Throw exception on casting out-of-range value to byte type")
.exclude("ANSI mode: Throw exception on casting out-of-range value to short type")
.exclude("ANSI mode: Throw exception on casting out-of-range value to int type")
.exclude("ANSI mode: Throw exception on casting out-of-range value to long type")
.exclude("cast from invalid string to numeric should throw NumberFormatException")
.exclude("SPARK-26218: Fix the corner case of codegen when casting float to Integer")
Why they were likely added: on Spark 3.3 these cases live in AnsiCastSuiteBase and
TryCastSuite builds them through TryCast, a 3.3-only expression class. Gluten's
expression mapping is keyed by class, Sig(expClass: Class[_], name: String), and there
is no Sig[TryCast], so on 3.3 a plan containing TryCast cannot be offloaded and falls
back to vanilla Spark. A 3.3 run logs "Skipping evaluation - Nonempty inputRow cannot be
converted to DataFrame" ten times for this suite, and Gluten's own try_cast validation
is gated the same way with testWithMinSparkVersion("try_cast", "3.4"). Spark 3.4
replaced TryCast with Cast(child, dataType, timeZoneId, EvalMode.TRY), which Gluten does
offload, so the cases went from never reaching Velox to genuinely exercising it. Whatever
failed then passes now.
Found while inventorying which vanilla cases run on gluten-ut/spark33 but are excluded on
every surviving version, a prerequisite for removing that module under apache#12807. That
inventory produced 17 candidates; these six are the only ones that pass once enabled. The
others either fail in a way consistent with the reasons written beside them, or are
ClickHouse-only and need a CH build to check.
Verification: removing the six from gluten-ut/spark34 and running the suite against Spark
3.4 locally takes it from 72 to 78 cases with no new failures, and the six report as
passed. The same run also fails Gluten - data type casting and Gluten - cast string to
timestamp, two pre-existing timezone cases that are green in CI, so this machine is not
authoritative and the PR opens as a draft for CI to confirm on all four versions.
test-compile passes on -Pspark-3.4, -Pspark-3.5 -Pscala-2.13, -Pspark-4.0 -Pscala-2.13
and -Pspark-4.1 -Pscala-2.13, and spotless:check is clean.
Contributor
Author
|
Test first |
|
Run Gluten Clickhouse CI on x86 |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR re-enables six previously excluded GlutenTryCastSuite test cases for the Velox backend on Spark 3.4, 3.5, 4.0, and 4.1 by removing their exclusions from VeloxTestSettings. This aligns the enabled vanilla test set across Spark versions, supporting the ongoing effort to deprecate/remove gluten-ut/spark33 while preserving coverage.
Changes:
- Removed four ANSI out-of-range numeric cast exclusions (byte/short/int/long) from
GlutenTryCastSuitesettings on Spark 3.4+. - Removed exclusions for invalid-string-to-numeric and SPARK-26218 float-to-integer try-cast behavior from
GlutenTryCastSuitesettings on Spark 3.4+. - Applied the same un-exclusion consistently across Spark 3.4 / 3.5 / 4.0 / 4.1 Velox UT modules.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| gluten-ut/spark34/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala | Removes six GlutenTryCastSuite .exclude(...) entries so the cases run on Spark 3.4. |
| gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala | Removes the same six exclusions so the cases run on Spark 3.5. |
| gluten-ut/spark40/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala | Removes the same six exclusions so the cases run on Spark 4.0. |
| gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala | Removes the same six exclusions so the cases run on Spark 4.1. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
Author
jackylee-ch
approved these changes
Aug 25, 2026
Contributor
Author
|
Thank you @jackylee-ch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
VeloxTestSettingsexcludes six cases fromGlutenTryCastSuiteon Spark 3.4, 3.5, 4.0 and 4.1. This removes those exclusions. All six pass.The diff is 24 deleted lines and nothing else.
gluten-ut/spark33never excluded these six, so the gap is specific to 3.4+, and unlike the neighbouring entries in that block, none of the six carries a comment saying what failed.Evidence
CI on this PR ran the suite on all four versions. I pulled the surefire XML out of the run instead of reading the job summaries, so the six could be checked by name:
velox_backend_ansi.ymldoes not run on PR events and was not triggered here. For this suite it makes no difference.FallbackOnANSIModeonly consultsspark.gluten.sql.ansiFallback.enabledwhen session ANSI is on, andTryCastSuiteon 3.4+ never turns it on: neitherCastWithAnsiOnSuitenorCastSuiteBasecontains asetConf(ANSI_ENABLED, true), and the "AnsiOn" in the name refers to the expression-levelevalMode, whichTryCastSuiteoverrides toEvalMode.TRY. TheenableSuite[GlutenTryCastSuite]block also sits outside theif (ansiNoFallback)guard in the 4.0 and 4.1 settings.Why they were probably there
On Spark 3.3
TryCastSuitebuildsTryCast(...), a separate expression class that Spark 3.4 removed. Gluten's expression mapping is keyed by class,Sig(expClass: Class[_], name: String), and there is noSig[TryCast].TryCastalso extendsCastBaseas a sibling ofCast, so it does not reach thecase c: Castdispatch inExpressionConvertereither. A plan containing it cannot be offloaded, so on 3.3 these cases run on vanilla Spark. Spark 3.4 folded try-cast intoCast(child, dataType, timeZoneId, EvalMode.TRY), which Gluten does offload (Spark34Shims.withTryEvalMode, thenCastTransformerinUnaryExpressionTransformer.scala). So 3.4 is where these cases first started reaching Velox. What failed at that point is not recorded and I have not tried to reconstruct it. They pass now on all four versions.Context
Found while inventorying which vanilla cases still run on
gluten-ut/spark33but are excluded on every later version, a prerequisite for removing that module under #12807 and tracked in #12863. These six are the only candidates from that inventory that pass once enabled. The rest either fail for the reasons already written beside them, or are ClickHouse-only and need a ClickHouse build to check.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-5