Skip to content

[GLUTEN-12863][VL][TEST] Re-enable six GlutenTryCastSuite cases excluded on Spark 3.4+ - #12886

Merged
jackylee-ch merged 1 commit into
apache:mainfrom
LuciferYang:spark33-drop-p05-vanilla-excludes
Aug 25, 2026
Merged

[GLUTEN-12863][VL][TEST] Re-enable six GlutenTryCastSuite cases excluded on Spark 3.4+#12886
jackylee-ch merged 1 commit into
apache:mainfrom
LuciferYang:spark33-drop-p05-vanilla-excludes

Conversation

@LuciferYang

@LuciferYang LuciferYang commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

VeloxTestSettings excludes six cases from GlutenTryCastSuite on Spark 3.4, 3.5, 4.0 and 4.1. This removes those exclusions. All six pass.

.exclude("ANSI mode: Throw exception on casting out-of-range value to byte type")
.exclude("ANSI mode: Throw exception on casting out-of-range value to short type")
.exclude("ANSI mode: Throw exception on casting out-of-range value to int type")
.exclude("ANSI mode: Throw exception on casting out-of-range value to long type")
.exclude("cast from invalid string to numeric should throw NumberFormatException")
.exclude("SPARK-26218: Fix the corner case of codegen when casting float to Integer")

The diff is 24 deleted lines and nothing else. gluten-ut/spark33 never excluded these six, so the gap is specific to 3.4+, and unlike the neighbouring entries in that block, none of the six carries a comment saying what failed.

Evidence

CI on this PR ran the suite on all four versions. I pulled the surefire XML out of the run instead of reading the job summaries, so the six could be checked by name:

job tests failures errors the six
spark34 83 0 0 all present, passed
spark35 83 0 0 all present, passed
spark35-scala213 83 0 0 all present, passed
spark40 88 0 0 all present, passed
spark41 104 0 0 all present, passed

velox_backend_ansi.yml does not run on PR events and was not triggered here. For this suite it makes no difference. FallbackOnANSIMode only consults spark.gluten.sql.ansiFallback.enabled when session ANSI is on, and TryCastSuite on 3.4+ never turns it on: neither CastWithAnsiOnSuite nor CastSuiteBase contains a setConf(ANSI_ENABLED, true), and the "AnsiOn" in the name refers to the expression-level evalMode, which TryCastSuite overrides to EvalMode.TRY. The enableSuite[GlutenTryCastSuite] block also sits outside the if (ansiNoFallback) guard in the 4.0 and 4.1 settings.

Why they were probably there

On Spark 3.3 TryCastSuite builds TryCast(...), a separate expression class that Spark 3.4 removed. Gluten's expression mapping is keyed by class, Sig(expClass: Class[_], name: String), and there is no Sig[TryCast]. TryCast also extends CastBase as a sibling of Cast, so it does not reach the case c: Cast dispatch in ExpressionConverter either. A plan containing it cannot be offloaded, so on 3.3 these cases run on vanilla Spark. Spark 3.4 folded try-cast into Cast(child, dataType, timeZoneId, EvalMode.TRY), which Gluten does offload (Spark34Shims.withTryEvalMode, then CastTransformer in UnaryExpressionTransformer.scala). So 3.4 is where these cases first started reaching Velox. What failed at that point is not recorded and I have not tried to reconstruct it. They pass now on all four versions.

Context

Found while inventorying which vanilla cases still run on gluten-ut/spark33 but are excluded on every later version, a prerequisite for removing that module under #12807 and tracked in #12863. These six are the only candidates from that inventory that pass once enabled. The rest either fail for the reasons already written beside them, or are ClickHouse-only and need a ClickHouse build to check.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude claude-opus-5

…ded on Spark 3.4+

VeloxTestSettings excludes six cases from GlutenTryCastSuite on 3.4, 3.5, 4.0 and 4.1.
All six pass once the exclusions are removed, so they cost coverage and buy nothing.
Unlike the neighbouring entries in that block, none of them carries a comment saying
what fails.

  .exclude("ANSI mode: Throw exception on casting out-of-range value to byte type")
  .exclude("ANSI mode: Throw exception on casting out-of-range value to short type")
  .exclude("ANSI mode: Throw exception on casting out-of-range value to int type")
  .exclude("ANSI mode: Throw exception on casting out-of-range value to long type")
  .exclude("cast from invalid string to numeric should throw NumberFormatException")
  .exclude("SPARK-26218: Fix the corner case of codegen when casting float to Integer")

Why they were likely added: on Spark 3.3 these cases live in AnsiCastSuiteBase and
TryCastSuite builds them through TryCast, a 3.3-only expression class. Gluten's
expression mapping is keyed by class, Sig(expClass: Class[_], name: String), and there
is no Sig[TryCast], so on 3.3 a plan containing TryCast cannot be offloaded and falls
back to vanilla Spark. A 3.3 run logs "Skipping evaluation - Nonempty inputRow cannot be
converted to DataFrame" ten times for this suite, and Gluten's own try_cast validation
is gated the same way with testWithMinSparkVersion("try_cast", "3.4"). Spark 3.4
replaced TryCast with Cast(child, dataType, timeZoneId, EvalMode.TRY), which Gluten does
offload, so the cases went from never reaching Velox to genuinely exercising it. Whatever
failed then passes now.

Found while inventorying which vanilla cases run on gluten-ut/spark33 but are excluded on
every surviving version, a prerequisite for removing that module under apache#12807. That
inventory produced 17 candidates; these six are the only ones that pass once enabled. The
others either fail in a way consistent with the reasons written beside them, or are
ClickHouse-only and need a CH build to check.

Verification: removing the six from gluten-ut/spark34 and running the suite against Spark
3.4 locally takes it from 72 to 78 cases with no new failures, and the six report as
passed. The same run also fails Gluten - data type casting and Gluten - cast string to
timestamp, two pre-existing timezone cases that are green in CI, so this machine is not
authoritative and the PR opens as a draft for CI to confirm on all four versions.
test-compile passes on -Pspark-3.4, -Pspark-3.5 -Pscala-2.13, -Pspark-4.0 -Pscala-2.13
and -Pspark-4.1 -Pscala-2.13, and spotless:check is clean.
@LuciferYang

Copy link
Copy Markdown
Contributor Author

Test first

@github-actions github-actions Bot added the CORE works for Gluten Core label Aug 24, 2026
@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@LuciferYang
LuciferYang marked this pull request as ready for review August 25, 2026 03:33
Copilot AI lite review requested due to automatic review settings August 25, 2026 03:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR re-enables six previously excluded GlutenTryCastSuite test cases for the Velox backend on Spark 3.4, 3.5, 4.0, and 4.1 by removing their exclusions from VeloxTestSettings. This aligns the enabled vanilla test set across Spark versions, supporting the ongoing effort to deprecate/remove gluten-ut/spark33 while preserving coverage.

Changes:

  • Removed four ANSI out-of-range numeric cast exclusions (byte/short/int/long) from GlutenTryCastSuite settings on Spark 3.4+.
  • Removed exclusions for invalid-string-to-numeric and SPARK-26218 float-to-integer try-cast behavior from GlutenTryCastSuite settings on Spark 3.4+.
  • Applied the same un-exclusion consistently across Spark 3.4 / 3.5 / 4.0 / 4.1 Velox UT modules.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
gluten-ut/spark34/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala Removes six GlutenTryCastSuite .exclude(...) entries so the cases run on Spark 3.4.
gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala Removes the same six exclusions so the cases run on Spark 3.5.
gluten-ut/spark40/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala Removes the same six exclusions so the cases run on Spark 4.0.
gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala Removes the same six exclusions so the cases run on Spark 4.1.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@LuciferYang

Copy link
Copy Markdown
Contributor Author

cc @zhouyuan @jackylee-ch @philo-he

@LuciferYang

Copy link
Copy Markdown
Contributor Author

Thank you @jackylee-ch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CORE works for Gluten Core

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants