What is the problem the feature request solves?
PR #5679 fixed concat_ws correctness for array<string> arguments by routing those expressions through Spark codegen dispatch, with Spark fallback when the dispatcher is disabled. This avoids the native execution failure from #5675, but these valid Spark expressions still have no native path.
Spark accepts any ordered mix of string and array<string> arguments after the separator. It flattens array elements in argument order, skips null string arguments and null array elements, and returns null for a null separator. DataFusion's built-in concat_ws accepts only string arguments.
This follows the maintainer suggestion in #5679 (comment).
Describe the potential solution
Add a Spark-compatible native concat_ws implementation for mixed string and array<string> inputs, then allow the array shape through CometConcatWs instead of codegen dispatch.
Coverage should include:
- mixed scalar and array arguments, preserving argument and element order;
- multiple arrays;
- null scalar arguments, null arrays, and null array elements;
- empty arrays and a null separator;
- non-foldable column inputs through the native execution path.
Keep the existing dispatcher/fallback guard until the native implementation covers these semantics.
Additional context
What is the problem the feature request solves?
PR #5679 fixed
concat_wscorrectness forarray<string>arguments by routing those expressions through Spark codegen dispatch, with Spark fallback when the dispatcher is disabled. This avoids the native execution failure from #5675, but these valid Spark expressions still have no native path.Spark accepts any ordered mix of string and
array<string>arguments after the separator. It flattens array elements in argument order, skips null string arguments and null array elements, and returns null for a null separator. DataFusion's built-inconcat_wsaccepts only string arguments.This follows the maintainer suggestion in #5679 (comment).
Describe the potential solution
Add a Spark-compatible native
concat_wsimplementation for mixed string andarray<string>inputs, then allow the array shape throughCometConcatWsinstead of codegen dispatch.Coverage should include:
Keep the existing dispatcher/fallback guard until the native implementation covers these semantics.
Additional context
concat_wscurrently rejects list inputs, so this likely needs a Spark-compatible Comet implementation rather than directly exposing the existing DataFusion function.