Skip to content

[SPARK-58204][SQL] Fix unprotected mutable state in RegExpReplace, RegExpExtractBase, StringTranslate, FormatNumber, and NamedLambdaVariable by marking them stateful#57353

Open
vinodkc wants to merge 1 commit into
apache:masterfrom
vinodkc:fix-spark58154-stateful-expressions

Conversation

@vinodkc

@vinodkc vinodkc commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Override stateful: Boolean = true on five Catalyst expressions that maintain mutable
@transient state but were not marked stateful:

  • RegExpReplace
  • RegExpExtractBase (RegExpExtract, RegExpExtractAll)
  • StringTranslate
  • FormatNumber
  • NamedLambdaVariable

For NamedLambdaVariable, also override withNewChildrenInternal to reset the AtomicReference on copy while preserving exprId. LeafLike.withNewChildrenInternal returns this by default, which would make freshCopyIfContainsStatefulExpression() a no-op even with the stateful flag set.

Why are the changes needed?

freshCopyIfContainsStatefulExpression() only copies subtrees containing a stateful = true node. Without the override, these expressions are shared across projections. When two projections evaluate concurrently — e.g. via ConvertToLocalRelation on the driver or pipelined execution on executors — threads race on the shared @transient cache fields and silently corrupt results.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Added one test per affected group in the existing suites (HigherOrderFunctionsSuite, RegexpExpressionsSuite, StringExpressionsSuite), each asserting stateful == true and that freshCopyIfContainsStatefulExpression() returns a distinct instance with independent state.

Was this patch authored or co-authored using generative AI tooling?

No

…gExpExtractBase, StringTranslate, FormatNumber, and NamedLambdaVariable by marking them stateful
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant