[SPARK-58205][SQL] Mark JSON/CSV/XML expressions stateful to prevent shared-evaluator data races#57354
Open
vinodkc wants to merge 2 commits into
Open
[SPARK-58205][SQL] Mark JSON/CSV/XML expressions stateful to prevent shared-evaluator data races#57354vinodkc wants to merge 2 commits into
vinodkc wants to merge 2 commits into
Conversation
vinodkc
force-pushed
the
fix-spark58204-json-csv-xml-stateful
branch
from
July 19, 2026 15:49
128a44f to
0427728
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Adds
override def stateful: Boolean = trueto ten JSON/CSV/XML expressions:GetJsonObject,JsonTuple,JsonToStructs,StructsToJson,SchemaOfJson,CsvToStructs,StructsToCsv,SchemaOfCsv,XmlToStructs, andStructsToXml.Why are the changes needed?
These expressions hold mutable state in
@transient lazy val evaluatorfields but did not overridestateful(which defaults tofalse). As a result,freshCopyIfContainsStatefulExpression()never made fresh copies of them. When the same expression node appears more than once in a query plan — e.g.df.select(from_json(col, schema), from_json(col, schema))— both references share the same underlying evaluator instance. In interpreted execution this causes data-race corruption and silently incorrect results.Does this PR introduce any user-facing change?
No. Queries that were previously correct continue to produce the same results. Queries that previously silently produced incorrect results due to evaluator sharing now produce correct results.
How was this patch tested?
Added one unit test per suite (
JsonExpressionsSuite,CsvExpressionsSuite,XmlExpressionsSuite)Was this patch authored or co-authored using generative AI tooling?
No