[SPARK-59146][SQL] Retain qualified access to source columns affected by pipe SET - #58760
Open
AMC-hawk wants to merge 1 commit into
Open
[SPARK-59146][SQL] Retain qualified access to source columns affected by pipe SET#58760AMC-hawk wants to merge 1 commit into
AMC-hawk wants to merge 1 commit into
Conversation
… by pipe SET The pipe SET operator is built as a star expansion that excludes the assigned column and appends a replacement of the same name, so the original attribute is dropped from the project list and a later qualified reference such as `t.a` has nothing to resolve to. This contradicts the documented behavior that table aliases keep referring to the original row values after an assignment. Retain the excluded attribute as hidden output on the Project that SET builds, via the existing Project.hiddenOutputTag mechanism that USING joins use for duplicated join keys. AddMetadataColumns splices it back only when referenced, so the output schema of SET and the plan for `|> SELECT *` are unchanged. The retention is flagged from the parser because pipe SET and regular `SELECT * EXCEPT` / `* REPLACE` produce an identical node at star-expansion time, and the latter must not be affected.
Author
|
I’m not sure who would be the right reviewers for this PR. Adding @uros-b and @cloud-fan, could you please help out. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The SQL pipe
SEToperator is built inAstBuilder.visitOperatorPipeSetas a starexpansion that excludes the assigned column and appends a replacement of the same name:
The excluded source attribute is dropped from the project list entirely, so nothing is
left for a later qualified reference such as
t.ato resolve to.This PR retains the excluded attribute as the hidden output of the
ProjectthatSETbuilds, using the existing
Project.hiddenOutputTagmechanism that USING joins alreadyuse to hide their duplicated join keys. Concretely:
UnresolvedStarExceptOrReplacegainsretainExceptedColumnsAsHidden(defaultfalse),so that ordinary
SELECT * EXCEPT/* REPLACEis unaffected. At the point of starexpansion the two are otherwise indistinguishable.
visitOperatorPipeSetsets it totrue.ResolveReferences, in the same place it expands the star and builds the newProject,records the excluded attributes on
Project.hiddenOutputTag, mirroringcommonNaturalJoinProcessingandResolveAsOfJoin.The retained attribute is spliced back into the project list by the existing
AddMetadataColumnsrule, and only when it is actually referenced:The docs already describe this behavior, so no wording change was needed. The
SETsection did contain the same example twice by accident; the duplicate is replaced with one
that demonstrates the alias retention. Happy to drop that hunk if it is considered out of
scope.
Why are the changes needed?
docs/sql-pipe-syntax.mddocuments that after aSETassignment "top-level column namesare updated but table aliases still refer to the original row values". That holds for
columns
SETdid not touch, but not for the assigned column itself:The suggestion list shows the asymmetry:
aresolves andt.bresolves, butt.adoes not. SPARK-50772 (#49420) introduced the alias retention for
SET/EXTEND/DROP,but its regression scenario reads only untouched qualified source columns; it does not
read back a column that
SETassigned.This blocks the documented before/after use of pipe
SET-- auditing, computing a deltaagainst the source value, and join disambiguation.
Does this PR introduce any user-facing change?
Yes. A qualified reference to a column affected by pipe
SETnow resolves to the originalinput value instead of failing analysis. This is a change relative to master and to
released versions; it makes the behavior match the documentation.
Before:
After:
The output schema of
SETitself is unchanged; the retained column is hidden and is onlymaterialized when referenced by name.
How was this patch tested?
Golden file tests added to
pipe-operators.sql, covering the reported repro, reading thenew and original values together, that
SELECT *still yields the original schema,qualified access through an alias introduced by
|> AS, and a sequence of twoSETs onthe same column.
The full
SQLQueryTestSuitepasses, which covers the existingSELECT * EXCEPT/* REPLACEgolden files. The regenerated golden diff contains only additions; no existingexpected output changed.
Was this patch authored or co-authored using generative AI tooling?
Yes.