Skip to content

[SPARK-59146][SQL] Retain qualified access to source columns affected by pipe SET - #58760

Open
AMC-hawk wants to merge 1 commit into
apache:masterfrom
AMC-hawk:SPARK-59146
Open

[SPARK-59146][SQL] Retain qualified access to source columns affected by pipe SET#58760
AMC-hawk wants to merge 1 commit into
apache:masterfrom
AMC-hawk:SPARK-59146

Conversation

@AMC-hawk

Copy link
Copy Markdown

What changes were proposed in this pull request?

The SQL pipe SET operator is built in AstBuilder.visitOperatorPipeSet as a star
expansion that excludes the assigned column and appends a replacement of the same name:

UnresolvedStarExceptOrReplace(
  target = None, excepts = Seq(Seq(ident)), replacements = Some(Seq(replacement)))

The excluded source attribute is dropped from the project list entirely, so nothing is
left for a later qualified reference such as t.a to resolve to.

This PR retains the excluded attribute as the hidden output of the Project that SET
builds, using the existing Project.hiddenOutputTag mechanism that USING joins already
use to hide their duplicated join keys. Concretely:

  • UnresolvedStarExceptOrReplace gains retainExceptedColumnsAsHidden (default false),
    so that ordinary SELECT * EXCEPT / * REPLACE is unaffected. At the point of star
    expansion the two are otherwise indistinguishable.
  • visitOperatorPipeSet sets it to true.
  • ResolveReferences, in the same place it expands the star and builds the new Project,
    records the excluded attributes on Project.hiddenOutputTag, mirroring
    commonNaturalJoinProcessing and ResolveAsOfJoin.

The retained attribute is spliced back into the project list by the existing
AddMetadataColumns rule, and only when it is actually referenced:

-- |> SELECT t.a
Project [a#x]
+- Project [(a#x + 1) AS a#x, b#x, a#x]      <- original retained on demand
   +- SubqueryAlias t
      +- LocalRelation [a#x, b#x]

-- |> SELECT *
Project [a#x, b#x]
+- Project [(a#x + 1) AS a#x, b#x]           <- unchanged, no extra column
   +- SubqueryAlias t
      +- LocalRelation [a#x, b#x]

The docs already describe this behavior, so no wording change was needed. The SET
section did contain the same example twice by accident; the duplicate is replaced with one
that demonstrates the alias retention. Happy to drop that hunk if it is considered out of
scope.

Why are the changes needed?

docs/sql-pipe-syntax.md documents that after a SET assignment "top-level column names
are updated but table aliases still refer to the original row values". That holds for
columns SET did not touch, but not for the assigned column itself:

VALUES (1, 10) AS t(a, b)
|> SET a = a + 1
|> SELECT t.a;
[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter with name
`t`.`a` cannot be resolved. Did you mean one of the following?
[`a`, `t`.`b`]. SQLSTATE: 42703

The suggestion list shows the asymmetry: a resolves and t.b resolves, but t.a
does not. SPARK-50772 (#49420) introduced the alias retention for SET/EXTEND/DROP,
but its regression scenario reads only untouched qualified source columns; it does not
read back a column that SET assigned.

This blocks the documented before/after use of pipe SET -- auditing, computing a delta
against the source value, and join disambiguation.

Does this PR introduce any user-facing change?

Yes. A qualified reference to a column affected by pipe SET now resolves to the original
input value instead of failing analysis. This is a change relative to master and to
released versions; it makes the behavior match the documentation.

Before:

VALUES (1, 10) AS t(a, b) |> SET a = a + 1 |> SELECT t.a;
-- [UNRESOLVED_COLUMN.WITH_SUGGESTION] ... `t`.`a` cannot be resolved

After:

VALUES (1, 10) AS t(a, b) |> SET a = a + 1 |> SELECT t.a;
-- 1

VALUES (1, 10) AS t(a, b) |> SET a = a + 1 |> SELECT a, t.a, t.b;
-- 2  1  10

The output schema of SET itself is unchanged; the retained column is hidden and is only
materialized when referenced by name.

How was this patch tested?

Golden file tests added to pipe-operators.sql, covering the reported repro, reading the
new and original values together, that SELECT * still yields the original schema,
qualified access through an alias introduced by |> AS, and a sequence of two SETs on
the same column.

The full SQLQueryTestSuite passes, which covers the existing SELECT * EXCEPT /
* REPLACE golden files. The regenerated golden diff contains only additions; no existing
expected output changed.

Was this patch authored or co-authored using generative AI tooling?

Yes.

… by pipe SET

The pipe SET operator is built as a star expansion that excludes the assigned
column and appends a replacement of the same name, so the original attribute is
dropped from the project list and a later qualified reference such as `t.a` has
nothing to resolve to. This contradicts the documented behavior that table
aliases keep referring to the original row values after an assignment.

Retain the excluded attribute as hidden output on the Project that SET builds,
via the existing Project.hiddenOutputTag mechanism that USING joins use for
duplicated join keys. AddMetadataColumns splices it back only when referenced,
so the output schema of SET and the plan for `|> SELECT *` are unchanged.

The retention is flagged from the parser because pipe SET and regular
`SELECT * EXCEPT` / `* REPLACE` produce an identical node at star-expansion
time, and the latter must not be affected.
@AMC-hawk

Copy link
Copy Markdown
Author

I’m not sure who would be the right reviewers for this PR.

Adding @uros-b and @cloud-fan, could you please help out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant