[SPARK-59433][SQL] Render TIMESTAMP_LTZ explicitly when spark.sql.timestampType=TIMESTAMP_NTZ - #58734
[SPARK-59433][SQL] Render TIMESTAMP_LTZ explicitly when spark.sql.timestampType=TIMESTAMP_NTZ#58734pan3793 wants to merge 1 commit into
Conversation
…estampType=TIMESTAMP_NTZ ### What changes were proposed in this pull request? Make LTZ type rendering follow spark.sql.timestampType: when the session default is TIMESTAMP_NTZ, SHOW CREATE TABLE / schema DDL and LTZ literals spell TIMESTAMP_LTZ explicitly, so the emitted text re-parses to the same type. ### Why are the changes needed? Under spark.sql.timestampType=TIMESTAMP_NTZ the emitted DDL is not round-trippable: SHOW CREATE TABLE prints "ltz TIMESTAMP" for a TIMESTAMP_LTZ column, and re-executing it creates a TIMESTAMP_NTZ column. LTZ literals have the same problem in plan and column text. ### How was this patch tested? Added unit tests for both defaults and NTZ-default golden variants (goldens regenerated with SPARK_GENERATE_GOLDEN_FILES=1 and reviewed). Ran the catalyst literal and type suites and the sql timestamp and SHOW CREATE TABLE suites. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex
| // A bare `TIMESTAMP` resolves to the session default timestamp type, so render this type | ||
| // explicitly when that default is not this type, keeping the generated DDL round-trippable. | ||
| override def sql: String = | ||
| if (SqlApiConf.get.timestampType == TimestampNTZType) "TIMESTAMP_LTZ" else "TIMESTAMP" |
There was a problem hiding this comment.
[P1] Do not use client-local config to render LTZ. SqlApiConf is explicitly hardcoded in sql-api-only/Spark Connect, so this branch is never taken by a Scala Connect client even when the remote session has spark.sql.timestampType=TIMESTAMP_NTZ. For example, DataFrameReader sends a concrete StructType as schema.toDDL; the server reparses that DDL under the remote NTZ session, so a requested TimestampType is sent as bare TIMESTAMP and silently becomes TimestampNTZType. The concrete-schema overloads of from_json, from_csv, and from_xml have the same issue. Please use an unambiguous schema transport/rendering for concrete types and add Scala Connect coverage under an NTZ remote session.
|
|
||
| // A bare `TIMESTAMP` resolves to the session default timestamp type, so render this type | ||
| // explicitly when that default is not this type, keeping the generated DDL round-trippable. | ||
| override def sql: String = |
There was a problem hiding this comment.
This changes auto-generated column names too, not only DDL. Cast.sql and Literal.sql use dataType.sql, so under spark.sql.timestampType=TIMESTAMP_NTZ, SELECT CAST(c AS TIMESTAMP_LTZ) is now named CAST(c AS TIMESTAMP_LTZ) instead of CAST(c AS TIMESTAMP).
This can break existing persistent views after upgrade. A view created in an NTZ session, e.g. CREATE VIEW v AS SELECT CAST(ntz_col AS TIMESTAMP_LTZ) FROM t, stores the old query output column name and captures spark.sql.timestampType=TIMESTAMP_NTZ. On read, the view query is re-analyzed under the captured conf, produces the new name, and GetViewColumnByNameAndOrdinal fails with INCOMPATIBLE_VIEW_SCHEMA_CHANGE.
CTAS / DataFrame column names and toSQLType error messages change in the same way. Could you check this case and mention it in the user-facing change section?
| } | ||
| } | ||
|
|
||
| test("SPARK-59433: SHOW CREATE TABLE keeps TIMESTAMP_LTZ columns when the default timestamp " + |
There was a problem hiding this comment.
SHOW CREATE TABLE ... AS SERDE also goes through StructField.toDDL (showHiveTableHeader in tables.scala). It is documented as generating Hive DDL, but under NTZ it will now emit TIMESTAMP_LTZ for a timestamp column of a Hive SerDe table, which Hive cannot parse. Is this intended? Either way, it would be good to cover the AS SERDE path in a test.
3 participants
What changes were proposed in this pull request?
Make LTZ type rendering follow
spark.sql.timestampType: when the session default isTIMESTAMP_NTZ,SHOW CREATE TABLE/ schema DDL and LTZ literals spellTIMESTAMP_LTZexplicitly, so the emitted text re-parses to the same type. Nested fields are covered.
No new configuration.
Why are the changes needed?
Under
spark.sql.timestampType=TIMESTAMP_NTZthe emitted DDL is not round-trippable:LTZ literals have the same problem in plan and column text.
Does this PR introduce any user-facing change?
Yes, only when
spark.sql.timestampType=TIMESTAMP_NTZ: LTZ columns show asTIMESTAMP_LTZin
SHOW CREATE TABLE/ schema DDL, and LTZ literals render asTIMESTAMP_LTZ '...'. Thedefault (
TIMESTAMP_LTZ) output is unchanged.How was this patch tested?
Added unit tests for both defaults and NTZ-default golden variants (goldens regenerated
with
SPARK_GENERATE_GOLDEN_FILES=1and reviewed). Verified with the catalyst literal andtype suites and the sql timestamp and
SHOW CREATE TABLEsuites.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex