Skip to content

fix: quote partition field name in sortBy to handle dotted partition fields (#17425) - #17550

Open
waterWang wants to merge 3 commits into
apache:mainfrom
waterWang:fix/rewrite-manifests-sortby-dotted-partition-field
Open

fix: quote partition field name in sortBy to handle dotted partition fields (#17425)#17550
waterWang wants to merge 3 commits into
apache:mainfrom
waterWang:fix/rewrite-manifests-sortby-dotted-partition-field

Conversation

@waterWang

Copy link
Copy Markdown

Description

When RewriteManifestsSparkAction.sortBy() is called with a partition field name that contains a dot (e.g., a nested struct field flattened to a.b), the string concatenation in sortColumn() produces col("data_file.partition.a.b"). Spark interprets this as a nested field traversal (partitionab) rather than a single field named a.b, causing AnalysisException: [FIELD_NOT_FOUND] No such struct field a in a.b.

Root cause

The sortColumn() method at line 320 builds the Spark column reference by raw string concatenation:

col(DATA_FILE_PARTITION_COLUMN_NAME + "." + p)

When p contains a dot (e.g., a.b), the resulting reference is ambiguous.

Fix

Wrap the partition field name in backticks so Spark treats it as a single literal field name:

col(DATA_FILE_PARTITION_COLUMN_NAME + ".`" + p + "`")

The sibling class FixedWriter (which handles byte[]) already does it correctly.

Affected versions

  • spark/v3.5
  • spark/v4.0
  • spark/v4.1

Testing

The existing tests pass because they use simple partition field names without dots (e.g., c1, c2_trunc, c3_bucket). A test case with a dotted partition field name would trigger the bug.

Closes #17425

@github-actions github-actions Bot added the spark label Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Spark RewriteManifestsSparkAction sortBy throws "AnalysisException: [FIELD_NOT_FOUND] No such struct field" for nested field in partition spec

1 participant