Skip to content

[SPARK-59459][CORE] Extract a helper for the HashPartitioner array-key check in PairRDDFunctions - #58758

Open
uros-b wants to merge 1 commit into
apache:masterfrom
uros-b:extract-hash-partitioner-array-check
Open

[SPARK-59459][CORE] Extract a helper for the HashPartitioner array-key check in PairRDDFunctions#58758
uros-b wants to merge 1 commit into
apache:masterfrom
uros-b:extract-hash-partitioner-array-check

Conversation

@uros-b

@uros-b uros-b commented Sep 12, 2026

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

PairRDDFunctions repeats the same guard -- throw hashPartitionerCannotPartitionArrayKeyError() when the partitioner is a HashPartitioner and the key type is an array -- in four places: partitionBy and the three cogroup overloads. This extracts it into a single private helper, failOnHashPartitionerWithArrayKey(partitioner), and calls it from those four sites.

Notes:

  • The partitionBy site wrote the condition in the opposite operand order (keyClass.isArray && partitioner.isInstanceOf[HashPartitioner]). Both operands are side-effect-free, so the helper's order is equivalent.
  • The nested check in combineByKeyWithClassTag -- inside a broader keyClass.isArray block that also guards map-side combining -- is intentionally left as-is, since it is a different shape.

Why are the changes needed?

The guard was duplicated verbatim; consolidating it removes the repetition and keeps the array-key / HashPartitioner rule in one place. Behavior is unchanged.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing tests; build/sbt core/compile passes. This is a behavior-preserving refactor.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.8)

@uros-b uros-b left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Waiting for CI here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant