Skip to content

Spark: Fix wap branch delete scan/write to the wrong branch#15592

Closed
yingjianwu98 wants to merge 1 commit intoapache:mainfrom
yingjianwu98:yingjianw/fix_spark_wap_branch_metadata_delete
Closed

Spark: Fix wap branch delete scan/write to the wrong branch#15592
yingjianwu98 wants to merge 1 commit intoapache:mainfrom
yingjianwu98:yingjianw/fix_spark_wap_branch_metadata_delete

Conversation

@yingjianwu98
Copy link
Copy Markdown
Contributor

@yingjianwu98 yingjianwu98 commented Mar 12, 2026

Under certain cases, SparkTable, deleteWhere() and canDeleteWhere() do not resolve the WAP (Write-Audit-Publish) branch from the session configuration.

This causes two issues:

  • canDeleteWhere() scans the wrong branch's files, so it cannot correctly determine whether the delete can be resolved at the metadata level, causing delete to fail. This is an existing issue across all spark version
  • deleteWhere() commits the delete to main instead of the WAP branch, incorrectly dropping data on the main branch. I think it's a recent regression from this PR. Not a Spark expert here but based on my stack trace, I think the spark optimizer: OptimizeMetadataOnlyDeleteFromTable doesn't have the branch information so it will always perform on the main branch.

Fix:

  • Use determineReadBranch() in canDeleteWhere() to scan the correct branch
  • Use determineWriteBranch() in deleteWhere() to commit to the correct branch

@github-actions github-actions Bot added the spark label Mar 12, 2026
@yingjianwu98 yingjianwu98 changed the title Yingjianw/fix spark wap branch metadata delete Spark: Fix wap branch metadata delete on the wrong branch Mar 12, 2026
@yingjianwu98 yingjianwu98 force-pushed the yingjianw/fix_spark_wap_branch_metadata_delete branch from 6a0f953 to bed4cd2 Compare March 13, 2026 22:46
@yingjianwu98 yingjianwu98 changed the title Spark: Fix wap branch metadata delete on the wrong branch Spark: Fix wap branch delete scan/write to the wrong branch Mar 13, 2026
@github-actions
Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Apr 14, 2026
@github-actions
Copy link
Copy Markdown

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions Bot closed this Apr 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant