Copy-on-Write and Merge-on-Read support

[glitchy]

What's the feature are you trying to implement?

iceberg-rust can append data files but has no support for row-level mutations. Query engines that need UPDATE, DELETE, MERGE, or compaction have no path through the transaction API today.

This epic tracks full Copy-on-Write (CoW) and Merge-on-Read (MoR) support. CoW comes first because it provides the foundation MoR depends on.

Copy-on-Write

• feat(transaction): add OverwriteAction with CoW delete support #2185 OverwriteAction--overwrite semantics with manifest rewriting for deleted entries
• Conflict detection (validateNoConflictingData / validateNoConflictingDeletes)
• Filter-based overwrite (overwriteByRowFilter)
• Compaction via rewrite

Merge-on-Read

• RowDeltaAction--write position and equality delete files (write path)
• Scan-side delete file reconciliation--apply delete files at read time (read path)
• V3 Deletion Vectors (Puffin DVs)

Related

• Java OverwriteFiles
• Java RowDelta
• Existing FastAppendAction (feat: support append data file and add e2e test #349)

Willingness to contribute

I can contribute to this feature independently

[wirybeaver]

Hi @glitchy, I am planning to contribute the merge_into for the datafusion iceberg-rust. The implementation requires to the fundanmental RowDeltaAction. I will implement the merge_into and RowDeltaAction for CoW at first. #2201

[glitchy]

hey @wirybeaver --thanks for the interest in this area! a few things worth flagging:

RowDeltaAction overlaps with OverwriteAction

the OverwriteAction in my PR (#2185) already handles adding data files and marking existing ones as deleted via manifest rewriting--add_data_files(), delete_data_files(), Operation::Overwrite, ManifestStatus::Deleted entries with preserved sequence numbers. looking at your RowDeltaAction (#2201), it reimplements the same core logic. RowDeltaAction is already planned for future PRs in this series (see checklist above)

datafusion is a consumer, not a dependency

the core Iceberg transaction primitives (OverwriteAction, RowDeltaAction, scan-side delete reconciliation) do not depend on datafusion--they live in the iceberg crate and need to work independently of any query engine. datafusion integration (INSERT OVERWRITE, MERGE INTO) is a layer on top that consumes these APIs. coupling them together makes both harder to review, and datafusion doesn't even support MERGE INTO parsing yet (there's no logical plan node for it and it falls through to not_impl_err!("Unsupported SQL statement") in their planner).

the sequencing for the core primitives is already outlined in the checklist above on this issue. once they go in then datafusion integration should be considered.

[wirybeaver]

Hi @glitchy , thanks for the context!

To keep the momentum going for both merge_into and the update DataFusion integration, I’d suggest we decouple the basic RowDeltaAction (CoW) from the more complex MoR implementation.

Since I have the CoW implementation ready and tested it together with for my merge_into full PoC #2201 and update full PoC #2205, merging the "bare" RowDeltaAction now would:

• Unblock downstream PRs for merge_into and update immediately.
• Reduce the scope of your MoR PR, making it easier and faster for maintainers to review.

I’m happy to rebase on your OverwriteAction as soon as it lands to ensure zero conflicts. This way, we aren't serializing the work—I can push the CoW foundation through now, and you can layer the complex MoR logic on top of it without having to worry about the boilerplate.

In terms of datafusion support on merge_into, I already noticed this unsupported problem, mentioned this in my epic note and plan to contribute a merge_into on the datafusion side. We can first implement a merge_into TableProvider hook like this PR do on the datafusion side: apache/datafusion#19142

Does that sound like a fair way to get both features landed faster?

[malon64]

Hi, I’m following this work from the perspective of a downstream Rust data ingestion tool.

From our side, the first valuable milestone would already be a clean CoW-oriented primitive that can support SCD1-style keyed upserts safely, even before full SQL MERGE ergonomics are available.

Happy to help by:

• testing proposed APIs/branches from a real downstream use case,
• sharing concrete acceptance scenarios for keyed upsert behavior,
• and possibly contributing on smaller enabling pieces if useful.