perf(crdt): make delta apply cost the delta, not the collection

[farhan-syah]

Summary

Applying one peer delta to a 1,500-row collection took 142 ms. It now takes 8.8 ms.

The apply path validates a delta by importing it into a copy of the collection, so a rejection can be discarded without touching authoritative state. It is reached per delta from the sync path, from committed transaction batches, and — the case that compounds — from WAL replay, where every delta ever written is re-applied on recovery.

I went in expecting the copy to be the cost. Measuring it says otherwise. Per delta, on 1,500 rows:

step	before
candidate export	2.9 ms
candidate decode	1.7 ms
write-set	6 µs
validate one row	129 ms

The copy was 3% of the apply. Both are addressed here, in that order of importance.

perf(crdt): resolve UNIQUE probes through the row container in hand

field_value_exists walked a collection's rows to answer one UNIQUE probe, and for each row built a collection/row/field string and re-resolved it from the document root:

let path = format!("{collection}/{key}/{field}");
if let Some(voc) = self.doc.get_by_str_path(&path) {

That is an allocation plus a path parse and walk per row, to reach a container the loop already holds. Its own sibling, field_value_exists_live, reaches the row through coll.get(&key) directly — this was the odd one out, not a considered trade.

Validating one row against 1,500 rows: 129 ms → 445 µs (290×).

The probe is O(collection) either way; this removes a large constant, not the scan. An index over unique fields would remove the scan, and is a larger change than this one.

perf(crdt): retain validation candidate across delta applies

The candidate's lifetime was one delta, so a run of deltas paid a full encode+decode of the collection per delta. Nothing required that: the correct lifetime is the run — a WAL replay, a committed batch, a sync burst. Loro's own fork is implemented as encode+decode too, so there is no cheaper copy to reach for; the fix is to copy once and reuse.

On a clean apply the candidate becomes authoritative, and the document it displaced takes the same delta and becomes the next candidate. Callers release candidates when their run ends (clear_apply_candidates, wired into WAL replay).

With validation no longer dominating: 14.6 ms → 8.8 ms per delta (1.7×).

Two things make reuse safe:

• A stale candidate is refused, not invalidated. Anything else that mutates a collection — a local write through state_mut, a transaction rollback, a snapshot import, a purge — leaves the candidate behind, and installing a behind candidate as authoritative would silently erase those writes. Rather than teach every mutation site about the candidate, the candidate is checked against the frontier it was cloned from and rebuilt if it does not match. No other site has to know this exists.
• A refused delta drops its candidate. Loro buffers even causally-pending operations, so a candidate that saw a rejected delta is poisoned and is never returned to the pool.

Fixed while here: a collection created by a delta apply got the wrong peer id

The candidate was built with the node's base peer_id, while state_mut builds collections with collection_peer_id(base, collection). Since the candidate becomes the authoritative document, a collection first written by a sync delta — and any collection whose first validated apply replaced its document — ended up carrying the base id.

That is precisely what the derivation exists to prevent, in its own words: the same operation identity minted for unrelated writes in different collections, so a peer that keeps those collections in one document "sees the second operation as a replay of the first and silently drops one of the rows."

Tests

• a_local_write_between_applies_is_not_erased_by_the_candidate — the load-bearing one. Verified to fail without the frontier check, with the local row genuinely gone.
• a_rejected_delta_does_not_poison_the_next_apply — a refused row must not resurface through a reused candidate.
• a_run_of_applies_reuses_one_candidate — one candidate serves a run, and releasing it clears it.
• a_collection_created_by_apply_gets_the_derived_peer_id — pins the peer-id fix.

Validation

• cargo nextest run -p nodedb-crdt — 163/163
• cargo nextest run -p nodedb --all-features — 7942/7942
• cargo nextest run -p nodedb-cluster-tests --all-features --retries 0 — 322/323
• cargo clippy --all-targets --all-features -p nodedb-crdt -p nodedb -- -D warnings — clean
• cargo fmt --all — clean

The cluster failure is a4_placement_convergence::placement_converges_to_min_rf_when_node_count_exceeds_rf, which is vShard placement with no CRDT involvement. It failed at 31.7 s here and at 31.7 s on an unrelated branch (#231) with the suite's usual single retry disabled, and passes standalone in 2.4 s. Pre-existing load flake, not caused by this change.

Scope

Measurements are from a 1,500-row collection with one UNIQUE constraint. The shape holds above that — both costs are O(collection) per delta — but the multipliers will differ with row count, field count and constraint count. Nothing here has been run against the store in #230.