Retire imbl: the radix fan becomes a hand-rolled sorted Fan

Cargo.toml

@@ -37,13 +37,13 @@ protocol-v1 = []
 
 [dependencies]
 before = { path = "crates/before", features = ["borsh"] }
-imbl = "7"
 bytes = "1"
 blake3 = "1.8"
 static_assertions = "1.1"
 itertools = "0.14"
 borsh = { version = "1.6", features = ["bytes", "derive", "de_strict_order"] }
 seq-macro = "0.3"
+smallvec = { version = "1.15", features = ["union"] }
 tinyvec = { version = "1.11", features = ["alloc"] }
 thiserror = "2.0"
 hex = "0.4"

design/streaming-latency-serialization.md

@@ -291,7 +291,7 @@ boundary; completion time is approximately
 so `K ≥ expected disputed scopes` recovers full pipelining, and any
 `K > 1` divides the latency term proportionally. Memory: worst case
 ~K·fan node handles per recursive boundary (handles, not nodes —
-`imbl` structural sharing keeps the bytes shared), which is exactly
+the per-node `Arc` sharing keeps the bytes shared), which is exactly
 the trade the assembly fan queue already makes at K = fan. A default
 of one full fan (256) is the natural unit: it collapses the
 pathology by 24×–30× here, costs at most one fan of `Query` values
@@ -698,9 +698,9 @@ session holds its pre-session root alive from handshake to output
 buffered `Query` or `Resolution` points into is resident *anyway*;
 K× more handles pin K× more refcounts, not K× more tree. And no path
 in the walk deep-copies: `Backend::children` iterates `Arc` clones
-out of the persistent map, and `Backend::parent` builds fresh spine
-nodes *from* shared children. The imbl `OrdMap` diff machinery the
-tree rests on prunes pointer-equal spans wholesale for the same
+out of the radix fan, and `Backend::parent` builds fresh spine
+nodes *from* shared children. The join's lockstep merge-walk prunes
+children equal by pointer-or-hash before descending for the same
 reason.
 
 ### 6.2 What K does not buy (paid identically at K = 1)

results/mirror-complexity.tex

@@ -181,9 +181,9 @@ \section{The system, in one page}
 vectors (\code{partition.rs:}\allowbreak\code{100-115, 197-203};
 \code{src/tree/typed/}\allowbreak\code{levels/level.rs:1-10}), so per-round work
 is linear in the
-frontier and message sizes --- no hidden log factors beyond the \code{imbl}
-\code{OrdMap} child maps (an $O(\log 256)$ constant per child operation,
-absorbed below).
+frontier and message sizes --- no hidden log factors beyond the sorted
+child fans (an $O(\log 256)$-bounded probe and $O(256)$-bounded rebuild
+per child operation, absorbed below).
 
 \textbf{Termination is in-band}: a side is done when its outgoing
 \code{requested} and \code{uncertain} are both empty

src/tree.rs

@@ -34,9 +34,11 @@
 //!
 //! # Memos and sharing
 //!
-//! Nodes are persistent (`imbl::OrdMap` children behind `Arc`), so cloning
-//! a tree — every [`Snapshot`](crate::Snapshot), every gossip session's
-//! working copy — is O(1) and shares structure; mutation is copy-on-write.
+//! Every node lives behind an `Arc` (its children a sorted radix fan of
+//! further handles), so cloning a tree — every
+//! [`Snapshot`](crate::Snapshot), every gossip session's working copy — is
+//! O(1) and shares structure; mutation is copy-on-write along the touched
+//! spine.
 //! Each branch lazily memoizes three pure functions of its subtree: the
 //! Merkle **hash** (mirror pruning), and the **ceiling** and **floor** of
 //! its leaves' versions. The version bounds power both deletion honoring

src/tree/mirror/alternating/message.rs

@@ -46,7 +46,7 @@
 //! Multi-child branches always carry at least two children; singletons
 //! appear on the wire only as `prefix_len > 0` and reconstruct through
 //! [`Node::beneath`](crate::tree::typed::Node::beneath). Branch radices
-//! are required to be strictly ascending (matching the backing `OrdMap`'s
+//! are required to be strictly ascending (matching the backing fan's
 //! canonical iteration order).
 //!
 //! ## The three channels

src/tree/mirror/streaming/remote/proxy/tests/greeting.rs

@@ -199,3 +199,90 @@ fn mixed_empty_and_populated_converges() {
     assert_eq!(hash_of(&left), expected);
     assert_eq!(hash_of(&right), expected);
 }
+/// WITNESS (the gossip install must merge-walk clone-derived fans):
+///
+/// Two honest, overlapping sessions at one peer must not silently delete a
+/// message nobody redacted.
+///
+/// The shape: session S2 (with a peer converged at T0) forks at T0; equal
+/// handshake versions resolve without opening the descent, so S2's
+/// reconciled root is the fork-time root handle itself — its children fan
+/// is the very object M0. Meanwhile session S1 (with a peer that redacted
+/// one message at root radix `r_h`) installs first: the install's
+/// `Tree::join` builds its merged fan as `ours.clone()` + `remove(r_h)` —
+/// a clone-derived sibling M1 sharing every remaining child handle with
+/// M0. S2's install then joins M1 against M0: a wide pair, identical
+/// except at `r_h`, whose equal children form pointer-shared run after
+/// pointer-shared run. A shortcut walk that elides entries after a shared
+/// run — anything less than pairing the two fans radix by radix — desyncs
+/// onto *neighboring, unchanged* radixes, which version dominance then
+/// redacts as phantom deletions.
+///
+/// T0 is S2's causal past, so S2's install must be an identity on the
+/// tree: the assertion is total (post-install hash equals pre-install
+/// hash), swept over the redaction of each leaf in turn so every radix
+/// position plays the divergence point in some round. `Tree::join`
+/// merge-walks the two radix fans in lockstep, pruning equal pairs by
+/// pointer-or-hash before descending; this witness holds the join to
+/// that, and fails if a shortcut walk over shared runs ever returns.
+#[test]
+fn overlapping_sessions_lose_innocent_leaf_after_honored_redaction() {
+    use crate::tree::Action;
+
+    // T0: 25 unit messages. Hash-uniform keys give the root a wide fan
+    // (~25 children), every radix holding one leaf, so honoring one leaf's
+    // redaction empties its root radix — and sweeping the redacted leaf
+    // puts shared runs on both sides of every divergence point.
+    let p = nth_party(0);
+    let mut t0 = Tree::new();
+    t0.act(&p, (0..25).map(|_| Action::Insert(Message::new(()))));
+    let keys: Vec<_> = t0.iter().map(|(k, _, _)| k).collect();
+
+    // S2's wire session against a peer converged at T0, forked at T0:
+    // equal versions resolve to the fork-time root.
+    let (s2_reconciled, _) = wire_reconcile(t0.root.clone(), t0.root.clone());
+
+    let mut lost = Vec::new();
+    for k in &keys {
+        // S1's counterparty: converged at T0, then redacted the leaf at
+        // `k` (a local act rebuilds its own fans afresh; the sharing that
+        // matters is created by our install below, not here).
+        let mut twin = Tree {
+            root: t0.root.clone(),
+        };
+        twin.act(&nth_party(1), [Action::Forget(*k)]);
+
+        // S1's session and install: reconcile T0 against the redacting
+        // twin over the wire, then join the result into the live tree.
+        // Deletion honoring drops radix `r_h`: the live tree's root fan is
+        // now a clone-derived sibling of M0 missing one radix.
+        let (s1_reconciled, _) = wire_reconcile(t0.root.clone(), twin.root.clone());
+        let mut live = Tree {
+            root: t0.root.clone(),
+        };
+        live.join(Tree {
+            root: s1_reconciled,
+        });
+        let expected = live.hash();
+
+        // S2's install, after S1's: joining our own causal past must be an
+        // identity on the tree.
+        live.join(Tree {
+            root: s2_reconciled.clone(),
+        });
+
+        if live.hash() != expected {
+            let missing: Vec<_> = keys
+                .iter()
+                .filter(|k2| *k2 != k && live.get(k2).is_none())
+                .copied()
+                .collect();
+            lost.push((*k, missing));
+        }
+    }
+    assert!(
+        lost.is_empty(),
+        "S2's install lost innocent leaves after S1 honored a redaction \
+         (redacted key, innocent leaves missing): {lost:#?}",
+    );
+}

src/tree/traverse/act.rs

@@ -103,7 +103,7 @@ where
                 .collect();
 
             // Mutably pull the existing child out of the parent:
-            let existing_child = existing_children.remove(&radix);
+            let existing_child = existing_children.remove(radix);
 
             // Short-circuit when solely trying to delete from a non-existent child:
             if existing_child.is_none()

src/tree/traverse/join.rs

@@ -21,12 +21,18 @@
 //! - **both have it, hashes equal**: the subtrees are identical (content
 //!   addressing makes equal hash ⟹ equal content, versions included), so keep
 //!   one verbatim.
-//! - **both have it, hashes differ**: explode both one level and recurse only
-//!   into the radixes whose child subtrees differ (an [`OrdMap::diff`] that
-//!   prunes the shared ones by pointer), reassembling with [`Node::branch`]
-//!   (which re-compresses singletons and recomputes the joined branch version).
+//! - **both have it, hashes differ**: explode both one level and merge-walk
+//!   the two ascending radix fans in lockstep, recursing only into the
+//!   radixes whose child subtrees differ — children equal by pointer or by
+//!   content hash carry over verbatim through the shared structure — and
+//!   reassembling with [`Node::branch`] (which re-compresses singletons and
+//!   recomputes the joined branch version).
 //!
-//! [`OrdMap::diff`]: imbl::OrdMap::diff
+//! The merge walk enumerates each *divergent* branch's full fan (≤ 256
+//! entries) rather than only its changed radixes; equal subtrees still
+//! prune by pointer-or-hash before any descent, so a small delta against a
+//! large shared tree costs work proportional to the delta at the tree
+//! level, with a per-divergent-node constant bounded by the fan.
 
 use crate::Version;
 
@@ -95,36 +101,58 @@ where
                     return Some(ours);
                 }
 
-                // Differing subtrees: descend one level, but only into the
-                // radixes that actually diverge. `OrdMap::diff` walks both
-                // persistent B-trees in lockstep and prunes whole spans that are
-                // pointer-equal — the shared backing a fork leaves behind — so it
-                // yields exactly the changed children, in ascending-radix order,
-                // without enumerating the full radix union or probing the
-                // unchanged children. A small delta against a large shared tree
-                // therefore costs work proportional to the delta, not to the
-                // fan-out. (`diff` classifies a child as unchanged via `Node`'s
-                // `PartialEq`, which is the same `ptr_eq`-or-hash short-circuit
-                // the node-level equality above uses: nothing is learned across
-                // an equal subtree, so it carries over verbatim.)
+                // Differing subtrees: descend one level, merge-walking the
+                // two ascending radix fans in lockstep and recursing only
+                // into the radixes that actually diverge. A child equal on
+                // both sides — by `Node`'s `ptr_eq`-or-hash equality, the
+                // same short-circuit the node-level check above uses —
+                // carries over verbatim: nothing is learned across an equal
+                // subtree. One-sided radixes recurse too: the asymmetric
+                // arms above filter them against the absent side's version,
+                // which is where deletion honoring drops what that side
+                // redacted.
                 //
-                // Collect the divergent radixes first — cloning only those few
-                // children — so we don't hold `diff`'s borrow of `ours` /
-                // `theirs` across the recursive rewrite of the merged map.
+                // The walk reads both fans directly rather than diffing the
+                // two maps against each other: the merged map starts from
+                // *ours* and only divergent radixes are rewritten, so every
+                // shared child persists by structural sharing.
                 let ours = ours.into_children();
                 let theirs = theirs.into_children();
 
-                // Start the merged map from *ours* (moved — `diff`'s borrow has
-                // ended) and rewrite only the divergent radixes; every shared
-                // child carries over verbatim by structural sharing.
                 let mut merged = ours.clone();
-                for (radix, our_child, their_child) in ours.diff_owned(&theirs) {
+                let mut ours = ours.iter().peekable();
+                let mut theirs = theirs.iter().peekable();
+                loop {
+                    let (radix, our_child, their_child) = match (ours.peek(), theirs.peek()) {
+                        (None, None) => break,
+                        (Some((radix, _)), None) => {
+                            (*radix, ours.next().map(|(_, child)| child), None)
+                        }
+                        (None, Some((radix, _))) => {
+                            (*radix, None, theirs.next().map(|(_, child)| child))
+                        }
+                        (Some((ours_radix, _)), Some((theirs_radix, _))) => {
+                            let radix = (*ours_radix).min(*theirs_radix);
+                            (
+                                radix,
+                                ours.next_if(|(r, _)| *r == radix).map(|(_, child)| child),
+                                theirs.next_if(|(r, _)| *r == radix).map(|(_, child)| child),
+                            )
+                        }
+                    };
+
+                    if let (Some(our_child), Some(their_child)) = (&our_child, &their_child)
+                        && our_child == their_child
+                    {
+                        continue;
+                    }
+
                     match Join::join(our_child, their_child, a_version, b_version) {
                         Some(child) => {
                             merged.insert(radix, child);
                         }
                         None => {
-                            merged.remove(&radix);
+                            merged.remove(radix);
                         }
                     }
                 }