libsql-server: make namespace creation all-or-nothing (stacked on #51)

[tszymczyszyn-shopify]

Summary

Stacked on #51 (targets tszymczyszyn/streaming-dump-importer; will be retargeted to v0.9.30-shopify-patches once #51 merges — only the last three commits are this PR: f51aca96ba code and tests, 1f688cb9ca design doc and admin API, bb446259f6 diagrams).

Makes POST /v1/namespaces/:ns/create all-or-nothing. Fixes the pre-existing lifecycle gap called out in #51's design doc §13: a dump that failed to import left a ghost namespace (config without data, name unusable, lazily turned into an empty DB on first access); a request cancelled mid-import ran no cleanup at all; the schema configurator had no cleanup even on error.

Design: docs/ATOMIC_NAMESPACE_CREATE_DESIGN.md.

📖 Reviewer walkthrough — before/after and component diagrams, sequence diagrams for the happy path, failure/cancellation and eviction, the lifecycle state machine, a suggested reading order with pinned links, and where reviewer judgment is most valuable. Live board: https://drawr.quick.shopify.io/b/GNUB8o_56hhfFGxv

Contract (new, documented in ADMIN_API.md)

After a non-2xx or a lost connection the namespace is either absent (no config, no directory, name reusable) or complete — never partial. For callers: 2xx ⇒ complete; anything else ⇒ retry; already exists on the retry ⇒ the earlier attempt did complete.

How

The metastore write becomes the commit point; everything before it is reversible.

create(name, restore, config)
├─ Reservation::acquire(name)        lock cache entry; reject if loaded or config exists
├─ handle(); store(cfg, flush=false) in-memory only — setup can read it, nothing on disk
├─ configurator.discard_incomplete   remove dbs/<name> iff it carries `.incomplete`
├─ make_namespace(...)               setup: FreshDir::begin → mkdir + marker … import … keep()
├─ handle.flush()                    ◄── COMMIT POINT
├─ ns.mark_complete()                remove marker (best effort)
└─ reservation.publish(ns)           entry := Some(ns); lock released
     Err/drop above   ⇒ Reservation::drop ⇒ remove in-memory config, THEN release lock
     Err/drop in setup ⇒ FreshDir::drop   ⇒ remove_dir_all

• Reservation generalises the pattern fork already used (write-lock the cache entry, config in memory only, flush on success). Dropping it unpublished — error or cancellation — removes the config on the blocking pool and releases the lock only afterwards, so waiters see "doesn't exist", never a half-built namespace. fork now uses it too; its old guard used block_in_place (panics on a current-thread runtime) and published before flushing.
• FreshDir removes a brand-new directory on drop unless keep()-ed; primary and schema configurators.
• .incomplete marker for crashes: written with the directory, removed after the flush; consulted in exactly one place (create, for a name with no config and nothing loaded) so it can never wipe a published namespace; maybe_recover_from_fs skips marked dirs. Marker-less directories keep today's semantics.
• Requests addressed to a namespace under creation wait (as during fork/reset).
• Needs async-lock = "3" for RwLock::write_arc — already in the lockfile; the bump removes the duplicate 2.x.

Behaviour changes visible to clients

• Failed create ⇒ GET …/config is 404 and user requests get "namespace doesn't exist" (was: 200 / "no such table").
• Single-namespace mode: create default from a dump while it exists ⇒ already exists (was: 200 with the dump silently skipped). The config-upsert behaviour of a plain create default is unchanged.

Tests

• 9 new lifecycle tests + 2 tightened cancellation tests (both importers); 9 of the 11 fail on the unfixed code, the other two pin behaviour that must hold before and after (requests wait; unmarked dirs untouched).
• Unit tests for FreshDir (incl. drop of the enclosing future), discard_incomplete, and the eviction-during-creation scenarios above.
• Full libsql-server suite under nextest (270 tests): identical results to the base branch (local_sync_with_writes and the wall-clock-ratio replica_no_resync_on_restart fail under parallel load on both; both pass alone).

What the adversarial review changed (now folded into the code commit)

Four reviewer agents plus a manual pass; the full table is design doc §9. The notable ones:

• Cache eviction during an operation. The reservation is the entry's write lock, which moka knows nothing about. If the cache evicted that entry mid-import (capacity pressure): a lookup lazily opened the directory being written; the eviction listener awaited the lock while moka awaits listeners inline from arbitrary cache ops, so that op hung and eventually every namespace load did; fork could deadlock on its own entry. Fixed with the in_flight pin, a pin-aware listener, and install publishing into a fresh entry when the locked one is gone (or handing over to destroy if that's why it's gone).
• Cleanup ordering. Directory removal was spawned and unordered against releasing the name, so a retry "right away" could race it (the tests had hidden this by polling). Now: atomic rename into .discarded/ + background delete + startup sweep; error paths await the cleanup, Drop is for cancellation only.
• MetaStore::remove's guards could leave the in-memory config behind on abort (ghost) → unconditional discard through the metastore's own queue.
• Flush → publish → unmark ordering, fork's temp dir marked, FreshDir::begin synchronous and fsynced.
• A rejected acquire left an empty cache entry that try_get_with then returned forever instead of loading the namespace — a retry after a restart made the namespace unreachable until eviction. Found while writing the tests; checkpoint had the same latent bug. Both fixed.
• MED/LOW: metastore lock-order inversion; shutdown no longer waits on an in-flight import; with() uses a non-registering lookup; publish cancellation-safe; log levels; doc fixes.

Full libsql-server suite under nextest: identical results to the base branch (local_sync_with_writes and the wall-clock-ratio replica_no_resync_on_restart fail under parallel load on both; both pass alone).

Accepted residues (design doc §9): the empty cache entry left after an abort (never served; healed on the next lookup or evicted), and a metastore rebuilt from the filesystem in the milliseconds between a namespace's flush and its unmark. Bottomless is not used in our deployment; this change is untested with it.

[tszymczyszyn-shopify]

Reviewer walkthrough

Pinned to bb446259f6. Stacked on #51; this PR is the last three commits: f51aca96ba (code and tests), 1f688cb9ca (design doc and admin API), bb446259f6 (diagrams). Suggested reading order below; ~45 minutes, a little more with the primer.

TL;DR

POST /v1/namespaces/:ns/create used to persist the namespace's config first and set the namespace up afterwards; nothing undid the config on failure and nothing at all ran on cancellation, so a failed import left a ghost (name taken, next request lazily opened an empty database). This PR makes the metastore flush the commit point and makes everything before it reversible — with two drop guards (one per layer), an on-disk marker for crashes, and a pin that keeps the operation safe from cache eviction.

Contract for callers: 2xx ⇒ complete. Anything else, including a lost connection ⇒ retry; already exists on the retry ⇒ the earlier attempt did complete.

Primer: the pieces this PR touches (if you don't know libsql-server's namespace code)

A namespace is one SQLite database served under a name (http://<name>.<host>). Four things cooperate to make one exist:

Piece	File	What it is
NamespaceStore	namespace/store.rs	The registry of loaded namespaces: a moka cache Name → Arc<RwLock<Option<Namespace>>>, bounded by --max-active-namespaces (default 100). Evicted namespaces are shut down and lazily reloaded from disk on the next request. with(name, f) is how every user request reaches a namespace; create, fork, reset, destroy are the admin operations.
MetaStore	namespace/meta_store.rs	Which namespaces exist and their DatabaseConfig: an in-memory map (configs, what exists() reads) backed by a SQLite table (namespace_configs). Writes go through a FIFO change queue processed on the blocking pool. Gotcha this PR works around: handle(name) inserts the name into the in-memory map as a side effect.
Configurators	namespace/configurator/{primary,schema,replica,fork}.rs	setup(name, RestoreOption) builds a Namespace from dbs/<name>/: directory, .sentinel (dirty-shutdown flag), replication logger (wallog), SQLite connection maker, and — for RestoreOption::Dump — the dump import from #51. cleanup removes a namespace's files.
Admin API	http/admin/mod.rs	POST /v1/namespaces/:ns/create resolves dump_url into a stream and calls NamespaceStore::create.

Two moka facts matter: (1) try_get_with(key, init) runs init at most once per key and makes concurrent callers wait for it — the base code relies on this to avoid loading a namespace twice; (2) moka 0.12 has no background thread: cache maintenance (including awaiting async eviction listeners) runs inline inside whatever cache operation trips it.

Before this PR, create did: metastore.store(config) (durable) → load_namespace → setup. The order is what this PR inverts.

Before / after

Who owns what

Live, editable board (Shopify-internal): https://drawr.quick.shopify.io/b/GNUB8o_56hhfFGxv

The happy path, with a concurrent lookup

sequenceDiagram
    autonumber
    participant A as Admin API
    participant S as NamespaceStore
    participant M as MetaStore
    participant C as Configurator
    participant FS as Filesystem
    participant U as User request (with foo)

    A->>S: create(foo, Dump, cfg)
    S->>S: Reservation::acquire — get_with entry, write_arc(), reject if Some or exists(foo), pin in in_flight
    S->>M: handle(foo); store(cfg, flush=false)
    Note over M: in memory only — exists(foo) is now true, nothing on disk
    S->>C: discard_incomplete(foo)
    Note over C,FS: removes dbs/foo only if it still carries .incomplete (a crashed earlier attempt)
    S->>C: setup(foo)
    C->>FS: FreshDir::begin — mkdir dbs/foo + .incomplete (sync)
    C->>C: logger, WAL, connections, dump import…
    U-->>S: with(foo): existing_handle → load_namespace
    Note over S,U: in_flight has foo → wait on the pinned entry's read lock (never a lazy load)
    C-->>S: Ok(Namespace)  (FreshDir.keep())
    S->>M: handle.flush()
    Note over M: COMMIT POINT — row written; a row now implies a complete namespace
    S->>S: publish → install(ns) into the locked entry, unpin, unlock
    S->>FS: remove dbs/foo/.incomplete (best effort)
    S-->>A: 2xx
    S-->>U: read lock acquired → Some(ns) → serve

Loading

Failure or cancellation

sequenceDiagram
    autonumber
    participant A as Admin API
    participant S as NamespaceStore
    participant M as MetaStore
    participant C as Configurator
    participant FS as Filesystem
    participant U as User request (with foo)

    A->>S: create(foo, Dump, cfg)
    S->>S: Reservation::acquire (lock + pin)
    S->>M: store(cfg, flush=false)
    S->>C: setup(foo)
    C->>FS: mkdir dbs/foo + .incomplete
    U-->>S: with(foo) → waits on the pinned entry
    alt import fails
        C->>FS: FreshDir::drop — rename dbs/foo → .discarded/foo-<uuid> (sync), delete in background
        C-->>S: Err
        S->>S: Reservation::abort() — awaited
    else client disconnects (future dropped)
        Note over C,S: FreshDir and Reservation are dropped during teardown
        C->>FS: FreshDir::drop — rename, same as above
        S->>S: Reservation::drop — cleanup handed to the blocking pool
    end
    S->>M: Discard(foo) via the change queue (unconditional, ordered after any queued store)
    S->>C: discard_incomplete(foo) — no-op if FreshDir already renamed it
    S->>S: unpin, then unlock
    S-->>A: error (only after the cleanup, on the Err path)
    S-->>U: read lock acquired → None; exists(foo) false → NamespaceDoesntExist
    Note over A: a retry now finds a free name and a free path

Loading

Lifecycle of a name, including crashes

stateDiagram-v2
    [*] --> Absent
    Absent --> Reserved: acquire (entry locked + pinned, config in memory)
    Reserved --> SettingUp: FreshDir::begin — dbs/foo + .incomplete
    SettingUp --> Committed: flush — row written
    Committed --> Published: install, unpin, unlock
    Published --> Complete: .incomplete removed
    Complete --> [*]

    Reserved --> Discarding: Err / drop
    SettingUp --> Discarding: Err / drop — dir renamed to .discarded/
    Committed --> Discarding: drop before publish — Discard deletes the row again
    Discarding --> Absent: unpin, unlock

    SettingUp --> MarkedOrphan: process dies
    Committed --> Published: process dies here — row + marker; serves after restart, marker dropped on load
    MarkedOrphan --> Absent: sweep at startup, or next create of the name (discard_incomplete)
    Published --> Complete: process dies before unmark — stale marker removed on next load

    note right of Discarding
        Order matters: config → directory → unpin → unlock.
        The name is unobservable until it is clean.
    end note
    note right of MarkedOrphan
        No row, so exists(foo) is false: invisible to clients.
        The marker is what tells it apart from a real namespace.
    end note

Loading

Eviction during an operation (commit 3)

sequenceDiagram
    autonumber
    participant S as NamespaceStore (create foo)
    participant K as moka cache
    participant L as eviction listener
    participant U as with(foo)

    S->>K: get_with(foo) → entry E; write_arc(E); pin E in in_flight
    Note over S: import running for minutes…
    K->>K: capacity pressure — evicts E
    K->>L: notify(foo, E)
    alt foo is pinned
        L->>L: spawn the shutdown (don't await E's lock inline — moka awaits listeners from arbitrary cache ops)
    else not pinned (a normal eviction)
        L->>L: await E.write() and shut down inline (serialises with a reload of the same name)
    end
    U->>S: load_namespace(foo): in_flight has foo → wait on E (not on the cache)
    S->>K: install: store.get(foo) ≠ E and exists(foo) → insert fresh entry F with Some(ns)
    Note over S: E stays empty — the spawned listener finds nothing to shut down
    S->>S: unpin, unlock
    U->>S: E is None → exists(foo) true → try_get_with → F → serve

Loading

If foo was destroyed meanwhile instead of evicted, install sees exists(foo) == false and hands the namespace to E, where destroy is waiting for it ("created, then destroyed").

Suggested reading order

1. The on-disk protocol (small, self-contained)

• namespace/incomplete.rs: INCOMPLETE_MARKER, create_marked_dir/mark_incomplete (fsynced), discard_dir (rename into .discarded/, background delete), discard_if_incomplete, sweep — with unit tests at the bottom (tombstone with an open file, sweep, FS recovery skipping marked dirs).
• FreshDir: the configurator's guard; synchronous begin, Drop ⇒ discard_dir. Used by primary and schema setup; fork marks its temp dir.
• Startup sweep call in lib.rs.

2. The reservation (the heart of the change)

• Reservation: acquire (prefers a pinned entry; invalidates the empty entry it created when rejecting), publish, abort (awaited on error paths), Drop (cancellation only), take_cleanup with the config → directory → unpin → unlock order.
• create / create_reserved and fork / fork_reserved: same shape; has_shutdown re-checked before the commit; publish before unmark.
• MetaStore: ChangeMsg::{Update, Discard}; discard is unconditional (unlike remove, whose linked-DB guards would leave a ghost); existing_handle is the non-registering lookup; the flush branch of process no longer holds configs across try_process (lock order vs remove).

3. Eviction safety

• InFlight pin map + unpin, pin-aware eviction listener, install, load_namespace (pinned entry first; empty-and-unlocked entry treated as a miss), destroy (prefers the pinned entry), shutdown (skips pinned entries).

4. Tests

• Store unit tests (store.rs::test): a real NamespaceStore with a channel-fed streaming import — lookup waits across an eviction; lookup sees failure across an eviction; destroy during creation; retry the instant the error returns (no polling); abort mid-import (JoinHandle::abort); discard vs remove's guards.
• Integration (tests/namespaces/lifecycle.rs, dumps.rs cancel tests): failed create leaves no trace (both importers + schema configurator), cancelled create (cancel fires only once dbs/foo/.incomplete exists), concurrent requests wait, crash remnant discarded, unmarked dir untouched, marker on a published namespace ignored (creates before any lookup after a restart), single-namespace dump create rejected.
• 9 of the original 11 integration tests fail on the unfixed code; the two that pass pin behaviour that must hold before and after.

Behaviour changes visible to clients

• Failed create ⇒ GET …/config is 404 and users get "doesn't exist" (was: 200 / "no such table").
• Single-namespace mode: create default from a dump while it exists ⇒ already exists (was: 200, dump silently skipped). Plain create default stays a config upsert.
• <db-path>/.discarded/ appears; it holds renamed directories until the background delete or the startup sweep removes them.

Where reviewer judgment is most valuable

1. The cleanup order in Reservation::take_cleanup — is there any interleaving where the name becomes observable before it is clean?
2. install's three-way decision (same entry / destroyed / evicted) and load_namespace's empty-entry healing: do they cover every way an entry can go stale?
3. MetaStore::discard bypassing remove's linked-DB guards — correct for an unpublished namespace; is there any path where it runs for a published one? (It only runs from a Reservation, whose acquire rejects existing names.)
4. The pin-aware eviction listener: inline for normal evictions, spawned for pinned — any entry that is long-locked but not pinned?
5. Whether the tombstone directory and startup sweep are acceptable operationally (disk use is bounded by the background delete; sweep logs each leftover).

Explicitly out of scope

• Bottomless: not used in our deployment; this change is untested with it.
• Two accepted residues (design doc §9): an empty cache entry after an abort (never served; healed on the next lookup or evicted), and a metastore rebuilt from the filesystem in the milliseconds between a namespace's flush and its unmark.
• An import timeout; idempotency keys / async create jobs.

Full rationale, failure matrix and the review table: docs/ATOMIC_NAMESPACE_CREATE_DESIGN.md.