Repository navigation
[fence 2/9] libsql-server: fence model, durable state and fail-closed recovery - #42
Draft
tszymczyszyn-shopify wants to merge 3 commits into
Draft
tszymczyszyn-shopify wants to merge 3 commits into
tszymczyszyn-shopify wants to merge 3 commits into
Conversation
Add the I/O-free core of the namespace fence described in docs/NAMESPACE_FENCE.md: - state.rs: roles, fence states (including the non-durable UNFENCED, ABSENT and derived UNKNOWN_UNAVAILABLE), operation classes and the permission matrix. - outcome.rs: stable outcome codes, their admin HTTP, user HTTP, Hrana, gRPC and proxy mappings (fence denials are never 5xx, 429 or gRPC UNAVAILABLE), bounded detail reasons and FenceError. - command.rs: fence commands and requests, and the canonical SHA-256 request fingerprint over everything except command_id. - record.rs: fence records, command receipts and the on-disk marker, with a strict protobuf encoding (proto/namespace_fence.proto) whose decoder rejects unknown versions, enum values, malformed ids and self-contradictory records instead of defaulting. - transition.rs: the pure apply() and complete_drain() functions. Replay and fingerprint conflicts are decided before owner, role, state and revision; TARGET_WRITABLE has no reverse transition; adoption changes only the owner. Nothing outside the module uses it yet; the metastore, controller and protocol layers follow. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
Add the additive `namespace_fences` and `namespace_fence_receipts` tables (created with --enable-namespace-fence, loaded and enforced whenever they exist) and run every fence command as a compare-and-swap in one BEGIN IMMEDIATE metastore transaction: read the record, the command's receipt and the config row, decide with the pure transition function, then commit the record, the receipt and the legacy block_* mirror together. The revision is stored, so it survives restart. Stored state is read strictly: an unknown format version, an undecodable payload, a revision column that disagrees with its payload, or a per-namespace `.fence` marker that is ahead of the metastore makes the namespace UNKNOWN_UNAVAILABLE instead of guessing. The marker is written after each commit (before it, for target creation) and rewritten on load when it fell behind. Ordinary config writes and deletes now take BEGIN IMMEDIATE and refuse while the fence denies lifecycle operations, and a config write that failed to persist is no longer published to the in-memory config. Receipts of finished operations are pruned after a retention period. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
When namespace fences are in use (the flag is on, the fence tables exist, or a namespace directory holds a fence marker), a namespace whose state startup cannot establish is registered UNKNOWN_UNAVAILABLE instead of being skipped or given a default config: - an undecodable config row, fence row or marker; - a directory with a marker that the metastore has no trustworthy record for: filesystem recovery, a metastore rebuilt by destroy_on_error, a metastore restored from an older backup or without fence tables, and a target whose creation was interrupted. Such a namespace is refused by lookups, config writes and deletes with FENCE_STATE_UNAVAILABLE, is reported by InspectFence, and is settled only by a fence command that commits for it. MetaStore::handle() no longer default-creates on read paths: the new non-creating MetaStore::lookup serves NamespaceStore::with, fork's source and ATTACH authorisation. handle(), used only by creating paths, refuses unavailable names and names without a config whose directory holds a marker. Filesystem recovery skips marked directories, destroy_on_error moves the broken metastore aside instead of deleting it, and a fence row for a namespace name that cannot be decoded stops startup. Servers that never used fences recover as before. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
This was referenced Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the
namespace::fencemodule with the fence types, stable outcome codes and the pure transition function (replay and fingerprint checks before owner, role, revision and transition checks), plus thenamespace_fence.protorecord encoding.Persists fences in the metastore: additive tables, a compare-and-swap on a restart-stable revision inside
BEGIN IMMEDIATE, command receipts with retention, the on-disk marker, and the config-write interlock. Makes metastore recovery fail closed when a fence may be involved: non-creatingMetaStore::lookup, a fallibleMetaStore::handle, marker-aware filesystem recovery, anddestroy_on_errorrenaming the metastore aside instead of deleting it.Fence activation and table creation are behind
--enable-namespace-fence(default off), while existing durable fences remain enforced with the flag off. The defensive metastore plumbing and config-write ordering in this piece apply generally, as called out below.Review focus
Metastore changes that apply to everyone, not only fenced namespaces: config writes now use
BEGIN IMMEDIATE, a config whose persist failed is no longer published in memory, andMetaStore::handlecan now fail. The fence module carries a transitional#![allow(dead_code)]until its callers land later in the stack.Commits
Stack
Part 2 of 9, based on
namespace-fence/0-contract. Retargeted from #35 with no feature change: applied in order, the 9 PRs carry #35's fence diff (stable patch ID70d97d6a) onv0.9.30-shopify-patches. Review and land bottom-up, restacking after each squash or rebase merge.