Skip to content

feat(protocol): publish Git and Xet state through capsules - #208

Open
forhappy wants to merge 126 commits into
mainfrom
feat/request-minimal-protocol
Open

forhappy wants to merge 126 commits into
mainfrom
feat/request-minimal-protocol

Conversation

@forhappy

@forhappy forhappy commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Latest qualification update

  • Fresh RustFS/Kubernetes run v2-k8s-5000-20260916-codex6 used installed binary commit 7f4973dc59f and a fresh bucket.
  • Seed: 244.174 s / 11 requests; checkpoint: 351.227 s / 10 requests; initial lazy clone: 102.980 s / 8 requests.
  • All first 500 replay pushes succeeded: 481.33 ms mean, 430 ms p50, 808 ms p95, 1.011 s p99, 1.914 s max; 9.012 requests mean, 8-request p95, 44-request max on batched compaction.
  • The first 500-commit fetch exceeded ten minutes at one saturated core and about 2.1 GB peak physical footprint, so the run was stopped and retained as negative evidence.
  • Root cause boundary: the reader must download and hash the complete 1.18 GB checkpoint object before using its visibility/locator metadata. Before another 5,000-push run, the checkpoint hard cut must root-authenticate one contiguous control suffix and range-load only that suffix plus selected pack ranges.
  • d1b85c7f fixes the Rust 1.98 compaction lint; 8c6a215c updates the protocol, request formulas, parity ledger, and retained qualification evidence.

Summary

  • hard-cut ordinary Git and Xet publication, clone/fetch/pull, remote snapshot inputs, repack, fsck, import, mirror reconciliation, repository GC, and HTTP/server composition to protocol v2
  • restore later partial-clone lazy-object recovery through the line-oriented helper using one authenticated v2 view, visible-ref authorization, selected pack generation, and atomic promisor installation
  • replace the repository-wide foreground publication root with independently mutable per-ref capsule heads
  • commit single-ref pushes with one ref-head CAS and multi-ref pushes with prepared heads plus one attempt-unique activation-record CAS
  • keep xorbs, shards, and LFS payloads outside capsules as immutable dependencies and verify their complete closure before ref visibility
  • make foreground publication history-flat: each push appends one leaf capsule without reading or rewriting older capsules
  • checkpoint authenticated repository views in background after 32 visible capsules, force a safety checkpoint at 56, and fail closed at the hard 64-entry frontier
  • preserve concurrently published suffixes when a checkpoint lags active writers
  • restore authenticated history checkpoints as new generations under a rotated per-ref authority epoch while retaining the displaced state and append-only xorb/shard catalog

Correctness model

  • disjoint refs share neither a mutable object nor a publication lock; same-ref writers serialize at the destination-head CAS
  • multi-ref transactions prepare every participating head, then one activation-record CAS selects all-old or all-new visibility
  • readers authenticate exact head bodies, provider versions, activation records, checkpoint bases, and per-ref predecessor chains before applying capsules
  • ref heads record the checkpoint transaction immediately preceding a trimmed suffix; readers accept either an untrimmed pre-checkpoint frontier or the explicitly rebased suffix and reject any gap
  • Git packs, xorbs, shards, and reachable LFS objects must be durable and verified before a capsule can publish refs
  • the compacted root is checkpoint, symbolic HEAD, GC, and maintenance authority, not a foreground push mutex

Requests, latency, and checkpointing

The prior v1 path used roughly 60-69 object-store attempts per incremental push. With a captured v2 view, the single-ref writer core is history-flat:

  • checksum-qualified: ref-head GET + immutable leaf PUT + ref-head CAS + root-epoch GET = 4 successful requests
  • mandatory readback: the same path plus leaf GET = 5 successful requests
  • end-to-end transport totals additionally include root/ref discovery, not-found attempts, namespace gates, leases, retries, and external xorb/shard/LFS work

Commit count does not change this foreground request count. Background checkpointing is outside the clean push budget. The server starts it at 32 visible capsules; if it falls behind, receive synchronously checkpoints at 56 so the 64-entry hard bound remains a fail-closed last resort. Checkpoint publication may lag concurrent pushes: the next writer drops only the exact captured prefix and retains the suffix.

The current checkpoint carries all verified Git packs. Shared pack consolidation and checkpoint byte-growth qualification remain release blockers; this PR does not claim that the present checkpoint representation is the final byte-optimal form.

Earlier RustFS 5,000-push results used the superseded binary-run foreground algorithm. They remain useful baseline evidence but do not qualify the leaf/checkpoint implementation. A fresh Kubernetes replay with periodic fetch/pull, final independent clone, strict fsck, request counters, bytes, and p50/p95/p99 latency is still required.

Xet and LFS behavior

Xorbs and shards remain separate immutable content-addressed objects; capsules contain authenticated catalogs and references, not large payload bytes. The canonical v2 publisher verifies or uploads reachable Xet and LFS dependencies before ref publication. Pointer clone, checkout, smudge, hydrate, prefetch, diff, and mount share one generation-pinned catalog and reconstruction path.

Verification

  • cargo test -p crab-metadata -p crab-read -p crab-write -p crab-http-server --locked: passed
    • metadata: 382 passed, 1 ignored in the broad run
    • read: 175 passed
    • write: 20 unit tests plus catalog/generation/journal integrations passed
    • HTTP server: 101 library tests passed, 3 credentialed live tests ignored; 4 binary tests passed
  • cargo test -p crab-remote -p crab-http-server --locked: passed
    • remote: 8 unit and 13 publication integration tests passed
    • HTTP server: 101 library and 4 binary tests passed
  • strict all-target Clippy passed for crab-metadata, crab-read, crab-write, and crab-http-server
  • checkpoint tests prove exact-prefix compaction, a checkpoint lagging two concurrent pushes, suffix preservation, and foreground safety checkpointing before the hard bound
  • writer tests prove leaf publication does not read/rewrite history and stays at no more than five successful object operations per push through 64 appends; the final root GET rejects a publication if restore rotated ref authority across its CAS

V1 parity boundary

This PR materially advances v2 parity but does not declare full v1 product parity. Remaining release blockers include:

  • checkpoint Git-pack consolidation and current-format 5,000-push/fetch/clone performance qualification
  • managed/active-active external consensus and hosted-provider concurrency qualification
  • S3 gateway reads and mutations on v2 packs, refs, and transactions
  • replica readiness, repair, failover/failback, and active-active reconciliation from authenticated v2 closure
  • tiering/archive restore integrated with v2 reachability, hydration, mount, and GC
  • complete FUSE/NFS range, cache, cancellation, restored-tier, and provider/platform qualification
  • browser/HTTP hosted-provider and load qualification, including checkpoint byte growth
  • transaction/ref history selectors, complete doctor reporting, live crash qualification of atomic restore-as-new, and verified one-way v1-to-v2 migration
  • direct LFS endpoint modes and repository-prefix backup/delete/restore discovery
  • exhaustive audit of every shipped CLI command, remote-helper verb, HTTP route, gateway operation, worker, and maintenance task

crab/docs/design/capsule-xorbs-shards.md now requires every shipped entry point to map to v2 proven, intentionally removed, or release blocker. Replica, tiering, mount, and browser are specified as one pinned-view cross-surface contract with combined fault scenarios; no missing surface may silently fall back to v1.

Generation-owner parity

  • crab metadb owner now selects a present v2 root exclusively, fails closed on corrupt v2 authority, and never reads or synthesizes a v1 manifest after v2 selection.
  • steady-state polling fingerprints the root and transaction-consistent ref heads without downloading stable capsule, checkpoint, or pack bodies; eligible work checkpoints the already authenticated view with exact-root CAS.
  • focused read/owner tests and the HTTP checkpoint/next-push integration test pass.
  • the container crash qualifier now watches both legacy pack objects and v2 capsule publications; the earlier image-job failure was its v1-only observation path, while the v2 receive itself completed.

Recovery authority parity

  • recover apply --rebuild-file-index now selects repository authority before mutation. A present v2 root is opened exclusively and planned file-to-shard mappings are checked against the complete authenticated pointer catalog.
  • the v2 path acquires no legacy metadata writer and creates no file_index_db; only a repository with no v2 root uses the existing shard-to-SlateDB rebuild. Corrupt v2 authority fails closed.
  • focused v2 no-legacy-object tests and all 34 recovery unit tests pass.

Diagnostics and rebuild parity

  • crab metadb diagnose selects a present v2 root exclusively. Its default probe reads only root/ref activity; --deep authenticates the full capsule/checkpoint/catalog/visibility view, reads and verifies every referenced shard and xorb, reconstructs the Git closure, and rejects a result if repository activity changed during the scan.
  • crab metadb rebuild now uses the same strict authority selection. It verifies the complete external pointer closure and Git closure from one pinned view, publishes a checkpoint only when a capsule frontier needs compaction, and uses exact-root CAS. Re-running against an already checkpointed view is verification-only, so it does not create a hot sequence of redundant checkpoints.
  • V2 rebuild never opens or creates legacy SlateDB state. Output distinguishes authenticated Git objects from records actually written and reports protocol, checkpoint publication, and verified catalog counts.
  • Focused proof passed: eight rebuild-related crab tests, the capsule-view accounting test, the real-pack checkpoint/next-push integration test, formatting, diff validation, and cargo check -p crab --locked --features gix-transport.

Shard-compaction parity

  • crab compact now selects a present capsule-v2 root before the legacy shard-list and fails closed on corrupt v2 authority.
  • v2 derives its source shards and retained files from one authenticated repository view, strips foreign shared-shard file recipes, retains each selected file complete Xorb metadata, and rejects missing or duplicate coverage.
  • replacement shards and GC closures use the configured storage scope; every replacement shard, Xorb body, and chunk catalog is verified before visibility.
  • the replacement catalog is published in a complete Git checkpoint against the exact captured root. Concurrent per-ref updates remain as a suffix, while a root-CAS loser leaves only immutable candidates.
  • source shards remain protected by monotonic registry roots and checkpoint history until later proven cleanup. V2 creates no legacy manifests/shard-list.
  • proof passed: all 16 compact unit/integration tests, the checkpoint/next-push integration test, feature-enabled cargo check, formatting/diff checks, and the 398-page/4,307-fragment docs link check. The strict Clippy attempt is blocked by the pre-existing crab-read double_must_use warning at crates/crab-read/src/capsule_protocol.rs:135; non-strict Clippy completes with the existing workspace warning inventory.

Xorb-optimization parity

  • crab optimize xorbs now selects a present capsule-v2 root as the exclusive authority, derives source xorbs only from the authenticated live file/shard closure, and fails closed on corrupt v2 state. Only repositories without a v2 root use the legacy global inventory and manifest path.
  • source and destination xorbs use the configured repository/global storage scope. Destination creation is immutable and validates logical Xorb identity, payload digest, and every chunk when an object already exists.
  • v2 reconciliation rewrites only authenticated shards, strips foreign records from shared shards, retains complete file coverage, and verifies the full replacement shard/Xorb catalog before publication.
  • the replacement catalog is published as a complete checkpoint against the exact captured root. Concurrent push suffixes are retained after CAS retry, legacy manifests and SlateDB are not touched, and old closures remain registered for conservative later GC.
  • proof passed: all 58 Xorb optimizer tests, including real v2 Xorb/shard/checkpoint publication; feature-enabled Cargo check; formatting and diff checks; non-strict Clippy with the existing warning inventory; and the 398-page/4,307-fragment docs link check.

Destination storage-class parity

  • --output-class now validates and canonicalizes provider-native S3, GCS, and Azure values before a run starts, and applies StorageClass on each newly created destination Xorb.
  • immutable reuse never rewrites an existing content address merely to change its class; its full Xorb body is authenticated and its existing class is retained. Local stores omit cloud-only attributes.
  • proof passed: all 208 crab-storage tests, all 61 Xorb optimizer tests, feature-enabled Cargo check, touched-surface Clippy, formatting/diff checks, and docs-link validation.

Qualification corrections

  • strict split-crate and SDK Clippy now pass after removing the redundant must_use annotation from a fallible capsule-view accessor.
  • unsupported-root fault injection now uses a permanently unsupported version instead of the current v3 envelope version, so mirror check/apply must fail closed before replay.
  • mirror namespace qualification now removes and restores the v2 root authority; it no longer assumes a hard-cut repository contains v1 layout or manifest objects.
  • the container SIGKILL qualifier now observes a non-LFS Git pack while it is still in local staging, freezes the server, and kills that exact in-flight receive. Publication-phase response-loss and root/head boundary behavior remain covered by the dedicated receive fault matrix.
  • local proof passed: exact split-crate and SDK all-feature -D warnings checks, 14 capsule read tests, six qualification-log tests, Python and shell syntax, pinned ShellCheck, formatting, and diff validation.

Classic shallow-fetch parity

  • the capsule-v2 helper now advertises shallow and handles classic-helper shallow clone, absolute and relative deepening, unshallow, and follow-tags through the canonical authenticated upload-pack planner
  • generated packs are self-contained because the classic helper has no negotiated have set; installation is serialized with the shared per-clone lock, ref tips are validated, and .git/shallow is replaced atomically
  • a new end-to-end helper test publishes a three-commit capsule history plus annotated tag, then proves depth-one omission, relative deepening, tag inclusion, unshallow, and depth-zero full-fetch behavior in real Git object databases
  • proof passed: all 136 remote-helper tests, focused depth-zero/shallow lifecycle rerun, cargo check -p crab --lib --locked, formatting, and diff validation. Strict all-target Clippy on this host is blocked by the existing repository-wide Rust 1.97 lint inventory, not this surface.

Container recovery gate follow-up

  • the earlier image job reached the deterministic in-flight staging boundary and recreated a healthy server, but Compose aborted when the shared-network proxy became temporarily unhealthy before the publication-lease recovery window expired
  • the qualifier now recreates the server and proxy without Compose early-abort semantics, polls the actual public repository API for the full recovery budget, and still fails with container status and logs if the data path never recovers
  • shell syntax and the workflow-pinned ShellCheck 0.10.0 image pass locally

Root-scoped server backup parity

  • route hosted-server xorbs, shards, and ref-registry state beneath the configured storage root, matching the workload-identity and backup boundary
  • use the same scoped layout for create, adopt, and runtime materialization; adoption remains a v2 hard cut
  • restore the configured root into a distinct RustFS bucket, hash every object, and prove shared .crab state is copied rather than read from the source bucket
  • server library: 173 passed, 4 ignored; server CLI: 11 passed; one repeat broad run hit the pre-existing free-space probe flake, which passed immediately in isolation

Fail-closed server adoption

  • authenticate the complete capsule-v2 view before publishing an adopted repository into the server catalog
  • deeply verify every scoped shard and xorb body and validate every embedded Git pack in a temporary object database
  • leave the catalog unchanged when any dependency is missing or corrupt
  • focused catalog suite: 10 passed

Complete parity closure ledger

  • adoption proof and ongoing integrity are now separate contracts: adoption deeply authenticates Git packs plus the full shard/xorb closure, while post-adoption shard, xorb, and LFS loss remains a bounded background-scrub release blocker
  • the 10-second readiness endpoint and ordinary fetch must not redownload large-file closure; foreground content reads continue to fail closed
  • backup/export evidence must include every per-ref head and activation record as well as retained history, Xet/LFS closure, GC roots, and application state
  • every acceptance artifact must bind its exact root digest and ref-head versions and prove many-ref writers do not regain a shared mutable hot object
  • parity closes only after the full authority, dependency durability, reader, operations/migration, and production-provider ledger passes; RustFS proof does not waive S3/GCS/Azure receipts

Docs link validation: 398 pages and 4,307 fragments passed.

Complete adoption dependency proof

  • adoption now performs one bounded all-ref Git walk after authenticating and installing the capsule-v2 pack closure
  • the shared scanner classifies both Crab and Git LFS pointers, byte-verifies every large blob omitted from pointer retention, and never accepts a partial result after cancellation or a resource-limit failure
  • every reachable Crab pointer must match an authenticated catalog file recipe; every catalog shard and xorb body is hash-verified; every distinct reachable LFS origin object is streamed and hash-verified
  • conflicting LFS sizes, missing content, corrupt content, or an unrepresented reachable Crab pointer leave the server catalog unchanged
  • Git batch verification uses a file-backed request stream, avoiding stdin/stdout pipe deadlock when a large first blob precedes a large request list
  • foreground push, fetch, and ordinary readiness paths do not run this deep proof; the bounded post-adoption scrub remains a release blocker for dependencies that disappear later

Proof: crab-git library 269 passed/1 ignored; capsule read 14/14; server adoption 13/13; maintenance 8/8; strict all-target Clippy passed for touched crates (server uses the existing result_large_err allowance); docs link validation passed at 398 pages/4,307 fragments.

Post-adoption dependency scrub

  • every server starts a background deep proof immediately and repeats it hourly without entering the foreground push, fetch, or 10-second readiness path
  • each repository has a three-minute cooperative budget; at most two run concurrently through the existing maintenance admission pool
  • the proof authenticates the exact v2 state digest, Git closure, reachable Crab pointers, catalog shards/xorbs, and reachable LFS bodies, then rechecks repository activity before publishing status
  • GET /integrityz on the private management listener returns 202 before a current proof, 200 with exact proof counts and digest after success, and 503 after dependency loss while retaining the prior complete proof
  • cancellation now drains the blocking Git verifier before its temporary object database or admission permit is released
  • focused deletion proof, endpoint/readiness separation, 14 capsule-reader tests, and the complete server library (179 passed, 4 ignored) pass; strict touched-crate Clippy and the 398-page/4,307-fragment docs link check pass
  • hosted-provider request/byte accounting and multi-pod scrub amplification remain explicit production qualification gates; this work adds no foreground push requests

Deployment-wide scrub ownership

  • one renewable object-store lease elects a single deep-scrub owner for the whole server deployment; healthy replicas reuse one bounded aggregate report instead of repeating shard, xorb, LFS, and Git closure reads
  • the report is operational evidence only, binds every repository UUID and placement generation, rejects unknown or malformed fields, and never becomes serving authority
  • publication uses strict create or ETag CAS after the scan, so an expired lease owner cannot overwrite a successor report; contenders retry without starting a duplicate scan
  • replicas schedule from the report remaining lifetime, preserving the one-hour proof-age bound even when they discover an existing report midway through its lifetime
  • a focused real-capsule test proves fresh-report reuse suppresses a deep scan, lease contention does not duplicate it, a stale report detects a deleted LFS body, and a stale ETag cannot overwrite current evidence
  • focused integrity tests: 2 passed; strict all-target server Clippy passed; the complete server library previously passed 179 tests with 4 ignored apart from the known free-space probe flake, which passed immediately in isolation; docs links passed at 398 pages and 4,307 fragments
  • foreground push request count is unchanged: the scheduler adds zero requests to push; exact hosted-provider scrub request/byte totals and lease-loss fault injection remain production qualification gates

Complete v2 authority restore qualification

  • the RustFS cold-restore gate now forces one atomic branch-plus-annotated-tag publication before backup, ensuring the fixture contains independently mutable ref heads and multi-ref activation evidence
  • source inventory must contain the v2 root, at least two ref heads, capsule, transaction record, immutable committed marker, and exact LFS body, while v1 root authority is forbidden
  • every object under the configured server root is copied to a separate bucket, matched by key and size, and hashed byte-for-byte as before
  • the independently started restored server must expose the exact commit and annotated-tag object IDs; the fresh clone now also passes git fsck --strict
  • local proof passed: bash syntax, workflow-pinned ShellCheck 0.10.0, diff validation, and the 398-page/4,307-fragment documentation link check
  • this strengthens complete-root disaster-recovery evidence; the authenticated per-repository export inventory, external Xet closure and hundreds-of-refs qualification remain explicit release gates

Scrub lease-loss fencing proof

  • a deterministic object-store fault test pauses a real reachable-LFS dependency read while the deployment scrub lease is held
  • the test replaces that lease with a successor, observes the old owner renewal CAS fail, and waits for cooperative cancellation before releasing the blocked read
  • the old owner returns cancelled without publishing an aggregate report, and its cleanup cannot tombstone the successor lease
  • the complete integrity suite now passes 3/3; strict all-target server Clippy and the 398-page/4,307-fragment documentation link check pass
  • provider-level lease-loss behavior and exact scrub request/byte accounting remain live qualification gates; foreground push request count remains unchanged

@forhappy forhappy changed the title perf(protocol): cut object-store push requests below five average feat(protocol): publish Git and Xet state through capsules Sep 15, 2026
@forhappy
forhappy force-pushed the feat/request-minimal-protocol branch 3 times, most recently from 1962719 to 9850e91 Compare September 16, 2026 08:40
@forhappy

Copy link
Copy Markdown
Contributor Author

Follow-up qualification and hardening (commit 9393caf):

  • Terminal upload-pack now retains delta bases external only when the request is unfiltered, non-shallow, non-deepen, advertises thin-pack, and every client have is authenticated in the pinned v2 visibility plan. OFS deltas are rewritten to REF_DELTA; the full dependency sort/materialization path remains for filtered, shallow, incomplete, and legacy requests.
  • Catalog-selected objects retain authenticated physical pack order; legacy materialized selection keeps canonical OID ordering.
  • Fresh release binary v2-tiny-external-final-20260917 on RustFS: two incremental pushes 304–327 ms at 8 requests each; two fetches 225–247 ms at 9 requests; final clone 501 ms at 18 requests; matching tip and strict fsck passed.
  • Focused tests: crab-remote-git 133, crab-read 188, upload-pack wire 35; cargo check -p crab, release build, architecture gates, and format/diff checks pass.

The Kubernetes 5,000-commit stress artifact remains explicitly negative at the 500-commit fetch/repack boundary because the interrupted pre-fix repository state still requires a large historical pack; it is not being reported as parity proof. Hosted-provider, multipart, replica/tiering, mount/browser, S3 gateway, migration/recovery, and backup/restore rows remain release gates.

@forhappy

Copy link
Copy Markdown
Contributor Author

Parity follow-up (commit d6431f8): removed the stale client rejection for protected pushes carrying a v2 mirror-plan ID. The plan ID is already authenticated in CapsuleTransaction::for_plan; the auth-server capsule publisher commits the same transaction-scoped capsule plan receipt. The protected capsule receive test now exercises that planned transaction, verifies the receipt, and retries successfully. cargo test -p crab-auth-server --lib (101), capsule-push tests (9), and the release build pass.

@forhappy

Copy link
Copy Markdown
Contributor Author

Added external_thin_subset_pack_keeps_the_proven_base_outside_the_pack (commit 36f2ea6). It generates the public external-base thin-pack API from a real REF_DELTA fixture, verifies the one-object thin pack with strict git index-pack --fix-thin, and passes.

@forhappy

Copy link
Copy Markdown
Contributor Author

Documentation follow-up (commit 0233c25): the main capsule publication design now records the authenticated external thin-base rule and protected mirror-plan receipt path alongside the parity inventory and RustFS evidence.

@forhappy
forhappy force-pushed the feat/request-minimal-protocol branch from 0233c25 to 6ad3b0b Compare September 17, 2026 07:53
@forhappy

Copy link
Copy Markdown
Contributor Author

V1 parity pass

Implemented and pushed in 6ad3b0b0e9a (rebased onto current main):

  • Legacy read replicas now use the v1 manifest/index/object readiness contract only when v2/root is absent; a present or corrupt v2 root never downgrades. The resolver accepts verified legacy manifests, and tests cover both acceptance and corrupt-root rejection.
  • migrate import, migrate export, and adopt --rewrite-history now use the built-in verified fast-export/fast-import engine with v2 Xet staging/hydration. Shared Git blobs are converted inline only for selected paths; ref rollback is attempted on post-import failures, checkout failures are surfaced, and staging is closed before success.
  • Remote snapshot download/export, mount, hydrator, browser/HTTP, protected publication, and checkpoint readers carry one authenticated v2 view and immutable pointer catalog through reconstruction.
  • Documentation now contains a complete v1 product-parity inventory, cross-surface contracts, closure order, and Level-3 acceptance gates.

Proof after rebase:

  • cargo check -p crab --locked passed.
  • cargo test -p crab --locked --lib: 4,250 passed, 0 failed, 3 ignored.
  • cargo fmt --all -- --check, git diff --check, and python3 crab/scripts/check-architecture-gates.py passed.

Remaining release blockers are intentionally explicit: hosted-provider checksum/multipart and 5,000-commit current-format replay; managed replica failover/repair; tier/archive restore; mount range/cancellation/unmount; browser/HTTP load and fault matrix; S3 gateway operation/concurrency/restart matrix; lifecycle/workflow/admin inventory; backup/restore export inventory; and migration fault/resume/provider plus older-Git/interrupted/adversarial qualification. Full v1 production parity is not claimed until those Level-3 gates pass.

@forhappy

Copy link
Copy Markdown
Contributor Author

Parity closure update (c2d87d8):

  • Hydrate and remote mount now share one restore-availability adapter. It is built from the resolved physical store identity, so managed and replica read views target the bucket that owns the authenticated v2 catalog instead of reconstructing a provider from the logical crab:// URL.
  • --no-restore does not construct a cloud restore client, but archived shard/xorb reads still fail closed with the typed archive-class admission error.
  • A standalone crab:// mount now refuses to start if its authenticated v2 read context cannot be built; it no longer starts with stub readers and defers the failure until first pointer access. Local Git-native mount fallback is unchanged.
  • Design matrix and closure notes are updated in crab/docs/design/capsule-xorbs-shards.md.

Local proof after this change:

  • cargo test -p crab --locked --lib: 4,254 passed, 0 failed, 3 ignored.
  • cargo test -p crab --locked --lib cmd::mount: 120 passed before the final guard, plus the two new fail-closed/fallback tests passed individually.
  • cargo check -p crab --locked, cargo fmt --all -- --check, git diff --check, and python3 crab/scripts/check-architecture-gates.py all pass.

This closes the local wiring gap, but is not a claim of complete v1 production parity. Release gates remain: live S3/GCS/Azure lifecycle and restore behavior; replica readiness/failover/repair; restored-content verification; the full FUSE/NFS range/cache/cancellation matrix; browser and smart-HTTP load/fault coverage; S3 gateway restart/concurrency/request-count coverage; and migration, backup inventory, and delete/restore qualification on populated v1/v2 repositories.

@forhappy

Copy link
Copy Markdown
Contributor Author

Follow-up test hardening: the focused mount module now passes 122/122. I also serialized the unmount test's HOME override through the existing test guard; this removes a process-global HOME race that could make local_pipeline_config_rejects_active_cache flaky when mount tests ran in parallel. This is test-only and does not alter local Git-native fallback behavior.

@forhappy

Copy link
Copy Markdown
Contributor Author

Final local rerun after the test-only race fix: cargo test -p crab --locked --lib --quiet passed 4,254 tests (3 ignored) in 99.51s; cmd::mount passed 122/122. No working-tree changes are pending other than pre-existing generated Python __pycache__ directories, which were not added.

@forhappy
forhappy force-pushed the feat/request-minimal-protocol branch from 4b94ade to cc8700f Compare September 18, 2026 05:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant