Conversation
1962719 to
9850e91
Compare
|
Follow-up qualification and hardening (commit 9393caf):
The Kubernetes 5,000-commit stress artifact remains explicitly negative at the 500-commit fetch/repack boundary because the interrupted pre-fix repository state still requires a large historical pack; it is not being reported as parity proof. Hosted-provider, multipart, replica/tiering, mount/browser, S3 gateway, migration/recovery, and backup/restore rows remain release gates. |
|
Parity follow-up (commit d6431f8): removed the stale client rejection for protected pushes carrying a v2 mirror-plan ID. The plan ID is already authenticated in |
|
Added |
|
Documentation follow-up (commit 0233c25): the main capsule publication design now records the authenticated external thin-base rule and protected mirror-plan receipt path alongside the parity inventory and RustFS evidence. |
0233c25 to
6ad3b0b
Compare
V1 parity passImplemented and pushed in
Proof after rebase:
Remaining release blockers are intentionally explicit: hosted-provider checksum/multipart and 5,000-commit current-format replay; managed replica failover/repair; tier/archive restore; mount range/cancellation/unmount; browser/HTTP load and fault matrix; S3 gateway operation/concurrency/restart matrix; lifecycle/workflow/admin inventory; backup/restore export inventory; and migration fault/resume/provider plus older-Git/interrupted/adversarial qualification. Full v1 production parity is not claimed until those Level-3 gates pass. |
|
Parity closure update (c2d87d8):
Local proof after this change:
This closes the local wiring gap, but is not a claim of complete v1 production parity. Release gates remain: live S3/GCS/Azure lifecycle and restore behavior; replica readiness/failover/repair; restored-content verification; the full FUSE/NFS range/cache/cancellation matrix; browser and smart-HTTP load/fault coverage; S3 gateway restart/concurrency/request-count coverage; and migration, backup inventory, and delete/restore qualification on populated v1/v2 repositories. |
|
Follow-up test hardening: the focused mount module now passes 122/122. I also serialized the unmount test's HOME override through the existing test guard; this removes a process-global HOME race that could make |
|
Final local rerun after the test-only race fix: |
4b94ade to
cc8700f
Compare
Latest qualification update
v2-k8s-5000-20260916-codex6used installed binary commit7f4973dc59fand a fresh bucket.d1b85c7ffixes the Rust 1.98 compaction lint;8c6a215cupdates the protocol, request formulas, parity ledger, and retained qualification evidence.Summary
Correctness model
Requests, latency, and checkpointing
The prior v1 path used roughly 60-69 object-store attempts per incremental push. With a captured v2 view, the single-ref writer core is history-flat:
Commit count does not change this foreground request count. Background checkpointing is outside the clean push budget. The server starts it at 32 visible capsules; if it falls behind, receive synchronously checkpoints at 56 so the 64-entry hard bound remains a fail-closed last resort. Checkpoint publication may lag concurrent pushes: the next writer drops only the exact captured prefix and retains the suffix.
The current checkpoint carries all verified Git packs. Shared pack consolidation and checkpoint byte-growth qualification remain release blockers; this PR does not claim that the present checkpoint representation is the final byte-optimal form.
Earlier RustFS 5,000-push results used the superseded binary-run foreground algorithm. They remain useful baseline evidence but do not qualify the leaf/checkpoint implementation. A fresh Kubernetes replay with periodic fetch/pull, final independent clone, strict fsck, request counters, bytes, and p50/p95/p99 latency is still required.
Xet and LFS behavior
Xorbs and shards remain separate immutable content-addressed objects; capsules contain authenticated catalogs and references, not large payload bytes. The canonical v2 publisher verifies or uploads reachable Xet and LFS dependencies before ref publication. Pointer clone, checkout, smudge, hydrate, prefetch, diff, and mount share one generation-pinned catalog and reconstruction path.
Verification
cargo test -p crab-metadata -p crab-read -p crab-write -p crab-http-server --locked: passedcargo test -p crab-remote -p crab-http-server --locked: passedcrab-metadata,crab-read,crab-write, andcrab-http-serverV1 parity boundary
This PR materially advances v2 parity but does not declare full v1 product parity. Remaining release blockers include:
crab/docs/design/capsule-xorbs-shards.mdnow requires every shipped entry point to map tov2 proven,intentionally removed, orrelease blocker. Replica, tiering, mount, and browser are specified as one pinned-view cross-surface contract with combined fault scenarios; no missing surface may silently fall back to v1.Generation-owner parity
crab metadb ownernow selects a present v2 root exclusively, fails closed on corrupt v2 authority, and never reads or synthesizes a v1 manifest after v2 selection.Recovery authority parity
recover apply --rebuild-file-indexnow selects repository authority before mutation. A present v2 root is opened exclusively and planned file-to-shard mappings are checked against the complete authenticated pointer catalog.file_index_db; only a repository with no v2 root uses the existing shard-to-SlateDB rebuild. Corrupt v2 authority fails closed.Diagnostics and rebuild parity
crab metadb diagnoseselects a present v2 root exclusively. Its default probe reads only root/ref activity;--deepauthenticates the full capsule/checkpoint/catalog/visibility view, reads and verifies every referenced shard and xorb, reconstructs the Git closure, and rejects a result if repository activity changed during the scan.crab metadb rebuildnow uses the same strict authority selection. It verifies the complete external pointer closure and Git closure from one pinned view, publishes a checkpoint only when a capsule frontier needs compaction, and uses exact-root CAS. Re-running against an already checkpointed view is verification-only, so it does not create a hot sequence of redundant checkpoints.crabtests, the capsule-view accounting test, the real-pack checkpoint/next-push integration test, formatting, diff validation, andcargo check -p crab --locked --features gix-transport.Shard-compaction parity
crab compactnow selects a present capsule-v2 root before the legacy shard-list and fails closed on corrupt v2 authority.manifests/shard-list.cargo check, formatting/diff checks, and the 398-page/4,307-fragment docs link check. The strict Clippy attempt is blocked by the pre-existingcrab-readdouble_must_usewarning atcrates/crab-read/src/capsule_protocol.rs:135; non-strict Clippy completes with the existing workspace warning inventory.Xorb-optimization parity
crab optimize xorbsnow selects a present capsule-v2 root as the exclusive authority, derives source xorbs only from the authenticated live file/shard closure, and fails closed on corrupt v2 state. Only repositories without a v2 root use the legacy global inventory and manifest path.Destination storage-class parity
--output-classnow validates and canonicalizes provider-native S3, GCS, and Azure values before a run starts, and appliesStorageClasson each newly created destination Xorb.crab-storagetests, all 61 Xorb optimizer tests, feature-enabled Cargo check, touched-surface Clippy, formatting/diff checks, and docs-link validation.Qualification corrections
must_useannotation from a fallible capsule-view accessor.SIGKILLqualifier now observes a non-LFS Git pack while it is still in local staging, freezes the server, and kills that exact in-flight receive. Publication-phase response-loss and root/head boundary behavior remain covered by the dedicated receive fault matrix.-D warningschecks, 14 capsule read tests, six qualification-log tests, Python and shell syntax, pinned ShellCheck, formatting, and diff validation.Classic shallow-fetch parity
shallowand handles classic-helper shallow clone, absolute and relative deepening, unshallow, and follow-tags through the canonical authenticated upload-pack plannerhaveset; installation is serialized with the shared per-clone lock, ref tips are validated, and.git/shallowis replaced atomicallycargo check -p crab --lib --locked, formatting, and diff validation. Strict all-target Clippy on this host is blocked by the existing repository-wide Rust 1.97 lint inventory, not this surface.Container recovery gate follow-up
Root-scoped server backup parity
Fail-closed server adoption
Complete parity closure ledger
Docs link validation: 398 pages and 4,307 fragments passed.
Complete adoption dependency proof
Proof:
crab-gitlibrary 269 passed/1 ignored; capsule read 14/14; server adoption 13/13; maintenance 8/8; strict all-target Clippy passed for touched crates (server uses the existingresult_large_errallowance); docs link validation passed at 398 pages/4,307 fragments.Post-adoption dependency scrub
GET /integrityzon the private management listener returns202before a current proof,200with exact proof counts and digest after success, and503after dependency loss while retaining the prior complete proofDeployment-wide scrub ownership
Complete v2 authority restore qualification
Scrub lease-loss fencing proof