Skip to content

store: compute template digests while publishing - #68

Merged
CMGS merged 7 commits into
mainfrom
codex/issue-64-store-digest-v2
Aug 11, 2026
Merged

store: compute template digests while publishing#68
CMGS merged 7 commits into
mainfrom
codex/issue-64-store-digest-v2

Conversation

@CMGS

@CMGS CMGS commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Summary

  • replace the pool-owned v1 full-tree pre-read and content_digest meta field with the canonical 16 MiB chunked v2 store contract
  • make dir hash with a bounded per-call worker pool, store digest-<gen> beside immutable exports, and commit meta.json last
  • hash the S3 source read used for upload, preserve multipart planning with explicit ContentLength, and return marker metadata without another object request
  • propagate the committed digest through Promote -> Fetch -> Claim while keeping checkpoint/archive Publish records digest-free

The dir implementation is based on current main's generation-addressed layout from #67. Hash/upload/sidecar failures leave the previous meta, export, and digest visible; a fresh retry commits one complete replacement.

Validation

  • make go-test
  • make go-lint
  • go test -race ./store/dir -run '^(TestFetchPinsGenerationAcrossStoreInstances|TestDigestWorkersStoreChunksByOffset|TestPublishDigestedFailureAndFreshRetry|TestSweepGenerationsPairsCurrentAndSupersededDigests)$' -count=20
  • go test -race ./pool -run '^(TestPromoteThenClaimClonesFromTemplate|TestRepromoteContentDigestTracksExportBytes)$' -count=20
  • Linux and Darwin ASL: store packages clean; the only pool output is the three pre-existing pool.go method-partition findings on unchanged main lines 584/605/624
  • independent fixed-vector recomputation matched all published vectors, including 16 MiB - 1 / exact / + 1
  • local MinIO RELEASE.2025-09-07T16-13-09Z: the 16 MiB + 1 real multipart upload matched the v2 vector; 64 MiB + 1 A/B (5 publishes/mode x 3) had overlapping ranges, with per-run medians plain 575/511/382 MB/s and digested 513/473/464 MB/s, so no fallback path was added
  • M2 Max dir acceptance: a same-key claim during three 1 GiB re-promotes waited 88.8-151.5 ms

Production diff: +590/-214 lines. Test and test-support diff: +962/-301 lines.

Closes #64

CMGS added 7 commits August 11, 2026 23:18
Staging is single-writer under the template lock and every chunk read
already fails on a short read; the extra per-file Lstat guarded nothing
reachable.
Restatements, perf-advertising clauses, and cross-duplicated godoc go;
contract sentences (locking, crash/retry, security invariants) stay.
@CMGS
CMGS merged commit 0b99e05 into main Aug 11, 2026
4 checks passed
@CMGS
CMGS deleted the codex/issue-64-store-digest-v2 branch August 11, 2026 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

store-owned v2 chunked export digest: PublishDigested with dir parallel hash and s3 upload tee

1 participant