index-pack: hash full blobs in a bounded worker pool - #67
Draft
ttaylorr-oai wants to merge 8 commits into
Draft
Conversation
Integrate the current tb/codex/automation topic into the internally distributed codex branch. Codex-Integration: tb/codex/automation@5cd3b3771c0aac541add6fc3a7a97bcfe1749cbb
Integrate the current tb/codex/geometric-maintenance-promisor topic into the internally distributed codex branch. Codex-Integration: tb/codex/geometric-maintenance-promisor@dc2fffc37cead551f8036c9ecab5e52a4cbee37b
Integrate the current tb/codex/release topic into the internally distributed codex branch. Codex-Integration: tb/codex/release@40589b5333835ecd5e1b6187cbcec45d8382303e
Integrate the current dr/codex/dugite topic into the internally distributed codex branch. Codex-Integration: dr/codex/dugite@6e6ec362f3c5e03f01bf1fd0fbdea5589cf4caec
Integrate the current tb/codex/lto-pgo topic into the internally distributed codex branch. Codex-Integration: tb/codex/lto-pgo@f0ee69fd806fa486eb3d00e799cfe564a499f8ab
Integrate the current tb/codex/packfile-uri-concurrency topic into the internally distributed codex branch. Codex-Integration: tb/codex/packfile-uri-concurrency@4cc8b3223b233aa3b795b1265e6e7cf3d63c7170
Integrate the current af/codex/pack-bytes topic into the internally distributed codex branch. Codex-Integration: af/codex/pack-bytes@9abdcdc650f32a4b03c31d0595e0b29892749574
The first pass through a pack inflates and hashes each object on the main thread. `pack.threads` applies later while resolving deltas, so SHA1DC work for full blobs remains serial even when CPUs are idle. In a 99 Hz profile of an 844,020,252-byte pack dominated by full blobs, SHA1DC accounted for 72.14% of sampled user CPU. Add an opt-in worker pool for complete heap-backed blobs. The producer continues parsing, inflating, computing CRCs, and writing the pack. The workers use the normal object hashing backend, including SHA1DC, while the main thread performs the existing ODB, collision, and content checks as deferred results are retired in queue order. Bound retained data by both bytes and job count. Strict, fsck, promisor, non-threaded, streamed, and otherwise ineligible objects retain the serial path. The default is disabled. Across three runs of the original prototype on that pack, two workers reduced median wall time from 36.076 to 19.227 seconds. CPU changed from 50.121 to 50.955 seconds and peak RSS from 413.7 to 409.8 MiB. The object/delta-heavy control did not show a wall-time improvement. The fixed-size queue used here has not been rebenchmarked. t5352 covers output equivalence, queue limits, serial fallbacks, collision and duplicate handling, REF/OFS ordering, corrupt input, and configuration validation. p5352 compares serial, one-worker, and two-worker indexing on a reproducible full-blob pack. Existing SHA-1 and SHA-256 index-pack tests also pass.
ttaylorr-oai
force-pushed
the
tb/codex/index-pack-perf-wip
branch
from
August 22, 2026 03:14
4bb2a85 to
633e355
Compare
ttaylorr-oai
added
kind:review-only
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.