fix: use bitfield.FromBytes for HAMT bitmaps - #100
Merged
Conversation
bitField built an empty bitfield and then filled it with SetBytes, which panics when the bitmap is wider than the fanout the shard declared. Such a bitmap addresses children the shard cannot hold, so it is not usable either way. FromBytes does the same construction and returns an error for that case, and the function already returns an error, so the caller gets one instead of a panic.
|
Suggested tag:
Cutting a Release (and modifying code files)This PR is modifying both Automatically created GitHub ReleaseA draft GitHub Release has been created. |
lidel
added a commit
to ipfs/boxo
that referenced
this pull request
Jul 26, 2026
Picks up the HAMT bitmap fix from ipfs/go-unixfsnode#100. Pinned to that branch for now; swap for the tagged release before merging. examples is a separate module with its own go.sum and a replace back to the root, so it carries the same bump.
lidel
marked this pull request as ready for review
July 27, 2026 00:11
gammazero
approved these changes
Jul 27, 2026
lidel
added a commit
to ipfs/boxo
that referenced
this pull request
Jul 27, 2026
* fix: reject hamt bitfield wider than fanout A sharded directory declares a fanout and carries a bitmap marking which slots are filled. NewHamtFromDag passed that bitmap straight to makeChilder, which copies it into a bitfield sized from the fanout and has no room to spare if the bitmap is wider. A bitmap wider than the fanout marks slots the shard cannot hold, so reject it while reading the node. * fix: recover from panics during CAR traversal GetCAR walks the DAG on a goroutine detached from the request, decoding each block with whatever codec its CID names. That runs third-party code this package does not control, and a panic on a detached goroutine ends the process rather than the response. Recover there, log it, and close the pipe with an error, which is what the other error paths in that goroutine already do. bsnet has the same guard around its detached stream close. * feat: bound dag traversal depth Traversal keeps per-level state, so walking a DAG costs more the deeper it goes. Nothing bounded how far a CAR response would descend. blockOpener now refuses to load a block deeper than the backend's limit, using the traversal path the link was reached by. Both the CAR traversal and CAR path resolution load blocks through it, so one check covers walkGatewaySimpleSelector and nodeGetterFetcherSingleUseFactory. The plain Get path uses a different link system and is untouched. WithMaxTraversalDepth sets the limit and 0 removes it. The default of 1024 is far above anything UnixFS produces: a file reaches terabytes by depth 4, HAMT adds about 4 levels per million directory entries, and directory nesting follows the source tree. * fix: mark truncated CAR responses A CAR response that fails partway through was indistinguishable from a complete one. The status line and headers are long gone by then, and X-Stream-Error is set once the body is already streaming, so it rarely reaches the client and never reaches a proxy or CDN in between. Append a marker, as withRetrievalTimeout already does when it cuts a response short. CAR has no in-band way to say a stream is incomplete, so this makes the trailing bytes invalid where the next block header would start: a reader stops with an error instead of accepting a short DAG. The marker says nothing about why. The client already knows which DAG it asked for, and the underlying error names the path it stopped at, which can run to thousands of segments. That detail stays in the gateway log. * chore: update go-unixfsnode Picks up the HAMT bitmap fix from ipfs/go-unixfsnode#100. Pinned to that branch for now; swap for the tagged release before merging. examples is a separate module with its own go.sum and a replace back to the root, so it carries the same bump. * fix: do not reload an already resolved terminal block walkGatewaySimpleSelector takes the terminal block from callers that have already resolved it, and every scope but one uses it. dag-scope=block ignored it and called LoadRaw instead. CarBackend passes that block and backs its link system with a CAR stream read in order, which has already moved past it. The reload therefore reported the stream as unexpectedly short on a response that was complete, and callers saw a stream error on a request that succeeded. BlocksBackend passes nil and still loads the block itself. * chore: update go-unixfsnode to v1.10.6
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Jul 27, 2026
* fix: recover from panics in detached goroutines ls, dag get and dag export each hand their work to a goroutine and read the result back over a channel or pipe. Decoding and encoding there runs whatever codec a block's CID names, so it runs third-party code, and a panic on a detached goroutine ends the daemon rather than the command. Each now recovers, logs, and reports through the channel it already uses. In dag export the recover is registered after the existing cleanup defer so it runs first, while errCh is still open. * chore: update boxo, go-ipld-git and go-unixfsnode Picks up the CAR streaming and object parsing work from ipfs/boxo#1197, ipfs/go-ipld-git#77 and ipfs/go-unixfsnode#100. All three are pinned to their branches for now; swap for the tagged releases before merging. * chore: update boxo, go-ipld-git and go-unixfsnode go-ipld-git and go-unixfsnode are on their tagged releases. boxo is pinned to main, which carries ipfs/boxo#1197 but has not been released yet, so this still needs a boxo release before it can merge. Notes the CAR truncation marker in the v0.43 changelog, since that is the user-visible part of the boxo update. Also stops a slow Ubuntu mirror from failing the ipfs-webui job. That job installed Playwright OS dependencies for every browser although it declares no projects and so only ever runs chromium, and the install had no timeout of its own. When the mirror served 10.7 MB of package indices at 39 kB/s, apt-get update alone outlasted the job's 20 minute budget and the run was cancelled before any test started. The install is now scoped to chromium, capped, and best effort: the runner image already ships what headless chromium needs, and a library that really is missing surfaces when the browser fails to launch.
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Jul 27, 2026
* fix: recover from panics in detached goroutines ls, dag get and dag export each hand their work to a goroutine and read the result back over a channel or pipe. Decoding and encoding there runs whatever codec a block's CID names, so it runs third-party code, and a panic on a detached goroutine ends the daemon rather than the command. Each now recovers, logs, and reports through the channel it already uses. In dag export the recover is registered after the existing cleanup defer so it runs first, while errCh is still open. * chore: update boxo, go-ipld-git and go-unixfsnode Picks up the CAR streaming and object parsing work from ipfs/boxo#1197, ipfs/go-ipld-git#77 and ipfs/go-unixfsnode#100. All three are pinned to their branches for now; swap for the tagged releases before merging. * chore: update boxo, go-ipld-git and go-unixfsnode go-ipld-git and go-unixfsnode are on their tagged releases. boxo is pinned to main, which carries ipfs/boxo#1197 but has not been released yet, so this still needs a boxo release before it can merge. Notes the CAR truncation marker in the v0.43 changelog, since that is the user-visible part of the boxo update. Also stops a slow Ubuntu mirror from failing the ipfs-webui job. That job installed Playwright OS dependencies for every browser although it declares no projects and so only ever runs chromium, and the install had no timeout of its own. When the mirror served 10.7 MB of package indices at 39 kB/s, apt-get update alone outlasted the job's 20 minute budget and the run was cancelled before any test started. The install is now scoped to chromium, capped, and best effort: the runner image already ships what headless chromium needs, and a library that really is missing surfaces when the browser fails to launch. (cherry picked from commit f9baf8f)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A sharded directory says how many slots it has, then carries a bitmap marking which ones are filled.
bitFieldmade an empty bitmap of the right size and copied the node's bytes in withSetBytes, which panics if those bytes are wider than the slot count. A bitmap that wide marks slots the shard does not have, so it is unusable either way, but the caller gets a panic rather than an error.Fix
Use
bitfield.FromBytes, which builds the same bitmap and returns an error when the bytes do not fit.bitFieldalready returns an error, so nothing else changes and the code is shorter.Bitmaps that fit are unaffected, including short ones, which are the normal case since the encoder trims leading zero bytes.