fix: harden dag traversal and CAR responses - #1197
Merged
Merged
Conversation
A sharded directory declares a fanout and carries a bitmap marking which slots are filled. NewHamtFromDag passed that bitmap straight to makeChilder, which copies it into a bitfield sized from the fanout and has no room to spare if the bitmap is wider. A bitmap wider than the fanout marks slots the shard cannot hold, so reject it while reading the node.
GetCAR walks the DAG on a goroutine detached from the request, decoding each block with whatever codec its CID names. That runs third-party code this package does not control, and a panic on a detached goroutine ends the process rather than the response. Recover there, log it, and close the pipe with an error, which is what the other error paths in that goroutine already do. bsnet has the same guard around its detached stream close.
Traversal keeps per-level state, so walking a DAG costs more the deeper it goes. Nothing bounded how far a CAR response would descend. blockOpener now refuses to load a block deeper than the backend's limit, using the traversal path the link was reached by. Both the CAR traversal and CAR path resolution load blocks through it, so one check covers walkGatewaySimpleSelector and nodeGetterFetcherSingleUseFactory. The plain Get path uses a different link system and is untouched. WithMaxTraversalDepth sets the limit and 0 removes it. The default of 1024 is far above anything UnixFS produces: a file reaches terabytes by depth 4, HAMT adds about 4 levels per million directory entries, and directory nesting follows the source tree.
A CAR response that fails partway through was indistinguishable from a complete one. The status line and headers are long gone by then, and X-Stream-Error is set once the body is already streaming, so it rarely reaches the client and never reaches a proxy or CDN in between. Append a marker, as withRetrievalTimeout already does when it cuts a response short. CAR has no in-band way to say a stream is incomplete, so this makes the trailing bytes invalid where the next block header would start: a reader stops with an error instead of accepting a short DAG. The marker says nothing about why. The client already knows which DAG it asked for, and the underlying error names the path it stopped at, which can run to thousands of segments. That detail stays in the gateway log.
Picks up the HAMT bitmap fix from ipfs/go-unixfsnode#100. Pinned to that branch for now; swap for the tagged release before merging. examples is a separate module with its own go.sum and a replace back to the root, so it carries the same bump.
lidel
force-pushed
the
harden-car-traversal
branch
from
July 26, 2026 23:23
1edd387 to
ad9d9d3
Compare
Codecov Report❌ Patch coverage is @@ Coverage Diff @@
## main #1197 +/- ##
==========================================
- Coverage 63.89% 63.87% -0.02%
==========================================
Files 269 269
Lines 27115 27135 +20
==========================================
+ Hits 17325 17333 +8
- Misses 8080 8087 +7
- Partials 1710 1715 +5
... and 11 files with indirect coverage changes 🚀 New features to boost your workflow:
|
walkGatewaySimpleSelector takes the terminal block from callers that have already resolved it, and every scope but one uses it. dag-scope=block ignored it and called LoadRaw instead. CarBackend passes that block and backs its link system with a CAR stream read in order, which has already moved past it. The reload therefore reported the stream as unexpectedly short on a response that was complete, and callers saw a stream error on a request that succeeded. BlocksBackend passes nil and still loads the block itself.
gammazero
approved these changes
Jul 27, 2026
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Jul 27, 2026
go-ipld-git and go-unixfsnode are on their tagged releases. boxo is pinned to main, which carries ipfs/boxo#1197 but has not been released yet, so this still needs a boxo release before it can merge.
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Jul 27, 2026
go-ipld-git and go-unixfsnode are on their tagged releases. boxo is pinned to main, which carries ipfs/boxo#1197 but has not been released yet, so this still needs a boxo release before it can merge. Notes the CAR truncation marker in the v0.43 changelog, since that is the user-visible part of the boxo update.
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Jul 27, 2026
go-ipld-git and go-unixfsnode are on their tagged releases. boxo is pinned to main, which carries ipfs/boxo#1197 but has not been released yet, so this still needs a boxo release before it can merge. Notes the CAR truncation marker in the v0.43 changelog, since that is the user-visible part of the boxo update. Also scopes the ipfs-webui Playwright install to chromium. That job declares no Playwright projects, so it only ever runs the default browser, but it was installing OS dependencies for firefox and webkit too. Fetching that ffmpeg stack outlasted the job's 20 minute budget on a slow Ubuntu mirror and the run was cancelled before any test started. The install steps now have their own timeouts so a slow mirror fails there instead of starving the tests.
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Jul 27, 2026
go-ipld-git and go-unixfsnode are on their tagged releases. boxo is pinned to main, which carries ipfs/boxo#1197 but has not been released yet, so this still needs a boxo release before it can merge. Notes the CAR truncation marker in the v0.43 changelog, since that is the user-visible part of the boxo update. Also stops a slow Ubuntu mirror from failing the ipfs-webui job. That job installed Playwright OS dependencies for every browser although it declares no projects and so only ever runs chromium, and the install had no timeout of its own. When the mirror served 10.7 MB of package indices at 39 kB/s, apt-get update alone outlasted the job's 20 minute budget and the run was cancelled before any test started. The install is now scoped to chromium, capped, and best effort: the runner image already ships what headless chromium needs, and a library that really is missing surfaces when the browser fails to launch.
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Jul 27, 2026
* fix: recover from panics in detached goroutines ls, dag get and dag export each hand their work to a goroutine and read the result back over a channel or pipe. Decoding and encoding there runs whatever codec a block's CID names, so it runs third-party code, and a panic on a detached goroutine ends the daemon rather than the command. Each now recovers, logs, and reports through the channel it already uses. In dag export the recover is registered after the existing cleanup defer so it runs first, while errCh is still open. * chore: update boxo, go-ipld-git and go-unixfsnode Picks up the CAR streaming and object parsing work from ipfs/boxo#1197, ipfs/go-ipld-git#77 and ipfs/go-unixfsnode#100. All three are pinned to their branches for now; swap for the tagged releases before merging. * chore: update boxo, go-ipld-git and go-unixfsnode go-ipld-git and go-unixfsnode are on their tagged releases. boxo is pinned to main, which carries ipfs/boxo#1197 but has not been released yet, so this still needs a boxo release before it can merge. Notes the CAR truncation marker in the v0.43 changelog, since that is the user-visible part of the boxo update. Also stops a slow Ubuntu mirror from failing the ipfs-webui job. That job installed Playwright OS dependencies for every browser although it declares no projects and so only ever runs chromium, and the install had no timeout of its own. When the mirror served 10.7 MB of package indices at 39 kB/s, apt-get update alone outlasted the job's 20 minute budget and the run was cancelled before any test started. The install is now scoped to chromium, capped, and best effort: the runner image already ships what headless chromium needs, and a library that really is missing surfaces when the browser fails to launch.
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Jul 27, 2026
* fix: recover from panics in detached goroutines ls, dag get and dag export each hand their work to a goroutine and read the result back over a channel or pipe. Decoding and encoding there runs whatever codec a block's CID names, so it runs third-party code, and a panic on a detached goroutine ends the daemon rather than the command. Each now recovers, logs, and reports through the channel it already uses. In dag export the recover is registered after the existing cleanup defer so it runs first, while errCh is still open. * chore: update boxo, go-ipld-git and go-unixfsnode Picks up the CAR streaming and object parsing work from ipfs/boxo#1197, ipfs/go-ipld-git#77 and ipfs/go-unixfsnode#100. All three are pinned to their branches for now; swap for the tagged releases before merging. * chore: update boxo, go-ipld-git and go-unixfsnode go-ipld-git and go-unixfsnode are on their tagged releases. boxo is pinned to main, which carries ipfs/boxo#1197 but has not been released yet, so this still needs a boxo release before it can merge. Notes the CAR truncation marker in the v0.43 changelog, since that is the user-visible part of the boxo update. Also stops a slow Ubuntu mirror from failing the ipfs-webui job. That job installed Playwright OS dependencies for every browser although it declares no projects and so only ever runs chromium, and the install had no timeout of its own. When the mirror served 10.7 MB of package indices at 39 kB/s, apt-get update alone outlasted the job's 20 minute budget and the run was cancelled before any test started. The install is now scoped to chromium, capped, and best effort: the runner image already ships what headless chromium needs, and a library that really is missing surfaces when the browser fails to launch. (cherry picked from commit f9baf8f)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Serving a CAR walks a DAG the requester picked, and nothing bounded how deep it goes. Traversal keeps per-level state, so a deeply nested DAG costs much more than its size suggests. The walk runs on a goroutine detached from the request, where a panic in a codec ends the process rather than the response. And a walk that stops early still looks complete, since
X-Stream-Erroris set once the body is already streaming and rarely reaches the client, let alone a proxy in between.Fix
WithMaxTraversalDepthbounds the walk. Default 1024,0removes it. UnixFS never comes close: a file reaches terabytes by depth 4.GetCARgoroutine, asbsnetalready does for its detached stream close.withRetrievalTimeoutalready does, so a reader errors instead of accepting a short DAG.Well-formed content is unaffected: a 700 entry sharded directory and a 500 deep chain return byte-identical CARs.
Pinned to ipfs/go-unixfsnode#100 until it is tagged.