Skip to content

Batched BoxDilate passes for positive padded_grid / dual_grid on CUDA - #776

Open
swahtz wants to merge 4 commits into
openvdb:mainfrom
swahtz:batched/pad-dual
Open

swahtz wants to merge 4 commits into
openvdb:mainfrom
swahtz:batched/pad-dual

Conversation

@swahtz

@swahtz swahtz commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Part of #775 (item 1). Stacked on #835, which is stacked on #834. Until those merge, this PR also shows their commits. Review only the last commit, 0e0bb3d5.

Summary

dispatchBuildPaddedGrid<torch::kCUDA> sits behind GridBatch.dual_grid() and the build_padded_grid binding. It looped over batch members, ran one PadGrid pass per member per unit of padding, and merged the per-member storages. So its wall clock grew linearly with batch size.

This PR covers pure positive padding: bmin == 0 with exclude_border == False, which is dual_grid() and build_padded_grid(0, k). That case now builds the whole batch in k chained batched::TopologyPassSpec::boxDilate(Coord(0), Coord(1)) passes through batched::batchedTopologyStorage. After #835 those passes run on NanoVDB's TopologyBuilder.

  • One output buffer
  • Two host waits per pass
  • Empty members become valid empty grids inline
  • Sliced and non-contiguous views are read through the view-aware member pointers

Changes

  • src/fvdb/detail/ops/BuildPaddedGrid.cu includes BatchedTopologyBuilder.cuh. After the totalPasses == 0 identity branch, when numNegative == 0 && !excludeBorder, it builds bmax boxDilate(0, +1) passes and returns batchedTopologyStorage(...).
  • bmax chained Minkowski sums with {0,1}^3 equal the Minkowski sum with {0..bmax}^3, which is what buildPaddedGridFromGridCPU computes for [0, bmax]^3.
  • These paths are unchanged:

Benchmark

GridBatch.dual_grid(), median of 20 runs after 3 warmups. GPU: RTX PRO 6000 Blackwell Workstation Edition, CUDA 13.4. "main" is main's per-member path. Peak is the extra device memory one call allocates.

  • random128 members are 100k random coordinates in [-64, 64)^3. They straddle 0, so each grid spans 8 root tiles, which is TopologyBuilder's worst case.
  • shell64 members are sphere shells inside one root tile.
input B voxels main ms this PR ms speedup main peak MiB this PR peak MiB
random128 1 97,632 0.55 0.33 1.7x 133 133
random128 4 390,623 1.91 0.64 3.0x 141 532
random128 16 1,562,422 7.34 2.13 3.4x 175 1,085
random128 64 6,249,883 29.89 8.51 3.5x 360 1,218
shell64 1 14,096 0.36 0.27 1.3x 16.6 16.6
shell64 4 46,634 1.06 0.30 3.5x 17.5 66
shell64 16 180,695 3.85 0.56 6.9x 21.2 265
shell64 64 728,057 14.76 1.59 9.3x 39.9 1,060

Every output matched main's ijk and offsets exactly.

Peak memory grows because TopologyBuilder allocates about 16.5 MiB of lower-node masks per speculative root tile. Chunking by FVDB_TOPOLOGY_MAX_TILES (#835) caps it near 1.1 GiB plus the output. AcademySoftwareFoundation/openvdb#2392 tracks the upstream fix.

Tests

tests/unit/test_batched_topology_builder.py gains these tests:

  • test_dual_grid_matches_expected_and_cpu runs over _tricky_batches() and checks three things:
    • ijk.jdata elementwise against a from_ijk build of unique(ijk + {0,1}^3)
    • coordinate sets against the CPU dual_grid
    • the dual-transform fix-up: voxel size is preserved and the origin shifts by half a voxel
  • test_padded_grid_positive_matches_expected_and_cpu runs build_padded_grid(0, k) for k in (1, 2). It checks against unique(ijk + {0..k}^3) and CPU, and checks that the origin is unchanged.
  • test_dual_grid_of_sliced_batch runs dual_grid on a [1:6] slice of a batch with empty members. It checks the result against per-member builds and the expected coordinates.

Test plan

  • Build: ./build.sh wheel (CUDA 13.4, sm_120)
  • Full unit suite: cd tests && pytest unit -q gives 3130 passed, 85 skipped
  • clang-format-18 --style=file --dry-run, ruff check --select I and ruff format --check report nothing on the changed files

🤖 Generated with Claude Code

swahtz and others added 4 commits October 23, 2026 00:01
Pin NanoVDB to 71ae97d42, the head of openvdb's develop/nanovdb branch,
which adds the batch-capable TopologyBuilder
(AcademySoftwareFoundation/openvdb#2336) that fvdb's batched topology
construction will move onto (openvdb#785). The pin crosses 23 NanoVDB commits;
three of them need changes here, because fvdb builds with -Werror:

- AcademySoftwareFoundation/openvdb#2373 deprecates memory resources
  without operator==. TorchResource is stateless, so any two instances
  compare equal. Two TorchDeviceResources compare equal when they share
  a device type: torch's CUDA allocator frees a block on whichever
  device holds it, and PrivateUse1 memory frees only through the
  PrivateUse1 allocator.
- AcademySoftwareFoundation/openvdb#2378 deprecates
  cuda::detail::HandleFactory, which already forwarded to
  GridHandle::adopt and GridHandle::metaData. Call those directly in
  GridStorage.cu and DeviceGridHandleUtils.cuh; behavior is unchanged.
- AcademySoftwareFoundation/openvdb#2207 made the Vec3 constructor
  constexpr, so nvcc now reports the unused `half` variable in
  primalVoxelTransformForSizeAndOrigin as error 177-D. Remove it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Replace the sort-based batched topology engine from openvdb#757 with NanoVDB's
batch-capable tools::cuda::TopologyBuilder
(AcademySoftwareFoundation/openvdb#2336), as planned in openvdb#785. The API
(TopologyPassSpec, batchedTopologyStorage) and every call site are
unchanged; the emit/sort/dedup engine is removed.

Each pass reads every source grid's upper-node bounding boxes back with
one host wait and derives each output grid's speculative root tiles the
way NanoVDB's RefineGrid, CoarsenGrid and DilateGrid do. One launch over
all source leaves fills the builder's densified child masks, through
NanoVDB's Refine/Coarsen functors or fvdb's BoxDilate marking; the
builder counts nodes and writes every node; a second launch writes the
leaf value masks. BoxDilate runs one work item per (source leaf, target
leaf offset) pair.

The builder densifies about 16.5 MiB of lower-node masks per speculative
root tile, so a pass splits the batch into chunks of at most
FVDB_TOPOLOGY_MAX_TILES tiles (default 64). Chained passes read their
sources across chunks; the last pass's chunks are assembled with
assembleGridStorage, or adopted when there is one. The builder is cached
per stream so its pinned staging is allocated once.

Tests add a fixture spread over many root tiles and a check that a
chunked build matches an unchunked one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
dispatchBuildPaddedGrid<torch::kCUDA> looped over batch members, ran one
PadGrid pass per member per unit of padding, and merged the per-member
storages, so wall clock grew linearly with batch size regardless of
topology size.

For pure positive padding (bmin == 0, no exclude_border: dual_grid() and
build_padded_grid(0, k)) the whole batch is now built in k chained
batched::TopologyPassSpec::boxDilate(Coord(0), Coord(1)) passes through
batched::batchedTopologyStorage. The identity branch (bmin == bmax == 0)
and the negative-pad / erosion branch are unchanged; the latter is
issue openvdb#775 item 7. The dual-transform fix-up in buildPaddedGrid is
untouched.

Tests: dual_grid() and build_padded_grid(0, k) for k in (1, 2) over the
tricky batches, pinned elementwise against from_ijk builds of the
Minkowski sum ijk + {0..k}^3 and as coordinate sets against CPU, plus
the dual transform fix-up and a sliced (non-contiguous) batch view.

Part of issue openvdb#775 (item 1).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimization Performance or memory optimization Topology Operations Issues related to topology operations (prune, merge, dilate, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant