Skip to content

Improve convergence behavior with groupnorm#79

Open
michaelmckinsey1 wants to merge 1 commit into
LBANN:mainfrom
michaelmckinsey1:convergence-behavior
Open

Improve convergence behavior with groupnorm#79
michaelmckinsey1 wants to merge 1 commit into
LBANN:mainfrom
michaelmckinsey1:convergence-behavior

Conversation

@michaelmckinsey1

@michaelmckinsey1 michaelmckinsey1 commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator

Replaced BatchNorm3d in the 3D U-Net with GroupNorm, and made the number of groups configurable via group_norm_groups. This improves training stability for low batch sizes.

1 minibatch size

batchnorm
image

groupnorm
image

  • need to check if this is worse at higher batch sizes than batch norm

groupnorm does converge in up to 2x less epochs, even at higher batch sizes

8 minibatch size

batchnorm
image

groupnorm
image

  • should we give an option for either group norm or batch norm?
    • I think this is only necessary if groupnorm is significantly worse at higher batch sizes, which it isn't.

@michaelmckinsey1 michaelmckinsey1 self-assigned this Jun 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant