Batch Norm vs Layer Norm vs Group Norm, Compared on MNIST
·1490 words·7 mins
BatchNorm, LayerNorm and GroupNorm implemented from scratch in PyTorch and compared on MNIST: what each normalizes over, when to use which, and how they affect training and dead neurons.