Deep Learning
Layer normalisation
Layer normalization is a normalization technique applied in neural networks that normalizes the activations of each layer across all the neurons within that layer for each training sample. This helps to stabilize the learning process and allows for higher learning rates, leading to faster convergence and improved generalization performance.
Explanation
Layer normalization operates by computing the mean and variance of the activations within each layer for each individual training example. It then normalizes the activations using these statistics, scaling them to have zero mean and unit variance. Finally, it applies learnable gain and bias parameters to allow the network to adapt the normalization to the specific needs of each layer. Unlike Batch Normalization, Layer Normalization does not depend on the batch size, making it suitable for recurrent neural networks and other architectures where batch sizes can vary or are small. It also tends to be more robust to changes in the data distribution during training. By normalizing activations across features, Layer Normalization reduces internal covariate shift, which stabilizes training and allows the use of higher learning rates, potentially leading to faster convergence and better model performance, especially in deep networks. It's important to note that Layer Normalization is applied independently to each layer and each training example, whereas Batch Normalization is applied across the batch dimension.