Residual network (ResNet)

ResNets address the problem of diminishing returns (or even performance degradation) when increasing the depth of neural networks. As networks become deeper, training becomes increasingly difficult due to the vanishing gradient problem, where gradients become very small as they are backpropagated through many layers. This hinders the learning process in earlier layers. ResNets introduce residual blocks, which contain shortcut connections that bypass one or more layers. Instead of learning the underlying mapping H(x) directly, the network learns a residual function F(x) = H(x) - x. The original input x is then added back to the output of the layers through the shortcut connection, resulting in H(x) = F(x) + x. This identity mapping allows the network to easily learn the identity function if needed, and the residual function focuses on learning the differences between the input and output, which are often smaller and easier to learn. This architecture facilitates the training of much deeper networks (hundreds or even thousands of layers) and has been highly successful in various computer vision tasks.