Exploding and vanishing gradients in deep neural networks: the effect of residual connections

TL;DR

A theoretical study that rigorously proves the mechanism by which residual connections mitigate the exploding/vanishing gradient problem in deep learning using ergodic theory.

Problem

In deep neural network training, exploding gradients and vanishing gradients are fundamental issues that hinder convergence of deep networks. Although there has been much empirical evidence that residual connections alleviate these problems, the mathematical principles were not clearly understood.

Approach

The authors introduce multiplicative ergodic theory to model the forward propagation of neural networks as a linearized dynamical system. In particular, they utilize the characterization of Lyapunov exponents by Furstenberg and Kifer to analyze the effect of residual connections on the Lyapunov spectrum. Through this, they mathematically prove how residual connections improve the stability of gradient propagation.

Results & Contribution

This study provides the first rigorous mathematical explanation of how residual connections mitigate the exploding/vanishing gradient problem. It offers a theoretical foundation for the success of various architectures using residual connections, such as ResNet, and suggests directions for designing deeper and more stable neural networks in the future.