Residual neural network

The AlexNet model developed in 2012 for ImageNet was an 8-layer convolutional neural network. The neural networks developed in 2014 by the Visual Geometry Group (VGG) at the University of Oxford approached a depth of 19 layers by stacking 3-by-3 convolutional layers.[6] But stacking more layers led to a quick reduction in training accuracy,[7] which is referred to as the "degradation" problem.[1]

A deeper network should not produce a higher training loss than its shallower counterpart, if this deeper network can be constructed by its shallower counterpart stacked with extra layers.[1] If the extra layers can be set as identity mappings, the deeper network would represent the same function as the shallower counterpart. It is hypothesized that the optimizer is not able to approach identity mappings for the parameterized layers.

In a multi-layer neural network model, consider a subnetwork with a certain number (e.g., 2 or 3) of stacked layers. Denote the underlying function performed by this subnetwork as ${\textstyle H(x)}$, where ${\textstyle x}$ is the input to this subnetwork. The idea of "Residual Learning" re-parameterizes this subnetwork and lets the parameter layers represent a residual function ${\textstyle F(x):=H(x)-x}$. The output ${\textstyle y}$ of this subnetwork is represented as:

${\displaystyle {\begin{aligned}y&=F(x)+x\end{aligned}}}$

This is also the principle of the 1997 LSTM cell[3] computing ${\textstyle y_{t+1}=F(x_{t})+x_{t}}$, which becomes ${\textstyle y=F(x)+x}$ during backpropagation through time.

The function ${\textstyle F(x)}$ is often represented by matrix multiplication interlaced with activation functions and normalization operations (e.g., Batch Normalization or Layer Normalization).

This subnetwork is referred to as a "Residual Block".[1] A deep residual network is constructed by stacking a series of residual blocks.

The operation of "${\textstyle +\ x}$" in "${\textstyle y=F(x)+x}$" is approached by a skip connection that performs identity mapping and connects the input of a residual block with its output. This connection is often referred to as a "Residual Connection" in later work.[8]

The introduction of identity mappings facilitates signal propagation in both forward and backward paths. [9]

If the output of the ${\textstyle \ell }$-th residual block is the input to the ${\textstyle (\ell +1)}$-th residual block (i.e., assuming no activation function between blocks), we have:[9]

${\displaystyle {\begin{aligned}x_{\ell +1}&=F(x_{\ell })+x_{\ell }\end{aligned}}}$

Applying this formulation recursively, e.g., ${\displaystyle {\begin{aligned}x_{\ell +2}=F(x_{\ell +1})+x_{\ell +1}=F(x_{\ell +1})+F(x_{\ell })+x_{\ell }\end{aligned}}}$, we have:

${\displaystyle {\begin{aligned}x_{L}&=x_{\ell }+\sum _{i=l}^{L-1}F(x_{i})\\\end{aligned}}}$

where ${\textstyle L}$ is the index of any later residual block (e.g., the last block) and ${\textstyle \ell }$ is the index of any earlier block. This formulation suggests that there is always a signal that is directly sent from a shallower block ${\textstyle \ell }$ to a deeper block ${\textstyle L}$.

The Residual Learning formulation provides the added benefit of addressing the vanishing gradient problem to some extent. However, it is crucial to acknowledge that the vanishing gradient issue is not the root cause of the degradation problem, as it has already been tackled through the use of normalization layers. Taking the derivative w.r.t. ${\textstyle x_{\ell }}$ according to the above forward propagation, we have:[9]

${\displaystyle {\begin{aligned}{\frac {\partial {\mathcal {E}}}{\partial x_{\ell }}}&={\frac {\partial {\mathcal {E}}}{\partial x_{L}}}{\frac {\partial x_{L}}{\partial x_{\ell }}}\\&={\frac {\partial {\mathcal {E}}}{\partial x_{L}}}\left(1+{\frac {\partial }{\partial x_{\ell }}}\sum _{i=l}^{L-1}F(x_{i})\right)\\&={\frac {\partial {\mathcal {E}}}{\partial x_{L}}}+{\frac {\partial {\mathcal {E}}}{\partial x_{L}}}{\frac {\partial }{\partial x_{\ell }}}\sum _{i=l}^{L-1}F(x_{i})\\\end{aligned}}}$

Here ${\textstyle {\mathcal {E}}}$ is the loss function to be minimized. This formulation suggests that the gradient computation of a shallower layer ${\textstyle {\frac {\partial {\mathcal {E}}}{\partial x_{\ell }}}}$ always has a term ${\textstyle {\frac {\partial {\mathcal {E}}}{\partial x_{L}}}}$ that is directly added. Even if the gradients of the ${\textstyle F(x_{i})}$ terms are small, the total gradient ${\textstyle {\frac {\partial {\mathcal {E}}}{\partial x_{\ell }}}}$ is not vanishing thanks to the added term ${\textstyle {\frac {\partial {\mathcal {E}}}{\partial x_{L}}}}$.

A Basic Block is the simplest building block studied in the original ResNet.[1] This block consists of two sequential 3x3 convolutional layers and a residual connection. The input and output dimensions of both layers are equal.

A Bottleneck Block[1] consists of three sequential convolutional layers and a residual connection. The first layer in this block is a 1x1 convolution for dimension reduction, e.g., to 1/4 of the input dimension; the second layer performs a 3x3 convolution; the last layer is another 1x1 convolution for dimension restoration. The models of ResNet-50, ResNet-101, and ResNet-152 in [1] are all based on Bottleneck Blocks.

The Pre-activation Residual Block[9] applies the activation functions (e.g., non-linearity and normalization) before applying the residual function ${\textstyle F}$. Formally, the computation of a Pre-activation Residual Block can be written as:

${\displaystyle {\begin{aligned}x_{\ell +1}&=F(\phi (x_{\ell }))+x_{\ell }\end{aligned}}}$

where ${\textstyle \phi }$ can be any non-linearity activation (e.g., ReLU) or normalization (e.g., LayerNorm) operation. This design reduces the number of non-identity mappings between Residual Blocks. This design was used to train models with 200 to over 1000 layers.[9]

Since GPT-2, the Transformer Blocks have been dominantly implemented as Pre-activation Blocks. This is often referred to as "pre-normalization" in the literature of Transformer models.[10]

The Transformer architecture used in the original GPT model. A Transformer Block consists of two Residual Blocks: a Multi-Head Attention Block and a feed-forward Multi-Layer Perceptron (MLP) Block.

A Transformer Block is a stack of two Residual Blocks. Each Residual Block has a Residual Connection.

The first Residual Block is a Multi-Head Attention Block, which performs (self-)attention computation followed by a linear projection.

The second Residual Block is a feed-forward Multi-Layer Perceptron (MLP) Block. This block is analogous to an "inverse" bottleneck block: it has a linear projection layer (which is equivalent to a 1x1 convolution in the context of Convolutional Neural Networks) that increases the dimension, and another linear projection that reduces the dimension.

A Transformer Block has a depth of 4 layers (linear projections). The GPT-3 model has 96 Transformer Blocks (in the literature of Transformers, a Transformer Block is often referred to as a "Transformer Layer"). This model has a depth of about 400 projection layers, including 96x4 layers in Transformer Blocks and a few extra layers for input embedding and output prediction.

Very deep Transformer models cannot be successfully trained without Residual Connections.[11]