Neural Networks
Introduction
Activations
Suppose we want to recognize handwritten digits. Writing explicit rules for what makes an image a "3" or a "7" is difficult: different people write the same digit differently, and the useful visual patterns are difficult to specify manually. A neural network instead represents a flexible function whose behaviour can be learned from examples. Given an input image \(x\), we want to learn a function \(f(x)\) that maps the image to a prediction of which digit it represents.
A grayscale image contains pixels, each with a brightness value between \(0\) and \(1\), so the image can be represented as a vector. The input layer contains one neuron for each pixel. A neuron here can simply be understood as something that holds a number, called its activation. The output layer contains \(10\) neurons, corresponding to the digits \(0,1,\dots,9\). Ideally, the neuron corresponding to the correct digit should have the largest activation. Between the input and output are hidden layers. Thus information passes through the network as
\[ \text{input} \rightarrow \text{hidden layers} \rightarrow \text{output}. \]Weights
Consider one neuron in the next layer. Its activation should depend on the activations \(a_1,\ldots,a_n\) of neurons in the previous layer. Different inputs can matter by different amounts, so associate a weight \(w_j\) with each incoming activation. We first compute \(\sum_j w_j a_j\).
The weights determine which patterns of activation the neuron responds to. A large positive \(w_j\) means activation \(a_j\) pushes the neuron toward activation, while a negative \(w_j\) pushes against it.
For example, if the previous layer represents image pixels, a neuron could have positive weights over pixels in a particular region. Its weighted sum would then become large when that region of the image is bright. In this sense, the pattern of weights determines what feature the neuron is sensitive to.
However, we may not want the neuron to activate whenever this weighted sum is merely positive. We therefore introduce a bias \(b\):
\[ \sum_j w_j a_j + b. \]The bias shifts the threshold at which the neuron becomes meaningfully active. Therefore, weights determine what the neuron responds to, while the bias determines how easily it responds. Finally, we pass this quantity through a nonlinear activation function \(\sigma\):
\[ a' = \sigma\left( \sum_j w_j a_j+b \right). \]1 The sigmoid \(\sigma(z)=\frac{1}{1+e^{-z}}\) maps any real-valued input into \((0,1)\).
Vectorization
Doing this separately for every neuron becomes annoying. Collect the activations in one layer into a vector \(\mathbf{a}\), collect all weights connecting this layer to the next into a matrix \(W\), and collect the biases of the next layer into a vector \(\mathbf{b}\). Then the layer transformation can be written as
\[ \mathbf{a}' = \sigma(W\mathbf{a}+\mathbf{b}) \]where \(\sigma\) is applied component-wise.
The dimensions also explain what the matrix is doing. If a layer contains \(n\) neurons and the next contains \(m\), then
\[ \mathbf{a}\in\mathbb{R}^{n}, \qquad W\in\mathbb{R}^{m\times n}, \qquad \mathbf{b}\in\mathbb{R}^{m}, \]so
\[ W\mathbf{a}+\mathbf{b} \in \mathbb{R}^{m}, \]giving one value for each neuron in the next layer. A neural network simply repeats transformations of this form:
\[ \mathbf{a}^{(0)} \rightarrow \mathbf{a}^{(1)} \rightarrow \cdots \rightarrow \mathbf{a}^{(L)}. \]For layer \(\ell\),
\[ \mathbf{a}^{(\ell)} = \sigma\left( W^{(\ell)} \mathbf{a}^{(\ell-1)} + \mathbf{b}^{(\ell)} \right). \]Thus the entire neural network is ultimately just a function
\[ f_\theta(x), \]where \(\theta=\{W^{(1)},\mathbf b^{(1)},\ldots, W^{(L)},\mathbf b^{(L)}\}\) contains all of the network's parameters.
The motivating intuition behind the existence of multiple layers is that difficult problems may be decomposed into progressively more abstract representations. For digit recognition, we might imagine something like
\[ \text{pixels} \rightarrow \text{edges} \rightarrow \text{shapes} \rightarrow \text{digits}. \]Learning
The architecture tells us how an input becomes an output once the parameters are fixed:
\[ x \xrightarrow{\theta} f_\theta(x). \]However, we have not identified the innumerable weights and biases ourselves. Given examples of desired behaviour, we want to find parameters \(\theta^*\) such that \(f_{\theta^*}(x)\) performs the task well.
Learning a neural network is therefore done by finding useful values for its weights and biases. The remaining question is: how do we know which direction to change thousands of parameters so that the network improves? This motivates defining a numerical measure of how badly the network is performing and then changing the parameters so as to reduce it. This leads to the cost function and gradient descent.
Gradient Descent
Cost
A neural network's behaviour is determined by its parameters \(\theta\), containing all of its weights and biases. Learning therefore requires some way to determine which values of \(\theta\) make the network perform well.
Suppose an input \(x\) represents a handwritten \(3\). We want the output activation corresponding to \(3\) to be close to \(1\), and the remaining output activations to be close to \(0\). Let \(\mathbf{y}\) denote this desired output and \(\mathbf{a}^{(L)}(x)\) the network's actual output. We can measure how wrong the network is using
\[ C_x(\theta) = \left\| \mathbf{a}^{(L)}(x)-\mathbf{y} \right\|^2. \]The cost is small when the network's output is close to the desired output and large when it is far away. However, we do not want parameters that work well for only one image. Given \(N\) training examples, define the overall cost as the average
\[ C(\theta) = \frac{1}{N} \sum_{i=1}^{N} C_{x_i}(\theta). \]Training can therefore be viewed as the optimization problem
\[ \theta^* = \arg\min_\theta C(\theta). \]The problem of learning has now become the problem of finding a low point of a function. The unusual part is simply that \(C\) may take thousands, millions, or billions of parameters as inputs.