Gradient Descent
Gradient Descent
Consider first a function of one variable, \(C(w)\). Its derivative
\[ \frac{dC}{dw} \]tells us how the cost changes when \(w\) changes. If the derivative is positive, increasing \(w\) increases the cost, so we should move \(w\) in the negative direction. If it is negative, increasing \(w\) decreases the cost, so we should move in the positive direction. We can therefore update
\[ w \leftarrow w-\eta\frac{dC}{dw}, \]where \(\eta>0\) is the learning rate, controlling the size of the step. A neural network has many parameters rather than one. If
\[ \theta=(\theta_1,\ldots,\theta_d), \]then the relevant generalization of the derivative is the gradient
\[ \nabla C(\theta) = \begin{bmatrix} \frac{\partial C}{\partial\theta_1}\\ \vdots\\ \frac{\partial C}{\partial\theta_d} \end{bmatrix}. \]Each component tells us how sensitive the cost is to a small change in one parameter. The sign tells us which direction changes the cost, while the magnitude tells us how strongly that parameter matters locally. The gradient points in the direction of steepest increase in \(C\). Therefore, the negative gradient
\[ -\nabla C(\theta) \]points in the direction of steepest decrease. Gradient descent repeatedly updates
\[ \theta \leftarrow \theta-\eta\nabla C(\theta). \]The learning rate \(\eta\) determines how far we move on each update. If \(\eta\) is too small, learning can be very slow; if it is too large, we may repeatedly overshoot a minimum rather than approaching it.
Why Learning is Helpful
The gradient has one component for every weight and bias in the network. It therefore provides a direction in parameter space telling us how all of these parameters should change together to reduce the cost. For a particular weight \(w_{ij}\),
\[ \frac{\partial C}{\partial w_{ij}} \]measures how a small change in that weight affects the cost. A large magnitude means changing this weight has a relatively large local effect, while a value near zero means changing it has little immediate effect.
The resulting values of the weights and biases encode whatever patterns the optimization process found useful for reducing the cost on the training data.
Generalization
Minimizing training cost does not by itself mean that the network has learned the general rule we intended. We care about whether the learned function also performs well on examples it did not see during training. We therefore distinguish between training data, used to adjust the parameters, and test data, used to evaluate the resulting network on unseen examples.
This also means that the representations learned by hidden layers need not correspond to the human-interpretable features we imagined when designing the architecture. Although we might motivate hidden layers as learning
\[ \text{pixels} \rightarrow \text{edges} \rightarrow \text{shapes} \rightarrow \text{digits}, \]gradient descent is not explicitly instructed to discover edges or shapes. It simply finds parameter changes that reduce the chosen cost on the training data. What internal representations emerge depends on the architecture, data and optimization process.
Gradient descent tells us what parameter changes we want:
\[ -\nabla C(\theta). \]The remaining problem is how to efficiently calculate all of the partial derivatives making up \(\nabla C\) in a large neural network. This is the role of backpropagation.