1.5.1
Gradient Descent: The Foundation
What is it?
Gradient descent is an optimization method used to reduce a model training loss. It calculates how the loss changes with respect to model parameters and updates those parameters in the direction that reduces the loss. Repeated updates help the model learn from training data.
Key points
1. The model makes predictions using its current parameters.
2. A loss function measures prediction error.
3. Backpropagation calculates gradients.
4. Gradients indicate how parameters affect the loss.
5. Parameters are updated in the opposite direction of the gradient.
6. Training repeats over batches of data.
Example: A simple parameter update
Suppose a model has a parameter value of 5.0 and its gradient is 2.0.
Using a learning rate of 0.1:
New parameter = Old parameter - (Learning rate × Gradient)
New parameter = 5.0 - (0.1 × 2.0)
New parameter = 4.8
This is a simplified single-parameter illustration of gradient descent.
Real-world application
Gradient descent is fundamental to training neural networks, including language models, image models and other deep learning systems.
Key takeaway
Gradient descent improves model parameters by repeatedly moving them in a direction that reduces training loss.