Technology
Stochastic Gradient Descent: Understanding Weight Updates in Neural Networks
Stochastic Gradient Descent: Understanding Weight Updates in Neural Networks
In the realm of machine learning, particularly within the context of neural networks, Stochastic Gradient Descent (SGD) stands as a foundational optimization algorithm. This article delves into how weights in neural networks get updated during an iteration of SGD, addressing common misconceptions and providing clarity.
Understanding the Update Mechanism
The update of weights in a neural network through SGD is a process that hinges on the activation status of neurons during a given iteration. Unlike batch gradient descent, which uses the entire dataset to compute gradients, SGD processes one example or a small batch of examples at a time. This incremental approach allows for more dynamic and faster training.
Single Input Update
Backpropagation
Backpropagation is a critical component of the weight update process. During each iteration, gradients are computed for each weight based on the loss with respect to the output of the network for the specific input. The key point here is that if a neuron and its associated weights are not activated for a particular input, the gradient for those weights will be zero. Consequently, these weights will not be updated.
Weight Updates
The general formula for weight updates in SGD is given by:
Equation: w
- w
- η ??L/??w
Here, η is the learning rate, a hyperparameter that controls the size of the step in the direction of the gradient. If a weight is not involved in the current input's forward pass and thus the neuron is inactive, the gradient will be zero, meaning the weight will remain unchanged.
Batch Size Impact
In mini-batch SGD, the principle remains the same but the gradients are averaged over the mini-batch. This ensures that weights connected to active neurons across the mini-batch are updated, while weights connected to inactive neurons do not contribute to the gradient and remain unchanged.
In summary, during a single iteration of SGD, only the weights associated with activated neurons for the current input or mini-batch are updated. Non-activated weights do not receive an update in that iteration.
Impact of Weight Connections
It is important to recognize that weights are updated in proportion to their effect on the final loss. Weights whose connections ahead are inactive have a lower impact on the final loss, leading to a lower dloss/dw. Consequently, these weights will be moved very little, if at all, during the update process.
This article has provided a detailed exploration of how weights in neural networks get updated during SGD, demonstrating the critical dependency on the activation status of neurons. Understanding this mechanism is essential for anyone working with neural networks, as it significantly impacts the performance and efficiency of the learning process.