Machine Learning
Minibatch
A minibatch is a small subset of the training dataset used in each iteration of a machine learning model's training process. Instead of processing the entire dataset at once (batch gradient descent) or one example at a time (stochastic gradient descent), minibatches offer a compromise, providing a more efficient and stable learning process.
Explanation
In machine learning, especially when training deep neural networks, the entire training dataset is often too large to fit into memory or be processed efficiently in one go. Minibatch gradient descent divides the training data into small batches. The model's parameters (weights and biases) are updated after processing each minibatch. The size of the minibatch (e.g., 32, 64, 128, 256 examples) is a hyperparameter that can significantly impact training speed and convergence. Larger minibatches can leverage better vectorization and parallel processing capabilities of GPUs, leading to faster training. However, very large minibatches might lead to slower convergence or getting stuck in local minima. Smaller minibatches introduce more noise in the gradient estimation, which can help the model escape local minima but might also make the training process less stable. Minibatches provide a balance between the computational efficiency of batch gradient descent and the stochasticity of stochastic gradient descent.