Working with batches and mini-batches

When training a neural network, we feed the training data to our network. Each full scan of the training data is called an epoch. If we feed all of the training data in one step, we call it batch mode (the batch size equals the size of the training set). However, in most cases, we divide the training data into smaller subsets while feeding the data to our model, just as in other machine learning algorithms. This is called mini-batch mode. Sometimes, we are forced to do this because the complete training set is too big and doesn't fit in the memory. If we look at the training time, we would say: the bigger the batch size, the better (as long as the batch fits in the memory). However, using mini-batches also has other advantages. Firstly, it reduces the complexity of the training process. Secondly, it reduces the effect of noise...