Experimenting with different optimizers

The most popular and well optimizer is Stochastic Gradient Descent (SGD). This technique is widely used in other machine learning models as well. SGD is a to find minima or maxima by iteration. There are many popular variants of SGD that try to speed up convergence and less tuning by using an adaptive learning rate. The following table is an overview of the most commonly used optimizers in deep learning:

Optimizer	Hyperparameters	Comments
SGD	Learning rate, decay	+ Learning directly impacts performance (smaller learning rate avoids local minima) - Requires more manual tuning - Slow convergence
AdaGrad	Learning rate, epsilon, decay	+ Adaptive learning for all parameters (well suited for sparse data) - Learning becomes too small and stops learning
AdaDelta	Learning rate, rho, epsilon, decay	+ Faster convergence at start - Slows near minimum
Adam	Learning rate, beta 1, beta 2, epsilon, decay	+ Adaptive learning rate and momentum for all parameters
RMSprop	Learning rate...