Machine learning and numerical optimization
So far, we have discussed numerical optimization and its use cases from an optimization problems perspective. Whereas numerical optimization has several standalone industry use cases and applications, it is also very commonly used in several machine learning algorithms and use cases. Whether it’s supervised learning, unsupervised learning, or reinforcement, we are always solving some form of optimization problem using iterative processes at the very core of a machine learning algorithm.
In supervised learning, for example, let’s look at the case of linear regression. In linear regression, we are minimizing a cost function consisting generally of the mean squared error between the actual value of a target variable and the value predicted via the model.
Our algorithm arrives at the minimum value of the cost function (convex function with a global minimum if it is mean squared error, non-convex with local minima in most other...
