There are a couple of approaches to be used when it comes to model-based hyperparameter tuning and these approaches come together under Sequential Model-based Global Optimization (SMBO).
When you think about GridSearchCV or RandomizedSearchCV, you may rightfully feel that the way they cross validate hyperparameters is not very smart. Both pre-define sets of hyperparameters to be validated during training time and are not designed to benefit from the information that they might get during training. If you could find a way to learn from previous iterations of hyperparameter validation based on model performance, then you would have an idea about which hyperparameter set is likely to give a better performance in the next iteration.
SMBO approaches emanated from this reasoning and Bayesian-based hyperparameter optimization is one of these approaches...
