Hyperparameter Tuning with Python: Part 2
The implementation part!
In the first part of this series of blog posts, we have discussed a lot of things related to the concepts you need to know before performing hyperparameter tuning experiments, including:
- Evaluating Machine Learning Models
- Introducing Hyperparameter Tuning
- Exploring Exhaustive Search
- Exploring Bayesian Optimization
- Exploring Heuristic Search
- Exploring Multi-Fidelity Optimization
In this article, we will discuss more about how to utilize several powerful packages to implement all of the discussed hyperparameter tuning methods in the previous part. We will discuss the summary of the second section of the Hyperparameter Tuning with Python book which consists of 4 chapters:
- Chapter 7, Hyperparameter Tuning via scikit-learn
- Chapter 8, Hyperparameter Tuning via Hyperopt
- Chapter 9, Hyperparameter Tuning via Optuna
- Chapter 10, Advanced Hyperparameter Tuning with DEAP and Microsoft NNI
Without wasting any more time, let’s take a deep breath, make ourselves comfortable, and be ready to learn how to implement numerous hyperparameter tuning methods in Python!
Hyperparameter Tuning via scikit-learn
Scikit-learn (Sklearn) is one of the Python packages that is used the most by data scientists. This package provides a range of Machine Learning (ML)-related modules that are ready to be used with minimum effort, including for the task of hyperparameter tuning. One of the main selling points of this package is its consistent interface across many implemented classes, which almost every data scientist loves!
Apart from Sklearn, there are also other packages for the hyperparameter tuning task that are built on top of Sklearn or mimic the interface of Sklearn, such as scikit-optimize and scikit-hyperband.
In this chapter, we’ll learn about all of the important things to do with scikit-learn, scikit-optimize, and scikit-hyperband, along with how to utilize them to implement the hyperparameter tuning methods that we learned about in the previous chapters. We will learn how to implement the following hyperparameter tuning methods:
- Grid Search
- Random Search
- Coarse-to-Fine Search
- Succesive Halving
- Hyper Band
- Bayesian Optimization Gaussian Process
- Bayesian Optimization Random Forest
- Bayesian Optimization Gradient Boosted Trees
We’ll start by walking through how to install each of the packages. Then, we’ll learn not only how to utilize those packages with their default configurations but also discuss the available configurations along with their usage. Additionally, we’ll discuss how the implementation of the hyperparameter tuning methods is related to the theory that we learned in previous chapters, as there might be some minor differences or adjustments made in the implementation.
Finally, equipped with the knowledge from previous chapters, you will also be able to understand what’s happening if there are errors or unexpected results and understand how to set up the method configuration to match your specific problem.
Hyperparameter Tuning via Hyperopt
Hyperopt is an optimization package in Python that provides several implementations of hyperparameter tuning methods, including Random Search, Simulated Annealing (SA), Tree-Structured Parzen Estimators (TPE), and Adaptive TPE (ATPE). It also supports various types of hyperparameters with ranging types of sampling distributions.
In this chapter, we’ll introduce the Hyperopt package, starting with its capabilities and limitations, how to utilize it to perform hyperparameter tuning, and all the other important things you need to know about Hyperopt. We’ll learn not only how to utilize Hyperopt to perform hyperparameter tuning with its default configurations but also discuss the available configurations, along with their usage. Moreover, we’ll discuss how the implementation of the hyperparameter tuning methods is related
to the theory that we learned about in the first section (see the first part in this series of blog posts), since there some minor differences or adjustments may have been made in the implementation.
By the end of this chapter, you will be able to understand all the important things you need to know about Hyperopt and be able to implement various hyperparameter tuning methods available in this package. You’ll also be able to understand each of the important parameters of their classes and
how they are related to the theory that we learned about in the previous chapters. Finally, equipped with the knowledge from previous chapters, you will be able to understand what’s happening if there are errors or unexpected results, as well as how to set up the method configuration so that it matches your specific problem.
Hyperparameter Tuning via Optuna
Optuna is a Python package that provides various implementations of hyperparameter tuning methods, including but not limited to Grid Search, Random Search, and Tree-Structured Parzen Estimators (TPE). This package also enables us to create our own hyperparameter tuning method class and integrate it with other popular hyperparameter tuning packages, such as scikit-optimize.
In this chapter, you’ll be introduced to the Optuna package, starting with its numerous features, how to utilize it to perform hyperparameter tuning, and all of the other important things you need to know about Optuna. We will learn how to implement the following hyperparameter tuning methods:
- Tree-Structured Parzen Estimators (TPE)
- Random Search
- Grid Search
- Simulated Annealing
- Succesive Halving
- Hyper Band
Optuna has two main classes, namely samplers and pruners. Samplers are responsible for performing the hyperparameter tuning optimization, whereas pruners are responsible for judging whether we should prune the trials based on the reported values. In other words, pruners act like early stopping methods where we will stop a hyperparameter tuning iteration whenever it seems that there’s no additional benefit to continuing the process.
It is worth noting that the Optuna’s implementation of Succesive Halving and Hyper Band is different with the scikit’s. Here, Optuna is utilizing those methods as pruner which will support the main hyperparameter tuning method that is being used, the sampler.
By the end of this chapter, you will be able to understand all of the important things you need to know about Optuna and implement various hyperparameter tuning methods available in this package. You’ll also be able to understand each of the important parameters of the classes and how they are related to the theory that we have learned in previous chapters. Finally, equipped with the knowledge from previous chapters, you will also be able to understand what’s happening if there are errors or unexpected results and understand how to set up the method configuration to match your specific problem.
Advanced Hyperparameter Tuning with DEAP and Microsoft NNI
Distributed Evolutionary Algorithms in Python (DEAP) and Microsoft NNI are Python packages that provide various hyperparameter tuning methods that are not implemented in other packages that we have discussed in Chapters 7–9, including:
- Genetic Algorithm
- Particle Swarm Optimization
- Metis
- Sequential Model Algorithm Configuration (SMAC)
- Bayesian Optimization Hyper Band (BOHB)
- Population-Based Training
DEAP is a Python package that allows you to implement various evolutionary algorithms including (but not limited to) the Genetic Algorithm (GA) and Particle Swarm Optimization (PSO). DEAP allows you to craft your evolutionary algorithm optimization steps in a very flexible manner.
Neural Network Intelligence (NNI) is a package that is developed by Microsoft and can be utilized not only for hyperparameter tuning tasks but also for neural architecture search, model compression,
and feature engineering.
In this chapter, we’ll learn how to perform hyperparameter tuning using both DEAP and Microsoft NNI packages, starting from getting ourselves familiar with the packages, along with the important modules and parameters we need to be aware of. We’ll learn not only how to utilize both DEAP and Microsoft NNI to perform hyperparameter tuning with their default configurations but also discuss other available configurations along with their usage.
By the end of this chapter, you will be able to understand all of the important things you need to know about DEAP and Microsoft NNI and be able to implement various hyperparameter tuning methods available in these packages. You’ll also be able to understand each of the important parameters of the classes and how they are related to the theory that we have learned in the previous chapters. Finally, equipped with the knowledge from previous chapters, you will also be able to understand
what’s happening if there are errors or unexpected results and understand how to set up the method configuration to match your specific problem.
What you can expect from the book
The book consists of 14 chapters divided into 3 sections. This book covers all important things you need to know about hyperparameter tuning, starting from the concepts and theories, the implementations, and how to put things into practice. Alongside in-depth explanations of how each method works, you will use a decision map that can help you identify the best tuning method for your requirements. The book is also accompanied with code implementation which you can access freely from GitHub!
The book is intended for data scientists and Machine Learning engineers who are working with Python and want to further boost their ML model’s performance by utilizing the appropriate hyperparameter tuning method. You will need to have a basic understanding of ML and how to code in Python but will require no prior knowledge of hyperparameter tuning in Python.
About the Author
Louis Owen is a a data scientist/AI engineer from Indonesia who is always hungry for new knowledge. Throughout his career journey, he has worked in various fields of industry, including NGOs, e-commerce, conversational AI, OTA, Smart City, and FinTech. Outside of work, he loves to spend his time helping data science enthusiasts to become data scientists, either through his articles or through mentoring sessions. He also loves to spend his spare time doing his hobbies: watching movies and conducting side projects.
Currently, Louis is an NLP Research Engineer at Yellow.ai, a world’s leading CX automation platform. Check out Louis’ website to know more about him! Lastly, if you have any queries or any topics to be discussed, please reach out to Louis via LinkedIn.
