Last Updated on March 29, 2021

Machine learning algorithms have hyperparameters that indulge the algorithms to be tailored to explicit datasets.

Although the impact of hyperparameters may be understood generally, their explicit effect on a dataset and their interactions during learning may not be known. Therefore, it is important to tune the values of algorithm hyperparameters as part of a machine learning project.

It is worldwide to use naive optimization algorithms to tune hyperparameters, such as a grid search and a random search. An unorganized tideway is to use a stochastic optimization algorithm, like a stochastic hill climbing algorithm.

In this tutorial, you will discover how to manually optimize the hyperparameters of machine learning algorithms.

After completing this tutorial, you will know:

  • Stochastic optimization algorithms can be used instead of grid and random search for hyperparameter optimization.
  • How to use a stochastic hill climbing algorithm to tune the hyperparameters of the Perceptron algorithm.
  • How to manually optimize the hyperparameters of the XGBoost gradient boosting algorithm.

Let’s get started.

How to Manually Optimize Machine Learning Model Hyperparameters

How to Manually Optimize Machine Learning Model Hyperparameters
Photo by john farrell macdonald, some rights reserved.

Tutorial Overview

This tutorial is divided into three parts; they are:

  1. Manual Hyperparameter Optimization
  2. Perceptron Hyperparameter Optimization
  3. XGBoost Hyperparameter Optimization

Manual Hyperparameter Optimization

Machine learning models have hyperparameters that you must set in order to customize the model to your dataset.

Often, the unstipulated effects of hyperparameters on a model are known, but how to weightier set a hyperparameter and combinations of interacting hyperparameters for a given dataset is challenging.

A largest tideway is to objectively search variegated values for model hyperparameters and segregate a subset that results in a model that achieves the weightier performance on a given dataset. This is tabbed hyperparameter optimization, or hyperparameter tuning.

A range of variegated optimization algorithms may be used, although two of the simplest and most worldwide methods are random search and grid search.

  • Random Search. Pinpoint a search space as a regional domain of hyperparameter values and randomly sample points in that domain.
  • Grid Search. Pinpoint a search space as a grid of hyperparameter values and evaluate every position in the grid.

Grid search is unconfined for spot-checking combinations that are known to perform well generally. Random search is unconfined for discovery and getting hyperparameter combinations that you would not have guessed intuitively, although it often requires increasingly time to execute.

For increasingly on grid and random search for hyperparameter tuning, see the tutorial:

Grid and random search are primitive optimization algorithms, and it is possible to use any optimization we like to tune the performance of a machine learning algorithm. For example, it is possible to use stochastic optimization algorithms. This might be desirable when good or unconfined performance is required and there are sufficient resources misogynist to tune the model.

Next, let’s squint at how we might use a stochastic hill climbing algorithm to tune the performance of the Perceptron algorithm.

Perceptron Hyperparameter Optimization

The Perceptron algorithm is the simplest type of strained neural network.

It is a model of a each neuron that can be used for two-class nomenclature problems and provides the foundation for later developing much larger networks.

In this section, we will explore how to manually optimize the hyperparameters of the Perceptron model.

First, let’s pinpoint a synthetic binary nomenclature problem that we can use as the focus of optimizing the model.

We can use the make_classification() function to pinpoint a binary nomenclature problem with 1,000 rows and five input variables.

The example unelevated creates the dataset and summarizes the shape of the data.

Running the example prints the shape of the created dataset, confirming our expectations.

The scikit-learn provides an implementation of the Perceptron model via the Perceptron class.

Before we tune the hyperparameters of the model, we can establish a baseline in performance using the default hyperparameters.

We will evaluate the model using good practices of repeated stratified k-fold cross-validation via the RepeatedStratifiedKFold class.

The well-constructed example of evaluating the Perceptron model with default hyperparameters on our synthetic binary nomenclature dataset is listed below.

Running the example reports evaluates the model and reports the midpoint and standard deviation of the nomenclature accuracy.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the stereotype outcome.

In this case, we can see that the model with default hyperparameters achieved a nomenclature verism of well-nigh 78.5 percent.

We would hope that we can unzip largest performance than this with optimized hyperparameters.

Next, we can optimize the hyperparameters of the Perceptron model using a stochastic hill climbing algorithm.

There are many hyperparameters that we could optimize, although we will focus on two that perhaps have the most impact on the learning policies of the model; they are:

  • Learning Rate (eta0).
  • Regularization (alpha).

The learning rate controls the value the model is updated based on prediction errors and controls the speed of learning. The default value of eta is 1.0. reasonable values are larger than zero (e.g. larger than 1e-8 or 1e-10) and probably less than 1.0

By default, the Perceptron does not use any regularization, but we will enable “elastic net” regularization which applies both L1 and L2 regularization during learning. This will encourage the model to seek small model weights and, in turn, often largest performance.

We will tune the “alpha” hyperparameter that controls the weighting of the regularization, e.g. the value it impacts the learning. If set to 0.0, it is as though no regularization is stuff used. Reasonable values are between 0.0 and 1.0.

First, we need to pinpoint the objective function for the optimization algorithm. We will evaluate a configuration using midpoint nomenclature verism with repeated stratified k-fold cross-validation. We will seek to maximize verism in the configurations.

The objective() function unelevated implements this, taking the dataset and a list of config values. The config values (learning rate and regularization weighting) are unpacked, used to configure the model, which is then evaluated, and the midpoint verism is returned.

Next, we need a function to take a step in the search space.

The search space is specified by two variables (eta and alpha). A step in the search space must have some relationship to the previous values and must be unseat to sensible values (e.g. between 0 and 1).

We will use a “step size” hyperparameter that controls how far the algorithm is unliable to move from the existing configuration. A new configuration will be chosen probabilistically using a Gaussian distribution with the current value as the midpoint of the distribution and the step size as the standard deviation of the distribution.

We can use the randn() NumPy function to generate random numbers with a Gaussian distribution.

The step() function unelevated implements this and will take a step in the search space and generate a new configuration using an existing configuration.

Next, we need to implement the stochastic hill climbing algorithm that will undeniability our objective() function to evaluate candidate solutions and our step() function to take a step in the search space.

The search first generates a random initial solution, in this specimen with eta and start values in the range 0 and 1. The initial solution is then evaluated and is taken as the current weightier working solution.

Next, the algorithm iterates for a stock-still number of iterations provided as a hyperparameter to the search. Each iteration involves taking a step and evaluating the new candidate solution.

If the new solution is largest than the current working solution, it is taken as the new current working solution.

At the end of the search, the weightier solution and its performance are then returned.

Tying this together, the hillclimbing() function unelevated implements the stochastic hill climbing algorithm for tuning the Perceptron algorithm, taking the dataset, objective function, number of iterations, and step size as arguments.

We can then undeniability the algorithm and report the results of the search.

In this case, we will run the algorithm for 100 iterations and use a step size of 0.1, chosen without a little trial and error.

Tying this together, the well-constructed example of manually tuning the Perceptron algorithm is listed below.

Running the example reports the configuration and result each time an resurgence is seen during the search. At the end of the run, the weightier configuration and result are reported.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the stereotype outcome.

In this case, we can see that the weightier result involved using a learning rate slightly whilom 1 at 1.004 and a regularization weight of well-nigh 0.002 achieving a midpoint verism of well-nigh 79.1 percent, largest than the default configuration that achieved an verism of well-nigh 78.5 percent.

Can you get a largest result?
Let me know in the comments below.

Now that we are familiar with how to use a stochastic hill climbing algorithm to tune the hyperparameters of a simple machine learning algorithm, let’s squint at tuning a increasingly wide algorithm, such as XGBoost.

XGBoost Hyperparameter Optimization

XGBoost is short for Extreme Gradient Boosting and is an efficient implementation of the stochastic gradient boosting machine learning algorithm.

The stochastic gradient boosting algorithm, moreover tabbed gradient boosting machines or tree boosting, is a powerful machine learning technique that performs well or plane weightier on a wide range of challenging machine learning problems.

First, the XGBoost library must be installed.

You can install it using pip, as follows:

Once installed, you can personize that it was installed successfully and that you are using a modern version by running the pursuit code:

Running the code, you should see the pursuit version number or higher.

Although the XGBoost library has its own Python API, we can use XGBoost models with the scikit-learn API via the XGBClassifier wrapper class.

An instance of the model can be instantiated and used just like any other scikit-learn matriculation for model evaluation. For example:

Before we tune the hyperparameters of XGBoost, we can establish a baseline in performance using the default hyperparameters.

We will use the same synthetic binary nomenclature dataset from the previous section and the same test harness of repeated stratified k-fold cross-validation.

The well-constructed example of evaluating the performance of XGBoost with default hyperparameters is listed below.

Running the example evaluates the model and reports the midpoint and standard deviation of the nomenclature accuracy.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the stereotype outcome.

In this case, we can see that the model with default hyperparameters achieved a nomenclature verism of well-nigh 84.9 percent.

We would hope that we can unzip largest performance than this with optimized hyperparameters.

Next, we can transmute the stochastic hill climbing optimization algorithm to tune the hyperparameters of the XGBoost model.

There are many hyperparameters that we may want to optimize for the XGBoost model.

For an overview of how to tune the XGBoost model, see the tutorial:

We will focus on four key hyperparameters; they are:

  • Learning Rate (learning_rate)
  • Number of Trees (n_estimators)
  • Subsample Percentage (subsample)
  • Tree Depth (max_depth)

The learning rate controls the contribution of each tree to the ensemble. Sensible values are less than 1.0 and slightly whilom 0.0 (e.g. 1e-8).

The number of trees controls the size of the ensemble, and often, increasingly trees is largest to a point of diminishing returns. Sensible values are between 1 tree and hundreds or thousands of trees.

The subsample percentages pinpoint the random sample size used to train each tree, specified as a percentage of the size of the original dataset. Values are between a value slightly whilom 0.0 (e.g. 1e-8) and 1.0

The tree depth is the number of levels in each tree. Deeper trees are increasingly explicit to the training dataset and perhaps overfit. Shorter trees often generalize better. Sensible values are between 1 and 10 or 20.

First, we must update the objective() function to unpack the hyperparameters of the XGBoost model, configure it, and then evaluate the midpoint nomenclature accuracy.

Next, we need to pinpoint the step() function used to take a step in the search space.

Each hyperparameter is quite a variegated range, therefore, we will pinpoint the step size (standard deviation of the distribution) separately for each hyperparameter. We will moreover pinpoint the step sizes in line rather than as arguments to the function, to alimony things simple.

The number of trees and the depth are integers, so the stepped values are rounded.

The step sizes chosen are arbitrary, chosen without a little trial and error.

The updated step function is listed below.

Finally, the hillclimbing() algorithm must be updated to pinpoint an initial solution with towardly values.

In this case, we will pinpoint the initial solution with sensible defaults, matching the default hyperparameters, or tropical to them.

Tying this together, the well-constructed example of manually tuning the hyperparameters of the XGBoost algorithm using a stochastic hill climbing algorithm is listed below.

Running the example reports the configuration and result each time an resurgence is seen during the search. At the end of the run, the weightier configuration and result are reported.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the stereotype outcome.

In this case, we can see that the weightier result involved using a learning rate of well-nigh 0.02, 52 trees, a subsample rate of well-nigh 50 percent, and a large depth of 53 levels.

This configuration resulted in a midpoint verism of well-nigh 87.3 percent, largest than the default configuration that achieved an verism of well-nigh 84.9 percent.

Can you get a largest result?
Let me know in the comments below.

Further Reading

This section provides increasingly resources on the topic if you are looking to go deeper.

Tutorials

APIs

Articles

Summary

In this tutorial, you discovered how to manually optimize the hyperparameters of machine learning algorithms.

Specifically, you learned:

  • Stochastic optimization algorithms can be used instead of grid and random search for hyperparameter optimization.
  • How to use a stochastic hill climbing algorithm to tune the hyperparameters of the Perceptron algorithm.
  • How to manually optimize the hyperparameters of the XGBoost gradient boosting algorithm.

Do you have any questions?
Ask your questions in the comments unelevated and I will do my weightier to answer.