A Gentle Introduction to Ensemble Diversity for Machine Learning

Ensemble learning combines the predictions from machine learning models for nomenclature and regression.

We pursue using ensemble methods to unzip improved predictive performance, and it is this resurgence over any of the contributing models that defines whether an ensemble is good or not.

A property that is present in a good ensemble is the diversity of the predictions made by contributing models. Diversity is a slippery concept as it has not been precisely defined; nevertheless, it provides a useful practical heuristic for designing good ensemble models.

In this post, you will discover ensemble diversity in machine learning.

After reading this post, you will know:

• A good ensemble is one that has largest performance than any contributing model.
• Ensemble diversity is a property of a good ensemble where contributing models make variegated errors for the same input.
• Seeking self-sustaining models and uncorrelated predictions provides a guide for thinking well-nigh and introducing diversity into ensemble models.

Kick-start your project with my new typesetting Ensemble Learning Algorithms With Python, including step-by-step tutorials and the Python source code files for all examples.

Let’s get started.

Tutorial Overview

This tutorial is divided into three parts; they are:

1. What Makes a Good Ensemble
2. What Is Ensemble Diversity
3. Methods for Increasing Diversity

What Makes a Good Ensemble

An ensemble is a machine learning model that combines the predictions from multiple other models.

This often has the effect of reducing prediction error and improving the generalization of the model. But this is not unchangingly the case.

Sometimes the ensemble performs no largest than a well-performing contributing member to the ensemble. Plane worse, sometimes an ensemble will perform worse than any contributing member.

This raises the question as to what makes a good ensemble.

A good ensemble is an ensemble that performs largest than any contributing member. That is, it is a model that has lower prediction error for regression or higher verism for classification.

• Good Ensemble: A model that performs largest than any each contributing model.

This can be evaluated empirically using a train and test set or a resampling technique like k-fold cross-validation. The results can similarly be unscientific for each contributing model and the results compared directly to see if the definition of a “good ensemble” is met.

What properties of a good ensemble differentiate it from other ensembles that perform as good or worse than any contributing member?

This is a widely studied question with many ideas. The consistency is that a good ensemble has diversity.

What Is Ensemble Diversity

Ensemble diversity refers to differences in the decisions or predictions made by the ensemble members.

Ensemble diversity, that is, the difference among the individual learners, is a fundamental issue in ensemble methods.

— Page 99, Ensemble Methods, 2012.

Two ensemble members that make identical predictions are considered not diverse.

Ensembles that make completely variegated predictions in all cases are maximally diverse, although this is most unlikely.

Making variegated errors on any given sample is of paramount importance in ensemble-based systems. After all, if all ensemble members provide the same output, there is nothing to be gained from their combination.

— Page 5, Ensemble Machine Learning, 2012.

Therefore, some level of diversity in predictions is desired or plane required in order to construct a good ensemble. This is often simplified in discussion to the desire for diverse ensemble members, e.g. the models themselves, that in turn will produce diverse predictions, although diversity in predictions is truly what we seek.

Ideally, diversity would midpoint that the predictions made by each ensemble member are self-sustaining and uncorrelated.

… classifier outputs should be self-sustaining or preferably negatively correlated.

— Page 5, Ensemble Machine Learning, 2012.

Independence is a term from probability theory and refers to the specimen where one event does not influence the probability of the occurrence of flipside event. Events can influence each other in many variegated ways. One ensemble may influence flipside if it attempts to correct the predictions made by it.

As such, depending on the ensemble type, models may be naturally dependent or independent.

• Independence: Whether the occurrence of an event affects the probability of subsequent events.

Correlation is a term from statistics and refers to two variables waffly together. It is worldwide to summate a normalized correlation score between -1.0 and 1.0 where a score of 0.0 indicates no correlation, e.g. uncorrelated. A score of 1.0 or -1.0 one indicates perfect positive and negative correlation respectively and indicates two models that unchangingly predict the same (or inverse) outcome.

… if the learners are independent, i.e., [correlation] = 0, the ensemble will unzip a factor of T of error reduction than the individual learners; if the learners are totally correlated, i.e., [correlation] = 1, no gains can be obtained from the combination.

— Page 99, Ensemble Methods, 2012.

In practice, ensembles often walkout a weak or modest positive correlation in their predictions.

• Correlation: The stratum to which variables transpiration together.

As such, ensemble models are synthetic from plug-in models that may or may not have some dependency and some correlation between the decisions or predictions made. Constructing good ensembles is a challenging task.

For example, combining a tuft of top-performing models will likely result in a poor ensemble as the predictions made by the models will be highly correlated. Unintuitively, you might be largest off combining the predictions from a few top-performing individual models with the prediction from a few weaker models.

So, it is desired that the individual learners should be well-judged and diverse. Combining only well-judged learners is often worse than combining some well-judged ones together with some relatively weak ones, since complementarity is increasingly important than pure accuracy.

— Page 100, Ensemble Methods, 2012.

Sadly, there is no often well-set upon measure of ensemble diversity and ideas of independence and correlation are only guides or proxies for thinking well-nigh the properties of good ensembles.

Unfortunately, though diversity is crucial, we still do not have a well-spoken understanding of diversity; for example, currently there is no well-accepted formal definition of diversity.

— Page 100, Ensemble Methods, 2012.

Nevertheless, as guides, they are useful as we can devise techniques that struggle to reduce the correlation between the models.

Methods for Increasing Diversity

The objective for developing good ensembles is considered through the lens of increasing the diversity of ensemble members.

Models in an ensemble may be increasingly dependent if they share the same algorithm used to train the model and/or the same dataset used to train the model. Models in an ensemble may have higher correlation between their predictions for the same unstipulated reasons.

Therefore, approaches that make the models and/or the training data increasingly variegated are desirable.

Diversity may be obtained through variegated presentations of the input data, as in bagging, variations in learner design, or by subtracting a penalty to the outputs to encourage diversity.

— Page 93, Pattern Nomenclature Using Ensemble Methods, 2010.

It can be helpful to have a framework for thinking well-nigh techniques for managing the diversity of ensembles, and there are many to segregate from.

For example, Zhi-Hua Zhou in Chapter 5 of his 2012 typesetting titled “Ensemble Methods: Foundations and Algorithms” proposes a framework of four approaches to generating diversity. In summary, they are:

• Data Sample Manipulation: e.g. sample the training dataset differently for each model.
• Input Feature Manipulation: e.g. train each model on variegated groups of input features.
• Learning Parameter Manipulation: e.g. train models with variegated hyperparameter values.
• Output Representation Manipulation: e.g. train models with differently modified target values.

Perhaps the most worldwide approaches are changes to the training data, that is data samples and input features, often combined in a each approach.

For flipside example of a taxonomy for generating a diverse ensemble, Lior Rokach in Chapter 4 his 2010 typesetting titled “Pattern Nomenclature Using Ensemble Methods” suggest a similar taxonomy, summarized as:

• Manipulating the Inducer, e.g. manipulating in how models are trained.
  • Vary hyperparameters.
  • Varying starting point.
  • Vary optimization algorithm.
• Manipulating the Training Sample, e.g. manipulating the data used for training.
  • Resampling.
• Changing the target symbol representation, e.g. manipulating the target variable.
  • Vary encoding.
  • Error-correcting codes.
  • Label switching.
• Partitioning the search space, e.g. manipulating the number of input features.
  • Random subspace.
  • Feature selection.
• Hybridization, e.g. varying model types or a mixture of the whilom methods.

The hybridization tideway is worldwide both in popular ensemble methods that combine multiple approaches at generating diversity in a each algorithm, as well as simply varying the algorithms (model types) that subsume the ensemble.

Bundling competing models into ensembles scrutinizingly unchangingly improves generalization – and using variegated algorithms as the perturbation operator is an constructive way to obtain the requisite diversity of components.

— Page 89, Ensemble Methods in Data Mining, 2010.

This highlights that although we could investigate quantifying how diverse an ensemble is, that instead, most effort is on developing techniques that generate diversity.

We can harness this knowledge in developing our own ensemble methods, first trying standard and well-understood ensemble learning methods, then tailoring them for our explicit dataset with model independence and correlation of predictions in mind in an effort to get the most out of them.

Further Reading

This section provides increasingly resources on the topic if you are looking to go deeper.

Books

Articles

Summary

In this post, you discovered ensemble diversity in machine learning.

Specifically, you learned:

• A good ensemble is one that has largest performance than any contributing model.
• Ensemble diversity is a property of a good ensemble where contributing models make variegated errors for the same input.
• Seeking self-sustaining models and uncorrelated predictions provides a guide for thinking well-nigh and introducing diversity into ensemble models.

Do you have any questions?
Ask your questions in the comments unelevated and I will do my weightier to answer.