Ensemble Learning Algorithm Complexity and Occam’s Razor



Occam’s razor suggests that in machine learning, we should prefer simpler models with fewer coefficients over complex models like ensembles.

Taken at face value, the razor is a heuristic that suggests more complex hypotheses make more assumptions that, in turn, will make them too narrow and not generalize well. In machine learning, it suggests complex models like ensembles will overfit the training dataset and perform poorly on new data.

In practice, ensembles are almost universally the type of model chosen on projects where predictive skill is the most important consideration. Further, empirical results show a continued reduction in generalization error as the complexity of an ensemble learning model is incrementally increased. These findings are at odds with the Occam’s razor principle taken at face value.

In this tutorial, you will discover how to reconcile Occam’s Razor with ensemble machine learning.

After completing this tutorial, you will know:

  • Occam’s razor is a heuristic that suggests choosing simpler machine learning models as they are expected to generalize better.
  • The heuristic can be divided into two razors, one of which is true and remains a useful tool and the other that is false and should be abandoned.
  • Ensemble learning algorithms like boosting provide a specific case of how the second razor fails and added complexity can result in lower generalization error.

Let’s get started.

Ensemble Learning Algorithm Complexity and Occam's Razor

Ensemble Learning Algorithm Complexity and Occam’s Razor
Photo by dylan_odonnell, some rights reserved.

Tutorial Overview

This tutorial is divided into three parts; they are:

  1. Occam’s Razor for Model Selection
  2. Occam’s Two Razors for Machine Learning
  3. Occam’s Razor and Ensemble Learning

Occam’s Razor for Model Selection

Model selection is the process of choosing one from among possibly many candidate machine learning models for a predictive modeling project.

It is often straightforward to select a model based on its expected performance, e.g. choose the model with the highest accuracy or lowest prediction error.

Another important consideration is to choose simpler models over complex models.

Simpler models are typically defined as models that make fewer assumptions or have fewer elements, most commonly characterized as fewer coefficients (e.g. rules, layers, weights, etc.). The rationale for choosing simpler models is tied back to Occam’s Razor.

The idea is that the best scientific theory is the smallest one that explains all the facts.

— Page 197, Data Mining: Practical Machine Learning Tools and Techniques, 2016.

Occam’s Razor is an approach to problem-solving and is commonly invoked to mean that if all else is equal, we should prefer the simpler solutions.

  • Occam’s Razor: If all else is equal, the simplest solution is correct.

It is named for William of Ockham and was proposed to counter ever more elaborate philosophy without equivalent increases in predictive power.

William of Occam’s famous razor states that “Nunquam ponenda est pluralitas sin necesitate,” which, approximately translated, means “Entities should not be multiplied beyond necessity”.

Occam’s Two Razors: The Sharp and the Blunt, 1998.

It is not a rule, more of a heuristic for problem-solving, and is commonly invoked in science to prefer simpler hypotheses that make fewer assumptions over more complex hypotheses that make more assumptions.

There is a long-standing tradition in science that, other things being equal, simple theories are preferable to complex ones. This is known as Occam’s Razor after the medieval philosopher William of Occam (or Ockham).

— Page 197, Data Mining: Practical Machine Learning Tools and Techniques, 2016.

The problem with complex hypotheses with more assumptions is that they are likely too specific.

They may include details of specific cases that are at hand or easily available, and in turn, may not generalize to new cases. That is, the more assumptions a hypothesis has, the more narrow it is expected to be in its application. Conversely, fewer assumptions suggests a more general hypothesis with greater predictive power to more cases.

  • Simple Hypothesis: Fewer assumptions, and in turn, broad applicability.
  • Complex Hypothesis: More assumptions, and in turn, narrow applicability.

This has implications in machine learning, as we are specifically trying to generalize to new unseen cases from specific observations, referred to as inductive reasoning.

If Occam’s Razor suggests that more complex models don’t generalize well, then in applied machine learning, it suggests we should choose simpler models as they will have lower prediction errors on new data.

If this is true, then how can we justify using an ensemble machine learning algorithm?

By definition, ensemble machine learning algorithms are more complex than a single machine learning model, as they are composed of many individual machine learning models.

Occam’s razor suggests that the added complexity of ensemble learning algorithms means that they will not generalize as well as simpler models fit on the same dataset.

Yet ensemble machine learning algorithms are the dominant solution when predictive skill on new data is the most important concern, such as machine learning competitions. Ensembles have been studied at great length and have been shown not to overfit the training dataset in study after study.

It has been empirically observed that certain ensemble techniques often do not overfit the model, even when the ensemble contains thousands of classifiers.

— Page 40, Pattern Classification Using Ensemble Methods, 2010.

How can this inconsistency be reconciled?

Occam’s Two Razors for Machine Learning

The conflict between the expectation of simpler models generalizing better in theory and complex models like ensembles generalizing better in practice was mostly ignored as an inconvenient empirical finding for a long time.

In the late 1990s, the problem was specifically studied by Pedro Domingos and published in the award-winning 1996 paper titled “Occam’s Two Razors: The Sharp and the Blunt,” and follow-up 1999 journal article “The Role of Occam’s Razor in Knowledge Discovery.”

In the work, Domingos defines the problem as two specific commonly asserted implications of Occam’s Razor in applied machine learning, which he refers to as “Occam’s Two Razors” in machine learning, they are (taken from the paper):

  • First razor: Given two models with the same generalization error, the simpler one should be preferred because simplicity is desirable in itself.
  • Second razor: Given two models with the same training-set error, the simpler one should be preferred because it is likely to have lower generalization error.

Domingos then enumerates a vast number of examples for and against each razor from both theory and empirical studies in machine learning.

The first razor suggests if two models have the same expected performance on data not seen during training, we should prefer the simpler model. Domingos highlights that this razor holds and provides a good heuristic on machine learning projects.

The second razor suggests that if two models have the same performance on a training dataset, then the simpler model should be chosen because it is expected to generalize better when used to make predictions on new data.

This seems sensible on the surface.

It is the argument behind not adopting ensemble algorithms in a machine learning project because they are very complex compared to other models and expected to not generalize.

It turns out that this razor cannot be supported by the evidence from the machine learning literature.

All of this evidence points to the conclusion that not only is the second razor not true in general; it is also typically false in the types of domains KDD has been applied to.

Occam’s Two Razors: The Sharp and the Blunt, 1998.

Occam’s Razor and Ensemble Learning

The finding begins to sound intuitive once you mull on it for a while.

For example, in practice, we would not choose a machine learning model based on its performance on the training dataset alone. We intuitively, or perhaps after a lot of experience, tacitly expect the estimate of performance on a training set to be a poor estimate of performance on a holdout dataset.

We have this expectation because the model can overfit the training dataset.

Yet, less intuitively, overfitting the training dataset can lead to better performance on a holdout test set. This has been observed many times in practice in systematic studies.

A common situation involves plotting the performance of a model on the training dataset and a holdout test dataset each iteration of learning for the model, such as training epochs or iterations for models that support incremental learning.

If learning on the training dataset is set up to continue for a large number of training iterations and the curves observed, it can often be seen that performance on the training dataset will fall to zero error. This is to be expected as we might think that the model will overfit the training dataset given enough resources and time to train. Yet the performance on the test set will continue to improve, even while the performance on the training set remains fixed at zero error.

… occasionally, the generalization error would continue to improve long after the training error had reached zero.

— Page 40, Ensemble Methods in Data Mining, 2010.

This behavior can be observed with ensemble learning algorithms like boosting and bagging, where performance on the holdout dataset will continue to improve as additional model members are added to the ensemble.

One very surprising finding is that performing more boosting iterations can reduce the error on new data long after the classification error of the combined classifier on the training data has dropped to zero.

— Page 489, Data Mining: Practical Machine Learning Tools and Techniques, 2016.

That is, the model complexity is incrementally increased, which systematically decreases the error on unseen data, e.g. generalization error. The additional training cannot improve the performance on the training dataset; it has no possible improvement to make.

Performing more boosting iterations without reducing training error does not explain the training data any better, and it certainly adds complexity to the combined classifier.

— Page 490, Data Mining: Practical Machine Learning Tools and Techniques, 2016.

This finding directly contradicts the second razor and supports Domingos’ argument about abandoning the second razor.

The first one is largely uncontroversial, while the second one, taken literally, is false.

Occam’s Two Razors: The Sharp and the Blunt, 1998.

This problem has been studied and can generally be explained by the ensemble algorithms learning to be more confident in their predictions on the training dataset, which carry over to the hold out data.

The contradiction can be resolved by considering the classifier’s confidence in its predictions.

— Page 490, Data Mining: Practical Machine Learning Tools and Techniques, 2016.

The first razor remains an important heuristic in applied machine learning.

The key aspect of this razor is the predicate of “all else being equal.” That is, if two models are compared, they must be compared using their generalization error on a holdout dataset or estimated using k-fold cross-validation. If their performance is equal under these circumstances, then the razor can come into effect and we can choose the simpler solution.

This is not the only way to choose models.

We may choose a simpler model because it is easier to interpret, and this remains valid if model interpretability is a more important project requirement than predictive skill.

Ensemble learning algorithms are unambiguously a more complex type of model when the number of model parameters is considered the measure of complexity. As such, an open problem in machine learning involves alternate measures of complexity.

Further Reading

This section provides more resources on the topic if you are looking to go deeper.

Related Tutorials

Papers

Books

Articles

Summary

In this tutorial, you discovered how to reconcile Occam’s Razor with ensemble machine learning.

Specifically, you learned:

  • Occam’s razor is a heuristic that suggests choosing simpler machine learning models as they are expected to generalize better.
  • The heuristic can be divided into two razors, one of which is true and remains a useful tool, and the other that is false and should be abandoned.
  • Ensemble learning algorithms like boosting provide a specific case of how the second razor fails and added complexity can result in lower generalization error.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.



Author: Shantun Parmar

142 thoughts on “Ensemble Learning Algorithm Complexity and Occam’s Razor

  1. I got this site from my friend who told me about this site and now this time I am
    browsing this site and reading very informative posts at this time.

  2. What i don’t realize is in fact how you are not actually a lot more neatly-appreciated than you may
    be now. You’re very intelligent. You understand therefore significantly on the subject of this topic, produced
    me for my part believe it from numerous numerous angles.
    Its like men and women aren’t fascinated except it’s something
    to do with Girl gaga! Your individual stuffs nice.
    All the time handle it up!

    Have a look at my site Full Body Male Enhancement

  3. Hi there, just became alert to your blog through Google, and found that it’s truly informative.
    I’m gonna watch out for brussels. I’ll appreciate if you continue this in future.
    Numerous people will be benefited from your writing.
    Cheers!

  4. Wow that was strange. I just wrote an extremely long comment but after I clicked submit my comment didn’t appear.
    Grrrr… well I’m not writing all that over again. Anyway, just wanted
    to say wonderful blog!

  5. Hmm it seems like your website ate my first comment (it was extremely long) so I guess I’ll just sum it up what I submitted and
    say, I’m thoroughly enjoying your blog. I too am an aspiring blog blogger but I’m still
    new to everything. Do you have any tips for newbie blog
    writers? I’d genuinely appreciate it.

    my page :: Luminas Pain Patch

  6. I’m excited to uncover this web site. I need to to thank you for your
    time for this particularly fantastic read!!
    I definitely savored every little bit of it and i also have you
    saved to fav to look at new information in your blog.

    my page – Jolly CBD

  7. My wife and i got so satisfied Ervin managed to finish up his web research out of the ideas
    he grabbed using your weblog. It’s not at all simplistic to simply find yourself giving
    for free tricks which many others have been making money
    from. And we also remember we now have the website owner to appreciate because of that.
    The explanations you’ve made, the straightforward blog menu,
    the relationships you help engender – it’s got most
    great, and it’s making our son in addition to our family reckon that that matter is brilliant, and that is incredibly serious.
    Many thanks for the whole thing!

    Also visit my blog Quick Shred Keto

  8. Nice post. I was checking constantly this blog and I’m
    inspired! Very useful information specifically the closing section 🙂 I take care of such info much.
    I was looking for this particular information for a very lengthy time.
    Thank you and best of luck.

  9. Amazing blog! Do you have any hints for aspiring writers?
    I’m planning to start my own blog soon but I’m a little lost on everything.
    Would you advise starting with a free platform like WordPress or go for a paid
    option? There are so many choices out there that I’m totally overwhelmed ..
    Any tips? Thanks!

  10. I am curious to find out what blog system you have been utilizing?
    I’m experiencing some minor security problems with my latest blog
    and I’d like to find something more secure. Do you have any recommendations?

    Here is my website – CamHandy Reviews

  11. Do you mind if I quote a couple of your articles as long as I provide
    credit and sources back to your webpage? My website is in the very same area of interest as yours and my users
    would certainly benefit from a lot of the information you provide here.
    Please let me know if this alright with you. Cheers!

    My web page – Testoryze

  12. Hello! I could have sworn I?ve visited this site
    before but after browsing through many of the articles I realized it?s new to me.
    Anyways, I?m certainly happy I discovered it and I?ll
    be book-marking it and checking back often!

    my web blog – Jolly CBD Reviews

  13. Heya superb website! Does running a blog like this take a lot of work?
    I’ve absolutely no expertise in programming but I had been hoping to start my own blog soon.
    Anyways, should you have any recommendations or techniques for new blog owners please share.

    I know this is off topic nevertheless I just needed to ask.

    Cheers!

  14. Definitely consider that which you stated. Your favorite justification appeared to
    be at the internet the easiest factor to bear in mind of.
    I say to you, I definitely get annoyed at the same time as other folks think about issues that they plainly do not
    understand about. You managed to hit the nail upon the highest and outlined out the entire
    thing with no need side effect , people could take
    a signal. Will probably be back to get more. Thanks

  15. Hello, I think your blog might be having browser compatibility issues.
    When I look at your website in Safari, it looks fine but
    when opening in Internet Explorer, it has some overlapping.
    I just wanted to give you a quick heads up! Other then that, awesome blog!

  16. I know this if off topic but I’m looking into starting my own weblog and was wondering what all is
    needed to get set up? I’m assuming having a blog like
    yours would cost a pretty penny? I’m not very web savvy so
    I’m not 100% positive. Any recommendations or advice would be greatly appreciated.
    Thanks

  17. Oh my goodness! Amazing article dude! Thank you
    so much, However I am encountering issues with
    your RSS. I don’t understand why I cannot subscribe to it.
    Is there anyone else having similar RSS problems?

    Anyone who knows the answer will you kindly respond?
    Thanks!!

  18. Hi there to every single one, it’s genuinely a nice for me to visit this website,
    it consists of precious Information.

  19. Excellent post. I was checking continuously this blog and I’m impressed!
    Very useful info specially the last part 🙂 I care for such info a lot.
    I was looking for this certain information for a very long time.
    Thank you and good luck.

  20. Wow, awesome weblog format! How lengthy have you been running a blog for?
    you made blogging glance easy. The overall look
    of your site is fantastic, as well as the content![X-N-E-W-L-I-N-S-P-I-N-X]I simply could not
    leave your website prior to suggesting that I really loved the
    usual info an individual provide for your guests? Is gonna be
    back steadily in order to inspect new posts.

    My webpage; Full Body Male Enhancement

  21. whoah this blog is great i love studying your posts.
    Keep up the good work! You recognize, many people are looking round for this info, you can aid them greatly.

  22. I must thank you for the efforts you’ve put in penning this blog.
    I am hoping to see the same high-grade blog posts from you later on as well.
    In truth, your creative writing abilities has encouraged
    me to get my own site now 😉

  23. What i don’t understood is if truth be told how you are now not really much more well-favored than you might be right now.
    You are very intelligent. You recognize therefore
    significantly when it comes to this topic, produced me personally consider it
    from numerous varied angles. Its like men and women aren’t interested except it is
    one thing to accomplish with Girl gaga! Your own stuffs excellent.

    Always maintain it up!

  24. Hi there I am so excited I found your website, I
    really found you by mistake, while I was looking on Yahoo for something else, Nonetheless
    I am here now and would just like to say cheers for
    a fantastic post and a all round interesting blog (I also love the theme/design), I don’t have
    time to read it all at the moment but I have saved it
    and also added your RSS feeds, so when I have time I will be back to read more,
    Please do keep up the great job.

    My website … Jolly CBD

  25. I’m not that much of a internet reader to be honest but
    your sites really nice, keep it up! I’ll go ahead and bookmark your site to come back down the
    road. Many thanks

  26. I seldom comment, however i did some searching and wound up here Ensemble
    Learning Algorithm Complexity and Occam’s
    Razor – Cooding Dessign. And I do have a couple
    of questions for you if it’s allright. Is it simply me or does it
    appear like a few of the comments come across
    like coming from brain dead visitors? 😛 And, if you are writing at additional online sites, I would like
    to follow anything new you have to post. Would you list of every one of all your communal pages
    like your Facebook page, twitter feed, or linkedin profile?

    Here is my site: Jolly CBD Gummies

  27. Fantastic beat ! I would like to apprentice while you amend your site, how could i subscribe for a blog site?

    The account aided me a acceptable deal. I
    had been a little bit acquainted of this your broadcast provided bright clear idea

    Feel free to surf to my web-site :: Jolly CBD

  28. I blog frequently and I seriously thank you for your information. The
    article has truly peaked my interest. I’m going to book mark
    your website and keep checking for new details about once per week.
    I opted in for your Feed as well.

  29. Hmm it seems like your website ate my first comment (it was extremely
    long) so I guess I’ll just sum it up what I submitted and say,
    I’m thoroughly enjoying your blog. I as well am an aspiring blog blogger
    but I’m still new to everything. Do you have any helpful hints for newbie blog
    writers? I’d definitely appreciate it.

    Visit my website – Luminas Pain Patch

  30. If you’re ever suffered from sleeping difficulties caused by insomnia,
    a sleep disorder or stress and worries in your life, you will know just
    how frustrating this can be.
    Lying in bed staring at the ceiling only gives you more time to worry your problems and as the
    night passes you grow more and more anxious about the fact that you just can’t get the
    sleep you desperately need.
    The longer this problem goes on the worse it becomes and you become transformed into a living zombie,
    miserably trying to get through the day on very little sleep.
    Yet despite the fact that you are completely exhausted, the next night the whole process
    starts all over again and you toss and turn, begging
    for sleep to descend on you.
    Luckily there is a solution that can break this vicious
    cycle and help you to get the deep, peaceful sleep that your body
    needs and craves.

  31. I was suggested this web site by my cousin. I am now
    not sure whether or not this put up is written through him as no one else
    realize such designated approximately my trouble.
    You are wonderful! Thanks!

  32. Sweet blog! I found it while browsing on Yahoo News.

    Do you have any suggestions on how to get listed in Yahoo News?
    I’ve been trying for a while but I never seem to get there!
    Thanks

    my web-site :: Leonardo

  33. I liked up to you will receive carried out proper here.
    The comic strip is attractive, your authored material stylish.
    nonetheless, you command get bought an shakiness over that you wish be handing over the following.
    sick unquestionably come more in the past once more as precisely the same
    just about a lot frequently within case you shield this hike.

    Here is my site … Jolly CBD Gummies| Jolly CBD| Jolly CBD Gummies Review| Jolly CBD Gummies Reviews| Jolly CBD Gummies Cost}

  34. I cherished as much as you will receive performed proper
    here. The comic strip is attractive, your authored subject
    matter stylish. however, you command get got an impatience over
    that you wish be turning in the following. in poor health surely come further in the past once more since
    exactly the same nearly very frequently within case you defend this hike.

    Here is my webpage Jolly CBD Gummies| Jolly CBD| Jolly CBD Gummies Review| Jolly CBD Gummies Reviews| Jolly CBD Gummies Cost}

  35. That is really fascinating, You’re an excessively professional blogger.
    I’ve joined your rss feed and sit up for looking for extra of your excellent
    post. Also, I’ve shared your web site in my social networks

    my web-site: win at Casino

  36. I’ve been exploring for a little for any high-quality articles or weblog posts on this sort of space .
    Exploring in Yahoo I eventually stumbled upon this site.
    Reading this information So i’m glad to show that
    I have a very just right uncanny feeling I came upon just what I
    needed. I such a lot indisputably will make sure to don?t put out of
    your mind this web site and provides it a look regularly.

  37. I think this is among the most significant info for me. And i am glad reading your article.

    But want to remark on some general things, The site style is perfect, the articles is
    really nice : D. Good job, cheers

  38. Oh my goodness! Awesome article dude! Many thanks, However I am encountering
    issues with your RSS. I don’t know why I can’t join it.

    Is there anyone else having similar RSS problems? Anyone that knows the answer can you kindly respond?
    Thanx!!

    Feel free to visit my web site :: Laser Hair removal

Thanks for your support, You may click on ads to encourage us which assits to writers.

Leave a Reply

Your email address will not be published.