2
$\begingroup$

If I run a Random Forest on a set of data and get an accuracy of let's say 85% and I want to produce better results, I could just increase the amount of decision trees I use. Lets say I Increase the amount of trees I use and now I get an accuracy of 94%. If theoretically I used an infinite amount of trees would I get a perfect accuracy of 1 or as we increase the amount of trees is there a limit to how accurate we can become as we converge to a certain accuracy (like 98.76%). Of course there are many different factors to this but I think there should be a limit "n amount of trees" to where adding more wouldn't increase our accuracy or increase it by such a small amount that it doesn't matter anymore.

$\endgroup$
4
$\begingroup$

The limit of accuracy on your predictions is the Bayes error rate.

Adding additional decision trees does not always increase overall accuracy but will increase your training accuracy. Here is a picture that may be helpful to illustrate this:

When you increase the number of decision trees in a random forest, you are increasing your model complexity, which will cause your training accuracy to decrease but if you add too many decision trees, you begin to lose validation accuracy. Ideally, you want to build a model that balances this bias-variance tradeoff

$\endgroup$
1
$\begingroup$

The question asks about a "theoretical limit" and Joseph's answer correctly describes the tradeoff when you have a fixed, finite sample. Adding too many trees can lead to overfitting.

But what about the other kind of theoretical limit, as you get more data and increase the number of trees accordingly? It is not guaranteed that even with infinite data you will approach the Bayes error rate. This property is called consistency, and it depends on the exact algorithm used to generate the random forest, and on what assumptions you're willing to make about the data generating process.

Fortunately, a paper by Biau, Devroye and Lugosi (Consistency of Random Forest Classifiers, Journal of Machine Learning Research 2008) provides some algorithms that can achieve consistency under pretty general conditions. They also provide some conditions under which the original random forest algorithm is not consistent.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.