14 Combining Models
In earlier chapters, we have explored a range of different models for solving classifi-
cation and regression problems. It is often found that improved performance can be
obtained by combining multiple models together in some way, instead of just using
a single model in isolation. For instance, we might trainLdifferent models and then
make predictions using the average of the predictions made by each model. Such
combinations of models are sometimes calledcommittees. In Section 14.2, we dis-
cuss ways to apply the committee concept in practice, and we also give some insight
into why it can sometimes be an effective procedure.
One important variant of the committee method, known asboosting, involves
training multiple models in sequence in which the error function used to train a par-
ticular model depends on the performance of the previous models. This can produce
substantial improvements in performance compared to the use of a single model and
is discussed in Section 14.3.
Instead of averaging the predictions of a set of models, an alternative form of