The assessment and explanation of why, when, and how much the use of a combination of classifiers is better than the use of a single classifier is one of the most persistent topics in machine learning. In this paper, the authors study the generalization error of a variant of bagging, using the leave-one-out error and stability as theoretical tools for obtaining error bounds, in order to compare single and combined support vector machines (SVMs).

The authors are able to show, theoretically and experimentally, that the bounds are tighter for ensembles than for single classifiers when the learning system is not stable, as Breiman claimed in his original paper on bagging [1]. This is so because bagging achieves higher stability, which doesn’t necessarily mean that the test error is always lower. This increase in stability also explains why bagging reaches a saturation point when too many classifiers are combined.

Some results support combining several models trained on small subsets, rather than learning a single model from the whole dataset (a clear connection to the modern technique known as chunking) in order to increase accuracy. It would be more efficient, since learning time usually grows nonlinearly, with respect to the number of examples.

In this paper, only SVM and a variant of bagging are considered, and the conclusions leave many open questions. Some of the theoretical results are difficult to follow without the proper background on leave-one-out error estimates and kernel machines. Overall, though, the work represents one step forward in the understanding of ensembles.