Data Mining: Practical Machine Learning Tools and Techniques, Second Edition

90% used in 10-fold cross-validation. To compensate for this, we combine the

test-set error rate with the resubstitution error on the instances in the training

set. The resubstitution figure, as we warned earlier, gives a very optimistic esti-

mate of the true error and should certainly not be used as an error figure on its

own. But the bootstrap procedure combines it with the test error rate to give a

final estimate eas follows:

Then, the whole bootstrap procedure is repeated several times, with different

replacement samples for the training set, and the results averaged.

The bootstrap procedure may be the best way of estimating error for very

small datasets. However, like leave-one-out cross-validation, it has disadvantages

that can be illustrated by considering a special, artificial situation. In fact, the

very dataset we considered previously will do: a completely random dataset with

two classes. The true error rate is 50% for any prediction rule. But a scheme that

memorized the training set would give a perfect resubstitution score of 100%

so that etraining instances=0, and the 0.632 bootstrap will mix this in with a weight

of 0.368 to give an overall error rate of only 31.6% (0.632 ¥50% +0.368 ¥0%),

which is misleadingly optimistic.

5.5 Comparing data mining methods

We often need to compare two different learning methods on the same problem

to see which is the better one to use. It seems simple: estimate the error using

cross-validation (or any other suitable estimation procedure), perhaps repeated

several times, and choose the scheme whose estimate is smaller. This is quite

sufficient in many practical applications: if one method has a lower estimated

error than another on a particular dataset, the best we can do is to use the former

method’s model. However, it may be that the difference is simply caused by esti-

mation error, and in some circumstances it is important to determine whether

one scheme is really better than another on a particular problem. This is a stan-

dard challenge for machine learning researchers. If a new learning algorithm is

proposed, its proponents must show that it improves on the state of the art for

the problem at hand and demonstrate that the observed improvement is not

just a chance effect in the estimation process.

This is a job for a statistical test that gives confidence bounds, the kind we

met previously when trying to predict true performance from a given test-set

error rate. If there were unlimited data, we could use a large amount for train-

ing and evaluate performance on a large independent test set, obtaining confi-

dence bounds just as before. However, if the difference turns out to be significant

we must ensure that this is not just because of the particular dataset we

ee=¥+¥0 632 test instances 0 368...etraining instances

5.5 COMPARING DATA MINING METHODS 153