Data Mining: Practical Machine Learning Tools and Techniques, Second Edition

an expected error of one if the dataset contains no +instances and zero if all its

instances are +; the other always predicts -, giving the opposite performance.

The dashed horizontal line shows the performance of the classifier that is always

wrong, and the X-axis itself represents the classifier that is always correct. In

practice, of course, neither of these is realizable. Good classifiers have low

error rates, so where you want to be is as close to the bottom of the diagram as

possible.

The line marked A represents the error rate of a particular classifier. If you

calculate its performance on a certain test set, its false positive rate fpis its

expected error on a subsample of the test set that contains only negative exam-

ples (p[+] =0), and its false negative rate fnis the error on a subsample that

contains only positive examples (p[+] =1). These are the values of the inter-

cepts at the left and right, respectively. You can see immediately from the plot

that ifp[+] is smaller than about 0.2, predictor A is outperformed by the extreme

classifier that always predicts -, and if it is larger than about 0.65, the other

extreme classifier is better.

174 CHAPTER 5| CREDIBILITY: EVALUATING WHAT’S BEEN LEARNED

probability p [+]

expected

error

always pick –

always pick +

always wrong

always right

A

fn

fp

0

0

0.5

1

0.5 1

(a)

Figure 5.4Effect of varying the probability threshold: (a) the error curve and (b) the

cost curve.