smooth curve like that in Figure 5.2—although in reality such curves do not
generally look quite so smooth.
This is just one way of using cross-validation to generate ROC curves. A
simpler approach is to collect the predicted probabilities for all the various test
sets (of which there are 10 in a 10-fold cross-validation), along with the true
class labels of the corresponding instances, and generate a single ranked list
based on this data. This assumes that the probability estimates from the classi-
fiers built from the different training sets are all based on equally sized random
samples of the data. It is not clear which method is preferable. However, the
latter method is easier to implement.
If the learning scheme does not allow the instances to be ordered, you can
first make it cost sensitive as described earlier. For each fold of a 10-fold cross-
validation, weight the instances for a selection of different cost ratios, train the
scheme on each weighted set, count the true positives and false positives in the
test set, and plot the resulting point on the ROC axes. (It doesn’t matter whether
the test set is weighted or not because the axes in the ROC diagram are expressed
as the percentage of true and false positives.) However, for inherently cost-
sensitive probabilistic classifiers such as Naïve Bayes it is far more costly than
the method described previously because it involves a separate learning problem
for every point on the curve.
It is instructive to look at cross-validated ROC curves obtained using differ-
ent learning methods. For example, in Figure 5.3, method A excels if a small,
focused sample is sought; that is, if you are working toward the left-hand side
of the graph. Clearly, if you aim to cover just 40% of the true positives you170 CHAPTER 5| CREDIBILITY: EVALUATING WHAT’S BEEN LEARNED
020%40%60%80%100%0 20% 40% 60% 80% 100%
false positivestrue positives
ABFigure 5.3ROC curves for two learning methods.