Statistical Methods for Psychology

The procedure we will follow is called the Benjamini and Hochberg’s Linear Step

Up (LSU) procedure.^12 I will not take the space to develop the logic of this test, but the

paper by Benjamini and Hochberg (1995) and the chapter by Maxwell and Delaney (2004)

are reasonably clear. Instead I will frame this discussion in terms of the steps needed to per-

form the test.

Assume that we have performed 10 pairwise contrasts on Siegel’s morphine data. The

results are shown in Table 12.7 ordered by pvalue. The column labeled “i” is the index of

the comparison and simply ranks the pvalues from lowest to highest. The critical part of

the table is labeled pcrit, the critical value for our test. We define

where “i” is the index, “k” is the number of tests (here k 5 10), and ais the desired FDR

(here a5.05). To carry out the test we work our way down the table. If p.pcritwe retain

the null hypothesis and move on to the next row. As soon as p,pcritwe reject that null

hypothesis and all subsequent ones.

Using the Benjamini-Hochberg test we would declare that MM-SS, SM-McM, and

MS-MM are not different from each other pairwise. All other contrasts are judged

statistically significant. Had we used Ryan’s REGWQ test we would have failed to reject

the MS-SS contrast, whereas we rejected that null hypothesis here. In the preceding exam-

ple it would be perfectly appropriate to include non-pairwise contrasts as long as they are

orthogonal to the other contrasts that you have used. Thus you could remove the SM-MM

and the SM-McM comparisons and replace them with SM-(M-M 1 Mc-M) 2.

12.7 Comparison of the Alternative Procedures

Because the multiple-comparison techniques we have been discussing were designed for

different purposes, there is no truly fair basis on which they can be compared. There is

something to be gained, however, from summarizing their particular features and compar-

ing the critical differences they require for the same set of data. Table 12.8 lists the tests,

>

pcrit=a

i

k

ba

12.7 Comparison of the Alternative Procedures 397

Table 12.7 Benjamini-Hochberg test on Siegel’s data

Group tpipcrit Significance

MM-SS

SM-McM

MS-MM

MS-SS

SS-SM

MM-SM

SS-McM

MM-McM

MS-SM

MS-McM

2 .354

2 1.768

2 2.121

2 2.475

2 4.596

2 4.950

2 6.364

2 6.717

2 7.071

2 8.839

.726

.086

.041

.018

.00007

.00003

.00000

.00000

.00000

.00000

10

9 8 7 6 5 4 3 2 1

.05

.045

.040

.035

.030

.025

.020

.015

.010

.005

No

No

No

Ye s

Ye s

Ye s

Ye s

Ye s

Ye s

Ye s

(^12) Benjamini and Hochberg (2000) recommended a variation on the test given here, sometimes called their
“adaptive” test. It is more powerful than the LSU test, but somewhat more cumbersome. Both of these tests are
different from the Hochberg GT2 test produced by SPSS.
Benjamini and
Hochberg’s
Linear Step Up
(LSU) procedure