the error rate most commonly associated with them, the kinds of comparisons they are pri-
marily designed to test, and the type of test (range test, Ftest, or t—modified or not in
each case).12.8 Which Test?
Choosing the most appropriate multiple-comparison procedure for your specific situation
is not easy. Many tests are available, and they differ in a number of ways. The choice is a
bit easier if we consider the two extremes first.
If you have planned your test in advance and you want to run only one comparison,
I would suggest that you run a standard t test (correcting for heterogeneity of variance if
necessary), or, if you have a complex comparison, a linear contrast. If you have several a
priori contrasts to run, not necessarily pairwise, the multistage Bonferroni t proposed by
Holm does a good job of controlling FWwhile at the same time maximizing power.
If you have a large number of groups and wish to make many comparisons, whether or
not you are interested in all of the possible pairwise comparisons, you would probably be
better off using the Ryan REGWQ if you have it available or, if not, the Tukey. I can’t think
of a situation where I would personally recommend the Scheffé, but I presented it here
because it is a common test and real hard-liners like it. (In fact I recently wrote an article
about it for someone else and came to appreciate that it does have some redeeming virtues
[Howell, 2009]).
What about the Benjamini-Hochberg test? This is a difficult test to place in a table
because it controls an entirely different error rate. It is not fair to say that one test is more
powerful than another when they are working on different error rates. I have consider-
able fondness for the Benjamini-Hochberg test just because it is not based on the idea
that one false rejection invalidates a family of conclusions. If you are willing to accept
an occasional Type I error to gain power for other contrasts, there is much to recommend
this test.398 Chapter 12 Multiple Comparisons Among Treatment Means
Table 12.8 Comparison of alternative multiple-comparison procedures
Error A Priori/
Test Rate Comparison Type Post Hoc- Individual ttests PC Pairwise t A priori
- Linear contrasts PC Any contrasts F A priori
- Bonferroni t FW Any contrasts t‡ A priori
- Holm: Larzelere & Mulaik FW Any contrasts t‡ Either
- Fisher’s LSD FW† Pairwise t Post hoc
- Newman–Keuls FW† Pairwise Range Post hoc
- Ryan (REGWQ) FW Pairwise Range Post hoc
- Tukey HSD FW Pairwise Range‡ Post hoc
- Scheffé test FW Any contrasts F‡ Post hoc
- Dunnett’s test FW With control F‡ Post hoc
- Benjamini-Hochberg FDR Any contrasts t‡ Post hoc
†Against complete null hypothesis
‡ Modified