404 Statistical Methods
are the same. Other methods are available to help you adjust the p value for
multiple comparisons, including Tukey’s and Scheffé’s, but the Bonferroni
method is the easiest to implement in Excel, which does not provide a
correction procedure.
Note: Essentially, the difference between the Bonferroni procedure and a
t test is that for the Bonferroni procedure, the 5% applies to all six compari-
sons together but for t tests, the 5% applies to each of the six comparisons sep-
arately. In statistical language, the Bonferroni procedure is testing at the 5%
level experimentwise, whereas the t test is testing at the 5% level pairwise.
The pairwise comparison probabilities show that the three biggest differ-
ences are signifi cant (highlighted in red). The New York city room price is
higher than the room price in the other three cities, but none of those three
cities are signifi cantly different in price from each other.When to Use Bonferroni
As the size of the means matrix increases, the number of comparisons in-
creases as well. Consequently, the p values for the pairwise differences are
greatly infl ated. As you can imagine, there might be a point where there are
so many comparisons in the matrix that it is nearly impossible for any one
of the comparisons to be statistically signifi cant using the Bonferroni cor-
rection factor. Many statisticians are concerned about this problem and feel
that although the Bonferroni correction factor does guard well against incor-
rectly fi nding signifi cant differences, it is also too conservative and misses
true differences in pairs of mean values.
In such situations, statisticians make a distinction between paired com-
parisons that are planned before the data are analyzed and those that occur
only after we look at the data. For example, the planned comparisons here are
the differences in hotel room price between New York City and the others.
You should be careful with new comparisons that you come up with after
you have collected the data. You should hold these comparisons to a much
higher standard than the comparisons you’ve planned to make all along. This
distinction is important in order to ward off the effects of data “snooping”
(unplanned comparisons). Some statisticians recommend that you do the
following when analyzing the paired means differences in your analysis of
variance:- Conduct an F test for equal means.
- If the F statistic is signifi cant at the 5% level, make any planned compar-
isons you want without correcting the p value. For data snooping, use a
correction factor such as Bonferroni’s on the p value. - If the F statistic for equal means is not signifi cant, you can still consider
any planned comparisons, but only with a correction factor to the p value.
Do not analyze any unplanned comparisons (Milliken and Johnson 1984).
It should be emphasized that although some statisticians embrace this
approach, others question its validity.