Chapter 5
Choosing a Statistical Test
Having completed initial data analysis (IDA) you may now want to consider inferential
statistical procedures. This stage of the analysis will be based on some kind of probability
model and will more than likely involve one or more statistical tests. With a number of
alternative statistical tests to choose from the new researcher is often unsure about which
test to use. A few general considerations will be presented first, then a strategy for
choosing among the possible statistical tests will be described and summarized in the
form of a decision chart. Other considerations such as statistical power and its
relationship to statistical tests, alpha (Type I error), sample size, effect size, and variance
will be discussed, and the ubiquitous question about sample size will be addressed.
Examples of sample size and power calculations are presented. Finally, testing for
normality of distributions is illustrated and what to do when distributions are non-normal
is considered.
5.1 General ConsiderationsThe purpose of statistical inference and the use of statistical tests is to draw conclusions
from sample data. When choosing a statistical test a number of matters should be
considered: how data was (will be) generated, the study design, measurement issues,
distribution of response variable(s) in the population of interest, results of IDA,
specification of research questions and hypotheses to be tested, choice of an underlying
statistical model, specific statistical test assumptions. Regard should also be given to
statistical power and how this is related to the study design and choice of statistical
test(s). It is the author’s belief that generally too much stress is placed on hypothesis
testing and the reporting of p-values (see comments in Chapter 4). It is suggested,
therefore, that emphasis be given to estimation and the use of confidence intervals for
reporting tests of significance (p-values should of course also be reported).
How a Decision About Statistical Significance is ReachedFormal statistical inference used in hypothesis testing is based on probability theory. A
significance test is a test of a null hypothesis (hypothesis about population parameters),
the strength of evidence against the null hypothesis is assessed using the idea of