independent samples when sample sizes are small, unequal and there are outlier
observations. Another problem is performing statistical tests when they are simply not
required. Perusal of the psychology and education periodicals indicates that it is rare for
shape parameters (skewness and kurtosis) to be reported—a case of forgotten moments.
Results of the IDA will indicate when statistical tests are unnecessary. Typical
situations would be when a whole population is assessed, when means of two large
samples are identical, or when large samples of equal size have non-overlapping
confidence intervals for the means when plotted. Other more critical situations would be
when there was evidence of bias, outliers, and non-constant errors perhaps due to
inadequate randomization or non-probability samples.
Generally one data set should not be used to both generate and test hypotheses. If the
data set is of sufficient size it can be randomly partitioned into two equal data sets, a
training data set and a confirmatory data set. The training set is used to generate
hypotheses and the confirmatory set is used to test these hypotheses. It is always
necessary to be aware that statistical significance does not equate with educational or
clinical significance. The result of a statistical test is only an aid to decision making. For
example, a significant mean difference of three points on a vocabulary acquisition test,
between two groups, may not hold much educational significance when considered in the
context of natural growth in vocabulary acquisition.
5.3 Choosing a Statistical TestThe problem for the new researcher when choosing a statistical test is lack of experience
with applied statistical procedures. The reader may have studied an introductory course in
statistics but will not have experience of applied methods. This chapter and indeed the
whole book concentrates on, ‘the when to do this or that...and why’ rather than rote
learning and computational detail. A series of choices when choosing a statistical test are
presented and summarized in Figure 5.1. For those who want to check whether a
procedure they have in mind is appropriate, they can go directly to Figure 5.1 (pp. 130–1)
which is self explanatory. If you want to develop your applied skills then read through the
following three-point rationale on which the decision chart shown in Figure 5.1 is based.
When deciding what statistical test to use, three interrelated issues need to be
considered:
1 Research question. Is the main research question concerned with
association/relationship, dependence/prediction between measures (same individuals),
or comparison/differences between groups?
2 Research design. How many groups are there in the study and is there any relationship
between them? For example, if there are two or more groups of data are they related
or independent? If each set of scores is obtained from a different sample of subjects,
the groups are independent. If different measures are obtained from the same group of
subjects on two (or more) occasions, i.e. the same subjects take two or more tests, or
subjects take the one test on two or more occasions, then the measures are related or
dependent.
3 Data distributions. Are the distributions of the important variables discrete with
inferences based on count data, for example, binomial, nominal or ranked data? Or are
Choosing a statistical test 121