Statistical Analysis for Education and Psychology Researchers

1.1 Why Consider Research Design in a Text About Statistics?

It is important to appreciate that statistics is much more than a collection of techniques
for data analysis. Statistical ideas can and should be used to advantage in the design and
data collection stages of a study. A well designed study that has generated reliable data
but which has been poorly analyzed can be rescued by appropriate reanalysis. A poorly
designed study, however, that has generated data of dubious quality, is beyond
redemption, no matter how sophisticated the statistical analysis.

Use of Statistical Ideas in Research Planning

Sample size and statistical power

When planning a survey or an experiment a common problem for researchers is the
determination of sample size or number of subjects in experimental groups. It is possible
to estimate the number of subjects required either in a sample survey or in experimental
design so that sample or treatment differences would be detected at a specified
significance level. The significance level of a statistical test is the likelihood of
concluding there is a difference (rejecting a hypothesis of no difference) when in fact
there is a difference (the hypothesis of no difference is refuted). The estimation of sample
size is achieved through statistical power analysis. Given certain assumptions, a statistical
test is said to be powerful if it is able to detect a statistically significant difference should
one exist (statistical power analysis is considered in Chapter 5). The point of doing a
power analysis for a research plan based on a particular sample size is that if the design
turns out to have insufficient power, that is one is unable to detect any statistically
significant difference, then the researcher can revise the plan. One option would be to
increase the sample size. There are other options, including, at the extreme, dropping the
proposed study. Clearly, as little can be done after data has been collected, consideration
of sample size and statistical power is crucial at the planning stage.
It should also be emphasized that statistical significance does not always imply
educational significance. For example, a small gain in maths scores after an experimental
maths programme may be statistically significant but may be considered to be of no
educational importance. The researcher in planning an evaluation of this maths
programme would have to determine what maths gain score would be considered a
significant educational gain and design a study to be able to detect this magnitude of
treatment effect.

Validity and reliability of measurement

Attention should be given to the construction of measuring instruments like
questionnaires and sociometric indices. A common problem encountered with self-
completion questionnaires is missing responses, often referred to as ‘missing data’. The
best answer to this particular problem is to have none. If you do have missing data, this
often tells you as much about the design of your questionnaire as the knowledge,
opinions or thoughts of the respondent. The pattern of missing responses is also
informative. Descriptive analysis of pilot study data may reveal selective non-response or

Statistical analysis for education and psychology researchers 2