Statistical Analysis for Education and Psychology Researchers

• How many individuals should be allocated to each experimental group?


• How will individuals be allocated to experimental groups?

These questions involve the idea of chance. Provided a study is designed systematically
we know the role that chance plays in the generation of data. This enables assumptions to
be made, such as the equivalence of experimental groups before any treatment. Prior to
data analysis and preferably prior to data collection, consideration should be given to the
general issue of how data will be collected. In Example 3 an unbalanced design is used; a
balanced design would have been better. Balanced designs are the most efficient, because
they have most statistical power for a given number of subjects. Statistical power in
experimental research is the chance of detecting a treatment difference should one exist.
To maintain the same statistical power, the total number of subjects in an unbalanced
design needs to be greater than in a balanced design.

Summary

It is important to stress that at the design stage before you collect any data—or if given
data before you begin analysis—use your judgment to consider the context which
generated the data. For example, certain questions might usefully be asked: Were the data
collected to answer a specific question or for some other reason? How were subjects
included in the study? What exactly was observed, counted or measured? How was
measurement carried out and was it valid and reliable? What are the sources of error? Put
simply, you should assess the quality and structure of the data. Statistical guidelines are
presented in Figure 1.4 as a summary of the main points covered in this chapter. The last
two points in these guidelines relate to more detailed statistical concepts which will be
covered in later chapters. The guidelines should not be seen as definitive but as an aide-
mémoire to important aspects of statistical design that should be considered when
planning an empirical quantitative study.

1. Is the purpose of the study clear? e.g., Is it exploratory, predictive,
   causal?


2. Are the proposed method(s) commensurate with the main aims of the
   study? e.g., Is probability sampling to be used so that generalizations
   can be made?


3. Are criteria specified for recruitment of subjects? e.g., What is the
   sampling frame, who is included and who is excluded?


4. How will subjects be selected or allocated to groups? e.g., Will there be
   random/ non-random sampling, randomization, comparison groups not
   randomly allocated (nonequivalent groups)?


5. Are procedures for data generation clearly specified? e.g., Will
   questionnaires be constructed and if so are they valid and reliable or
   will fidelity of treatments in experimental designs be checked?


6. Have sample size and power been considered? e.g., What is the
   expected magnitude of effect and is the design sensitive enough to
   detects this?

Statistics and research design 15